I have a project in which I extract entities from multiple files, line by line. So I wrote a function that receives a file and both Roberta and it’s Tokenizer. The idea is to spawn multiple processes and run this function asynchronously for each file (actually at this point the files are already loa…

Problem with torch.multiprocessing and Roberta

thiagolira March 13, 2021, 4:41pm 2

This actually doesn’t work even with just one worker

p = ctx.Pool(1)

So I think it is related to the multiprocessing code

1 Like

Topic		Replies	Views
AutoTokenizer.encode with multiThread and mutliProcess 🤗Tokenizers	2	338	October 9, 2024
Problem with sharing models among processes via multiprocessing Models	0	943	May 11, 2023
Stucked on tokenization before training when using 3 GPU, but not when using 2 GPU Beginners	0	309	June 25, 2023
How to extract pytorch model from transformer pretrained model Models	0	529	August 13, 2022
Distributed inference on multiple files 🤗Transformers	1	1004	January 22, 2023