I am training a 3B LLM from scratch with 1 million text samples, and it takes 1/2 hour to get through the “Generating split train…”. I get ~500 samples per second maximum in bursts. I’m wanting to scale to 200 Million samples, which says it will take 7 days. If I tokenize and pack the data myself using map(), with num_proc of 64, and large batches, this can be done in less than 8 hours instead of days.
I see that you can pass in a sharded file if you have a streaming dataset, but using streaming gives other errors. How can I trick it to use multiple workers when Trainer is “Generating split” inside of Trainer or SFTTrainer?