Hi
I am trying out this tutorial here (Iink), which is about training a large language model from scratch, but the training process is really slow with Trainer. For the 6 GB dataset, it is mentioned that it will take about 20 hours for the full dataset, 2 hours for 10% of the dataset, but i see that it’ll take 15 days for training to complete for the whole dataset.
I think some part of the code is being executed on CPU, (I see something about iterations per second), and i was wondering if there is any way to speed it up?
I am a beginner at this. I have 2x3080TI gpus, and a really slow CPU. 64 gigs of RAM.
Thanks