Hi @lhoestq! I’m finally getting around to testing some set_transform workflows and I have a question.
I’ve passed a fairly CPU-heavy preprocessing function to set_transform. After about an hour of training, I forced the training to stop and then tried to resume from the last checkpoint.
It’s been over 15 minutes of heavy CPU activity since I resumed, and the training progress indicator is still on step zero. [UPDATE: training finally resumed after 46 minutes] Is it possible that my transform function being called on every sample as the trainer advances to the last checkpoint step?
If that is what’s happening, I’m not sure if it’s due to the dataset or trainer. Is there any way to avoid it?