I need to use IITCDIP dataset on a server where I get 6 hours time slot. The data_loader is taking more than 6 hours for caching. So, I can do my training. Is there any way:
load custom dataset with caching (Stream) using script similar to here.
Resume the caching process
Cache dataset on one system and use on other system.
Note that I have tried up to 64
num_proc but did not get any speed up in caching processing.