Too many open files on big datasets

🤗Datasets
3

This is helpful, but unfortunately I can’t use the solutions there since I don’t have root access. After much thought, I guess the only two solutions are:

  1. Remake my dataset
  2. Load random subsets of my dataset for each epoch