Does from_file try to read the whole dataset?
I am in a similar place where I spent days adding columns to a dataset using .map() but the execution failed when I tried to save_to_disk. So I am hoping to save it from the cache.
The map operation ended successfully making the dataset huge (1.3TB)
Originally i planned to load the dataset as an iterable to only use enough rows in a batch that will fit in Memory, I am currently trying Datasets.from_file(“the larger arrow file”). But it’s really taking its time so I am wondering if this is still usable. I am also not sure which arrow file I should be trying to read I am hoping that the one with the cache-prefix contains all the data and not just part of it.