AutoTrain Advanced Error on Massive Dataset

[John6666]

It would be easier if Streaming (IterableDataset) could be used, but I don’t know how to enable Streaming in AutoTrainAdvanced…

It might be quicker to split the dataset into several smaller datasets in advance.

Best practices for a large dataset

Hi, I have a ~1TB large dataset stored at the HF hub. I can download this on my disk and read it successfully. Nevertheless, it’s large enough that I can’t fit it in my RAM. What is the best practice for training a model on such a dataset? I tried loading the dataset with load_dataset(..., streaming=True) and then having two buffers: one that is being loaded by training process onto GPU and one that, in a separate thread, streams from the dataset and fills up the other buffer. Then, when the i…

LLM Finetuning with AutoTrain Advanced

Dataset streaming and options

Load Dataset and Save as Parquet

I am curious whether we can directly save a dataset as parquet as in the original dataset at HF. For example fineweb-edu/sample/10BT has 13 parquet files # download the dataset fw = load_dataset( "HuggingFaceFW/fineweb-edu", name="sample-10BT, split="train", num_proc=10, cache_dir=cache_dir, ) fw.to_parquet(parquet_dir) Loads these files as arrow although they are originally parquet, and when saving with to_parquet() it converts arrow to a single parquet file. So my que…