Hi friends👋 I want to use RedPajama for training a language model. I have access to a folder where the entire dataset is already downloaded and now I want to load the local files with :hugs:datasets for simple usage. Other people are relying on the downloaded data. Therefore I’m not allowed to chan…

Hi! load_dataset saves a generated dataset in the Arrow format to be able to memory-map it later, so this is the expected behavior. If you are not interested in random access, you can pass streaming=True to get an IterableDataset that doesn’t require cache files.

Using local dataset without changing cache

🤗Datasets

cwallenwein September 6, 2023, 4:37pm 3

Perfect, thank you for this straightforward solution!

Topic		Replies	Views
Load dataset from a specific cache file 🤗Datasets	3	1271	February 26, 2024
How to disable caching in load_dataset()? 🤗Datasets	6	6425	July 10, 2024
Datasets not using the cache dir 🤗Datasets	2	729	November 29, 2023
How to load cached dataset offline? Beginners	2	4608	May 29, 2022
How to load dataset that exist in cache path Beginners	5	4981	December 6, 2023

Using local dataset without changing cache

Related topics