Hello, I’m trying to use the load_dataset.set_transform(…) function along with DataCollatorForLanguageModeling and the Trainer class from the transformers library to pretrain a model. Since I have a large dataset, tokenization does not fit in RAM and using the .map() function uses way too much dis…

Using load_dataset.set_transform() function along with Trainer class

lewtun April 26, 2021, 11:10am 2

hey @tkon3, when you say “it fails” what do you mean exactly? there’s a related thread on set_transform here that might be useful: Understanding set_transform

Topic		Replies	Views
What is happening in the trainer api, with data collator? Beginners	0	239	April 29, 2023
Using IterableDataset with Trainer - `IterableDataset' has no len() 🤗Transformers	6	9723	September 6, 2023
Help understanding how to build a dataset for language as with the old TextDataset 🤗Datasets	7	10716	October 6, 2021
Dataset expected by Trainer Beginners	5	7462	September 28, 2020
Using Datasets, DataCollators and DataLoaders to create an NLP data pipeline 🤗Datasets	1	1797	June 21, 2023

Using load_dataset.set_transform() function along with Trainer class

Related Topics