I’ve been working on a side project that uses phonetic English language models for text generation. Since I’m not aware of any existing phonetic English datasets, I’ve been preprocessing existing English text datasets with my phonemization script to give myself enough training data. Mainly OSCAR for…

Understanding set_transform

BramVanroy February 20, 2021, 9:09am 2

set_transform does not cache the resulting data. Depending on the data/storage you have available you may want to opt for map. Both have a low memory footprint.

Topic		Replies	Views
Using load_dataset.set_transform() function along with Trainer class 🤗Datasets	4	2616	April 26, 2021
How to use set_transform when map becomes unfeasible? Intermediate	2	135	June 19, 2024
Pipeline with custom dataset tokenizer: when to save/load manually 🤗Datasets	18	5636	September 18, 2020
Transformed dataset to_json saves cache dataset Beginners	3	370	January 3, 2023
Set batch instead of full train dataset on Trainer 🤗Transformers	1	372	March 11, 2024

Understanding set_transform

Related topics