How does Huggingface Trainer handle Iterable dataset on TPU?

ibraheemmoosa · February 16, 2022, 3:03pm

Can anyone tell me about how iterable datasets are handled in this type of multi-device scenario? How is it made sure that the same data is not duplicated across multiple devices?

Topic		Replies	Views
Train through multiple datasets Beginners	1	1227	June 13, 2022
Training with IterableDataset is very slow when using a large number of workers 🤗Transformers	0	766	August 19, 2023
Using External Datasets with HuggingFace Data Loader Beginners	9	3415	April 27, 2022
Which URLs should be reachable to work with Huggingface hub 🤗Datasets	2	919	January 26, 2022
Use load dataset to load a sample of the dataset 🤗Datasets	3	1052	May 24, 2021

How does Huggingface Trainer handle Iterable dataset on TPU?

Related Topics