How can I use iterator to run ((transformers\examples\pytorch\summarization\run_summarization.py)) with huge data files (in JSON Lines format, size> 20 GB) to avoid loading the whole file into the memory? Any suggestions how to use the code with large size JSON files? Framework: Transformers 4.20…

Run_summarization.py + huge dataset

luisbnzsa March 1, 2023, 4:08pm 2

I would split the json file in 2 or 3 parts and do the training in 2 or 3 batches. You also have the batch size --device_train_batch_size maybe you can play with it.

Topic		Replies	Views
Generate multiple summaries 🤗Transformers	0	533	November 25, 2021
Output truncation of summaries models 🤗Transformers	0	442	March 30, 2023
AWS Lambda + Transformers + Docker = use High RAM for summarization model 🤗Transformers	1	596	June 26, 2023
Speed issues using tokenizer.train_new_from_iterator on ~50GB dataset 🤗Transformers	8	2324	August 29, 2025
Summarization on long documents 🤗Transformers	63	59158	August 16, 2024

Run_summarization.py + huge dataset

Related topics