[image] I create my custom dataset to train Donut and want to save it to disk: dataset.save_to_disk(“/content/drive/MyDrive/image-dataset/huggingface-dataset”) This is the structure of the dataset: DatasetDict({ train: Dataset({ features: [‘image’, ‘id’, ‘ground_truth’], num_rows: 528 }) …

Hi! Can you interrupt the process while it hangs and paste the error traceback here? This behavior is due to a bug in save_to_disk where we skip the last progress bar update. However, this bug doesn’t affect the actual saving of a dataset to disk - this can verified by reloading the saved dataset w…

Saving custom dataset does not finish

🤗Datasets

YanaS March 16, 2023, 5:47am 4

Thank you. Yes it seems that everything is fine after loading the saved dataset.

Topic		Replies	Views
Working with large datasets 🤗Datasets	5	4194	November 10, 2020
Saving checkpoints 🤗Datasets	1	883	November 5, 2021
Dataset slow during model training 🤗Datasets	1	2028	June 13, 2022
Save and load datasets 🤗Datasets	2	40097	August 16, 2021
Pipeline with custom dataset tokenizer: when to save/load manually 🤗Datasets	18	5647	September 18, 2020

Saving custom dataset does not finish

Related topics