Working on a translator, hoping to do fine-tuning with a utf-16 dataset so I can get all the French accents etc. Datasets load_dataset() doesn’t seem to like non-utf-8 Is there a way to specify or does it HAVE to be utf-8? If it has to be utf-8, any suggestions for special characters?

UTF-16 for datasets?

mariosasko June 19, 2023, 7:16pm 2

You can pass encoding="utf-16" to the load_dataset call.

2 Likes

Random utf-8 errors from dataset

Topic		Replies	Views
Datasets.load_datasets fails 🤗Datasets	12	892	October 11, 2024
UnicodeDecodeError when loading Mulit Lingual text file 🤗Datasets	1	2375	April 7, 2022
UniDecodeError: 'charmap' codec can't decode byte from Load_dataset Beginners	0	63	December 5, 2024
HF Datasets loading csv Beginners	1	1099	January 30, 2021
Random utf-8 errors from dataset Intermediate	10	3710	December 8, 2023