I’m getting random utf-8 encoding errors from my dataset: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xbc in position 117020: invalid start byte These are datasets saved (using dataset.save_to_disk('/out/file/dir') from the datasets library, and are saved as .arrow files. I say “random” …

Random utf-8 errors from dataset

sibiakash December 7, 2023, 8:01am 7

hey, stuck with the same error. found a fix ?

Topic		Replies	Views
Datasets.load_datasets fails 🤗Datasets	12	847	October 11, 2024
'utf-8' codec can't decode byte 0xff in position 0: invalid start byte Beginners	3	12081	August 23, 2023
'utf-8' codec can't decode byte 0xff in position 0: invalid start byte (dataset) Beginners	0	392	May 19, 2024
UTF-16 for datasets? 🤗Datasets	4	1486	June 21, 2023
Turkish characters gets corrupted when loading dataset via audiofolder 🤗Datasets	1	517	April 4, 2023