While tokenizing the dataset I'm getting this type of error

can anyone help how to solve it…

I can’t see the error message. Can you paste the trace along with error message.

sorry there is no error message but process stops at 95%

I’m using this example notebooks/language_modeling_from_scratch.ipynb at main · huggingface/notebooks · GitHub

and change the dataset to OSCAR

Hi! This is a (known) bug in the printing logic, not in the processing itself (see Progress bars have color red and never completed to 100% · Issue #5117 · huggingface/datasets · GitHub), which we’ve already addressed. You can either install from “main” (pip install git+https://github.com/huggingface/datasets.git) or wait for the next release to fetch the fix.

1 Like