Loading Custom Datasets

🤗Datasets
8

This IndexError you get during training means that it tries to query the first element of an empty dataset. However it looks like your dataset is not empty (2 elements), and that you are able to query the first element manually.

So it means that something makes the dataset empty at one point before passing the dataset to the Trainer. Can you also check the length of the dataset after applying this ?

dataset = dataset.map(features.encode_example, features=features)
EDIT: according to Kunal this is might be because `tokenize_and_align_labels` (from run_ner.py) needs to be applied on the dataset first. Indeed if the dataset doesn't contain the right input, the Trainer could see it as empty.