Dataset map occasionally throws error

🤗Datasets
1

Hello, I am getting this error occasionally while maping. Any ideas why and/or how to debug this ? Is this a issue ?

Map:  14%|█████████▏                                                       | 999/7030 [6:06:58<36:55:23, 22.04s/ examples]
Traceback (most recent call last):
  File "/Users/user/miniconda3/envs/gen310/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 3452, in _map_single
    writer.write(example)
  File "/Users/user/miniconda3/envs/gen310/lib/python3.10/site-packages/datasets/arrow_writer.py", line 491, in write
    self.write_examples_on_file()
  File "/Users/user/miniconda3/envs/gen310/lib/python3.10/site-packages/datasets/arrow_writer.py", line 445, in write_examples_on_file
    batch_examples[col] = [
  File "/Users/user/miniconda3/envs/gen310/lib/python3.10/site-packages/datasets/arrow_writer.py", line 446, in <listcomp>
    row[0][col].to_pylist()[0] if isinstance(row[0][col], (pa.Array, pa.ChunkedArray)) else row[0][col]
TypeError: 'NoneType' object is not subscriptable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/user/workspace/genlaw/model/data_curation.py", line 189, in <module>
    DataCurator().create_training_data()
  File "/Users/user/workspace/genlaw/model/data_curation.py", line 113, in create_training_data
    ds_train = ds_train.map(self.summarize,  with_indices=True)
  File "/Users/user/miniconda3/envs/gen310/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 591, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/Users/user/miniconda3/envs/gen310/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 556, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/Users/user/miniconda3/envs/gen310/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 3089, in map
    for rank, done, content in Dataset._map_single(**dataset_kwargs):
  File "/Users/user/miniconda3/envs/gen310/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 3497, in _map_single
    writer.finalize()
  File "/Users/user/miniconda3/envs/gen310/lib/python3.10/site-packages/datasets/arrow_writer.py", line 587, in finalize
    self.write_examples_on_file()
  File "/Users/user/miniconda3/envs/gen310/lib/python3.10/site-packages/datasets/arrow_writer.py", line 445, in write_examples_on_file
    batch_examples[col] = [
  File "/Users/user/miniconda3/envs/gen310/lib/python3.10/site-packages/datasets/arrow_writer.py", line 446, in <listcomp>
    row[0][col].to_pylist()[0] if isinstance(row[0][col], (pa.Array, pa.ChunkedArray)) else row[0][col]
TypeError: 'NoneType' object is not subscriptable
2

Hi! Can you share the map transform’s code?