One-to-many batch mapping with IterableDatasets and batch_size=1 doesn't work

🤗Datasets
1

Does batch mapping work properly for IterableDatasets? I have my processing code set up to return for each column a list of rows, but it seems to only ignore all other entries in the list except the first entry.

       labels_ids = [reasoning_labels, answer_labels]

        return {
            'labels_ids': labels_ids,
        }

However my dataset only includes the reasoning_labels rows.

I also changed the Dataset back to streaming=False and it includes the answer_labels rows as expected.

1 Like
2

Fixed. Turns out I had to remove all my original columns

1 Like
3

This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.