Which is the most efficient way to create batches with sequences of similar length to minimize padding in HF datasets? Just calling torchtext’s BucketIterator? torchtext.data — torchtext 0.4.0 documentation ? Is there any reference implementation?

Efficient bucketing implementation

jordiae May 16, 2022, 2:40pm 5

Thank you!

Topic		Replies	Views
Padding in datasets 🤗Datasets	6	5064	October 21, 2021
Make text data continuous from DatasetDict 🤗Datasets	1	1186	May 11, 2022
HuggingFace dataset: each element in list of batch should be of equal size 🤗Datasets	3	10419	August 10, 2023
Training with varying lengths of sequences Beginners	0	1634	May 31, 2023
Data sampler based on number of tokens 🤗Transformers	0	731	February 4, 2022