Which is the most efficient way to create batches with sequences of similar length to minimize padding in HF datasets? Just calling torchtext’s BucketIterator? torchtext.data — torchtext 0.4.0 documentation ? Is there any reference implementation?

Efficient bucketing implementation

jordiae May 12, 2022, 11:29am 3

Hi @conceptofmind, thanks for your detailed answer. I have in mind a seq2seq use case, not a LM one, that’s why bucketing is so important in this case.

Topic		Replies	Views
Padding in datasets 🤗Datasets	6	5064	October 21, 2021
Make text data continuous from DatasetDict 🤗Datasets	1	1186	May 11, 2022
HuggingFace dataset: each element in list of batch should be of equal size 🤗Datasets	3	10419	August 10, 2023
Training with varying lengths of sequences Beginners	0	1634	May 31, 2023
Data sampler based on number of tokens 🤗Transformers	0	731	February 4, 2022

Efficient bucketing implementation

Related topics