Which is the most efficient way to create batches with sequences of similar length to minimize padding in HF datasets? Just calling torchtext’s BucketIterator? torchtext.data — torchtext 0.4.0 documentation ? Is there any reference implementation?

Efficient bucketing implementation

conceptofmind May 12, 2022, 2:52pm 4

These articles on building a PyTorch Text Bucket Iterator with sequences of similar length and dynamic padding may be useful:

Here are other sources with a similar use case as well:

You can also sort and filter sequences by length and then use the .map() function to pad or truncate the rest of the batches.

Hopefully one of these will meet the criterion.

2 Likes

Topic		Replies	Views
Padding in datasets 🤗Datasets	6	5064	October 21, 2021
Make text data continuous from DatasetDict 🤗Datasets	1	1186	May 11, 2022
HuggingFace dataset: each element in list of batch should be of equal size 🤗Datasets	3	10419	August 10, 2023
Training with varying lengths of sequences Beginners	0	1634	May 31, 2023
Data sampler based on number of tokens 🤗Transformers	0	731	February 4, 2022