DataCollatorWithPadding does not take into account block_sparse thresholds of a model. If I use it to pad my batches to the longest sequence in a batch but all sequences in the batch are sufficiently small, block_sparse may be deactivated causing Out Of Memory issues. Why does something like the b…

DataCollatorWithPadding - Minimum Pad Argument

slyle August 17, 2023, 5:04pm 2

Edited this second post as I had confused myself lol. Are all models subject to different methodologies for determining minimum threshold for block_sparse?

Topic		Replies	Views
Attention type 'block_sparse' is not possible if sequence_length: 458 <= num global tokens: 🤗Transformers	4	1056	September 6, 2021
Bigbird pretraining Beginners	3	886	March 16, 2022
Padding should be True, please explain Beginners	1	13	August 18, 2024
Padding in datasets 🤗Datasets	6	5078	October 21, 2021
Padding strategy for classification Beginners	3	2504	July 20, 2020

DataCollatorWithPadding - Minimum Pad Argument

Related topics