In the following code, why are we not making padding = True inside the tokenizer, because don’t we want all the lists to be of the same size? max_length = 128 def preprocess_function(examples): inputs = [ex[“en”] for ex in examples[“translation”]] targets = [ex[“fr”] for ex in examples[“translat…

Padding should be True, please explain

Sanyam52 August 18, 2024, 9:07am 2

I just read up, padding is the job of DataCollator which pads -100 so that our loss function ignores it.
Is this correct?

Topic		Replies	Views
DataCollator not padding as expected Intermediate	0	668	August 17, 2022
Different padding behaviour of data collator 🤗Transformers	0	119	August 23, 2024
Padding in datasets 🤗Datasets	6	5080	October 21, 2021
It asks to add padding or truncation but I have already done it Beginners	1	841	October 6, 2023
Issues with Data Collator and Tokenizing with NER Datasets 🤗Tokenizers	1	2533	May 9, 2022