There doesn’t seem to be any padding occurring here: train_dataset = dataset.shard(10, 1) train_dataset.set_format(columns=['text']) train_dataset.cleanup_cache_files() encoded_dataset = train_dataset.map(lambda examples: tokenizer(examples['text'], padding=True)) encoded_d…

Not sure why padding isn't working for me

thecity2 January 22, 2021, 6:29pm 3

Thank you so much @sgugger! That makes sense now.

Topic		Replies	Views
Padding in datasets 🤗Datasets	6	5088	October 21, 2021
How to pad tokens to a fixed length on a single sentence? Beginners	1	3194	May 19, 2021
Padding strategy for classification Beginners	3	2511	July 20, 2020
How padding in huggingface tokenizer works? 🤗Tokenizers	4	6981	November 22, 2021
DataCollator not padding as expected Intermediate	0	670	August 17, 2022