Dataset Tokenization to Fine-Tune Gemma 3 1B

[John6666]

This error occurs when DataCollator is unable to collate the data as expected. There are multiple possible solutions, but I think the smartest thing to do would be to either use an existing DataCollator or write a new one.

Alternatively, you could leave tokenization entirely to the Trainer, but that would mean a significant change to the current code, so I’m not sure what to do in that case…

HuggingFace: ValueError: expected sequence of length 165 at dim 1 (got 128)

Data Collator

Why use tokenizer in Trainer with Tokenized Data

Hi there! I am getting started with the implementation of the SageMaker integration using this notebook. I realised that this notebook preprocesses the train and test set, such that they are tokenized when calling the estimator later in the notebook. However, assessing the script called, the tokenizer is also provided to the Trainer. Can somebody explain to me why this is necessary? This is both a question out of curiosity and for this concrete application, as I am a) worried to tokenize twice…