Cannot Start the training loop because of bad size tokenization and/or for (presumably) custom dataset settings

Hi! I’m new to the hugging face library, and I’m trying to do a fine tune for a sequence classification task on a cyberbullying dataset with 6 classes.

Originally, I wasn’t able to perform Training due to the tokenization of the dataset. It wasn’t performing in a good manner because of different dimensions for the tensor.

I started to change the code to try to perform properly the tokenization, and now I get this error:

“ValueError: too many values to unpack (expected 2)”

I think that, most probably, I manipulated in a bad manner the dataset(that I downloaded from another source).
Here’s what I’ve done:

def lowerWords(phrase):return phrase.lower()
def tokenize_sample(example):return tokenizer(example["tweet_text"], padding = "max_length", truncation = True, max_length = 512, return_tensors="pt")

train_path,validation_path,test_path = "./data/train_data.csv","./data/validation_data.csv","./data/test_data.csv"

labels_of_bullying = {'age' : 0, 'ethnicity' : 1, 'gender' : 2, 'not_cyberbullying' : 3, 'other_cyberbullying' : 4, 'religion' : 5}
keys_of_bullying = list(labels_of_bullying.keys())
datasets = {"train":train_path,"validation":validation_path,"test":test_path}
dataset = load_dataset("csv",data_files = datasets,cache_dir="./data")
dataset = dataset.rename_column(original_column_name="cyberbullying_type", new_column_name="labels")

train_dataset_label = ClassLabel(names = keys_of_bullying)
validation_dataset_label = ClassLabel(names = keys_of_bullying)
test_dataset_label = ClassLabel(names = keys_of_bullying)

dataset["train"] = dataset["train"].class_encode_column("labels",ClassLabel)
dataset["validation"] = dataset["validation"].class_encode_column("labels",ClassLabel)
dataset["test"] = dataset["test"].class_encode_column("labels",ClassLabel)

dataset = x: {"tweet_text": lowerWords(x["tweet_text"])})
dataset = x: tokenize_sample(x))

I’m adopting the “microsoft/MiniLM-L12-H384-uncased” checkpoint, and I’ve done a little change to the head of the model:

checkpoint = "microsoft/MiniLM-L12-H384-uncased"
tokenizer,model = AutoTokenizer.from_pretrained(checkpoint),AutoModelForSequenceClassification.from_pretrained(checkpoint)

model.classifier = nn.Sequential(
    nn.Softmax(dim = -1)

I’m also attaching the arguments of the trainer class:

data_collator = DataCollatorWithPadding(tokenizer, return_tensors="pt")
train_args = TrainingArguments(
    output_dir = "./trainingResults",
    num_train_epochs = 2,
    learning_rate= 5e-5,
    per_device_train_batch_size = 32, per_device_eval_batch_size = 32
trainer = Trainer(
    data_collator = data_collator

Please, help! I’m really struggling on it :pray:t3:
Thanks for any help in advance!

Can you please post the full error trace?

Traceback (most recent call last):
  File "/Users/alessio/Desktop/LPT/codici_progetti/2/cyberbullying classifier/", line 117, in <module>
  File "/Users/alessio/opt/anaconda3/envs/LPT/lib/python3.8/site-packages/transformers/", line 1317, in train
    return inner_training_loop(
  File "/Users/alessio/opt/anaconda3/envs/LPT/lib/python3.8/site-packages/transformers/", line 1554, in _inner_training_loop
    tr_loss_step = self.training_step(model, inputs)
  File "/Users/alessio/opt/anaconda3/envs/LPT/lib/python3.8/site-packages/transformers/", line 2183, in training_step
    loss = self.compute_loss(model, inputs)
  File "/Users/alessio/opt/anaconda3/envs/LPT/lib/python3.8/site-packages/transformers/", line 2215, in compute_loss
    outputs = model(**inputs)
  File "/Users/alessio/opt/anaconda3/envs/LPT/lib/python3.8/site-packages/torch/nn/modules/", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "/Users/alessio/opt/anaconda3/envs/LPT/lib/python3.8/site-packages/transformers/models/bert/", line 1554, in forward
    outputs = self.bert(
  File "/Users/alessio/opt/anaconda3/envs/LPT/lib/python3.8/site-packages/torch/nn/modules/", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "/Users/alessio/opt/anaconda3/envs/LPT/lib/python3.8/site-packages/transformers/models/bert/", line 971, in forward
    batch_size, seq_length = input_shape
ValueError: too many values to unpack (expected 2)