Rouge-L score in Trainer huggingface

🤗Transformers
1

I want to fine-tune t5-efficient-tiny model on a question-answering dataset. I want to use Cross-Entropy loss and ROUGE-L score as an evalution metric. Here is my code for Trainer:

# Define the TrainingArguments
training_args = TrainingArguments(
    output_dir="./qa-fine-tuned",
    per_device_train_batch_size=8,
    num_train_epochs=3,
    save_steps=5,
    save_total_limit=5,
    evaluation_strategy="steps",
    eval_steps=5,
    learning_rate=1e-4,  # Define the learning rate
    optim='adamw_torch',  # Define the optimizer. it is AdamW
    lr_scheduler_type="linear",
    load_best_model_at_end=True,
    include_inputs_for_metrics=True
)

rouge_metric = datasets.load_metric("rouge")

# Initialize Trainer
# To check the optimizer settings you should check the trainer.args
trainer = Trainer(
    model=model,
    args=training_args,
    data_collator=DataCollatorForSeq2Seq(
        tokenizer=tokenizer,
        model=model,
        padding=True,
        label_pad_token_id=tokenizer.pad_token_id,
    ),
    train_dataset=tokenized_train,
    eval_dataset=tokenized_validation,
    callbacks=[EarlyStoppingCallback(early_stopping_patience=3)],
    compute_metrics=rouge_metric
)

However, I am getting TypeError: 'list' object is not callable error in self.compute_metric. Do you have any idea what is the reason and how to solve it?

2

Hello !
You need to add a compute metrics func as you can find here : https://github.com/huggingface/transformers/blob/v4.33.2/examples/pytorch/summarization/run_summarization.py#L657