How to use the model from the chapter "Fine-tuning a model with the Trainer API"

Course
1

Hey everyone :blush:

I’m a bit stuck on the following problem and wanted to ask you if you might have a solution to the following problem?

I have done the training from the chapter “Fine-tuning a model with the Trainer API”:

import numpy as np
from transformers import AutoTokenizer, DataCollatorWithPadding
from transformers import AutoModelForSequenceClassification
from transformers import TrainingArguments
from datasets import load_dataset

raw_datasets = load_dataset("glue", "mrpc")
checkpoint = "bert-base-uncased"
tokenizer = AutoTokenizer.from_pretrained(checkpoint)

def tokenize_function(example):
    return tokenizer(example["sentence1"], example["sentence2"], truncation=True)

tokenized_datasets = raw_datasets.map(tokenize_function, batched=True)
data_collator = DataCollatorWithPadding(tokenizer=tokenizer)

training_args = TrainingArguments(
    output_dir="test-trainer",
    num_train_epochs=2,
    logging_steps=100,
    eval_steps=100,
    evaluation_strategy="steps"
)

model = AutoModelForSequenceClassification.from_pretrained(checkpoint, num_labels=2)

import evaluate
def compute_metrics(eval_preds):
    metric = evaluate.load("glue", "mrpc")
    logits, labels = eval_preds
    predictions = np.argmax(logits, axis=-1)
    return metric.compute(predictions = predictions, references = labels)

from transformers import Trainer
trainer = Trainer(
    model,
    training_args,
    train_dataset = tokenized_datasets["train"],
    eval_dataset = tokenized_datasets["validation"],
    data_collator = data_collator,
    tokenizer = tokenizer,
    compute_metrics = compute_metrics,
)

trainer.train()
trainer.save_model("test-trainer/final-model")

Now I want to use the model and get different results, once with the definition of the pipeline and once without.

Option 1 gives:

sentence1 = "PCCW's chief operating officer, Mike Butcher, and Alex Arena, the chief financial officer, will report directly to Mr So."
sentence2 = "Current Chief Operating Officer Mike Butcher and Group Chief Financial Officer Alex Arena will report to So."

fine_tuned_model = AutoModelForSequenceClassification.from_pretrained("test-trainer/final-model")
tokenizer = AutoTokenizer.from_pretrained("test-trainer/final-model")

inputs = tokenizer(sentence1, sentence2, return_tensors="pt", padding=True, truncation=True)
outputs = fine_tuned_model(**inputs)
probabilities = torch.softmax(outputs.logits, dim=1)
print(probabilities)

=> tensor([[0.0104, 0.9896]], grad_fn=)

and option 2 gives:

from transformers import pipeline

pipe = pipeline(
    task="text-classification",
    model=fine_tuned_model,
    tokenizer=tokenizer,
    framework="pt",
    padding=True,
    truncation=True
)

result = pipe((sentence1, sentence2))
print(result)

=> {‘label’: ‘LABEL_0’, ‘score’: 0.9904301762580872}

The result should be Label 1, as in option 1. What am I doing wrong in defining the pipeline?

Many thanks in advance :hugs: