Why does fine-tuning require creating two Trainers?

🤗Transformers
1

I’m following the tutorial here:

which creates two separate Trainers, one for training, the other for evaluation:


trainer = Trainer(
    model=model, args=training_args, train_dataset=small_train_dataset, eval_dataset=small_eval_dataset
)

trainer.train()

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=small_train_dataset,
    eval_dataset=small_eval_dataset,
    compute_metrics=compute_metrics,
)
trainer.evaluate()

Three questions:

  1. Why are two Trainers required? What can we just create the Trainer once, call .train() then call .evaluate()?
  2. Why is the validation loss not computed during training?
  3. Why is a custom compute_metrics function required? I’m referring to:

metric = load_metric("accuracy")

def compute_metrics(eval_pred):
    logits, labels = eval_pred
    predictions = np.argmax(logits, axis=-1)
    return metric.compute(predictions=predictions, references=labels)