Is Eval and Validation same in Trainer API?

Beginners
1

Hi, I am a bit confused if the eval dataset parameter is used during the training.

#Trainer itself.
trainer = Trainer(
    model,
    args,
    train_dataset=tokenized_datasets_train,
    eval_dataset=tokenized_datasets_val,
    tokenizer=tokenizer,
    compute_metrics=compute_metrics,
    data_collator = data_collator_
)

is the eval_dataset only used when we do trainer.evaluate() ?

2

Yes, it’s the default dataset used for that method (which will be used if you pass an eval_strategy to evaluate every epoch or n steps).

1 Like
3

Okay perfect, then i will not put my test set over there Thanks! @sgugger

Although i want to get the f1 score of my model on the test set, do you know if there is a metric api from hugginface i could use?

If so could you please link me a small script?

4

You can look at any of the examples or the course section on the Trainer on how to du this using a compute_metrics function.

1 Like
5

Huge thanks again!

Have a nice day

1 Like