How to evaluate before first training step?

🤗Transformers
1

Hi! I have a use case where I would like evaluation to happen at the beginning of training (before a training step has been taken) in addition to every n steps. I can easily get the latter using evaluation_strategy and eval_steps, but not sure how to get the former.

There is an argument, logging_first_step that sounds like it should do exactly what I need:

logging_first_step (bool, optional, defaults to False) — Whether to log and evaluate the first global_step or not.

But providing this argument does not lead to evaluation at the beginning of training like I would expect (tested with the run_summarization.py script). Does anyone have an idea how to get this behaviour from the HF Trainer?

1 Like
2

You can just add a line trainer.evaluate() before the call to train.

3

Thanks for the quick response! I guess this solution works, but the results do not end up in the log_history field of the training_state.json file, which is how I am tracking performance over time. So I guess two questions:

  • Is there a better way to track evaluation metrics over time (using the provided example scripts like run_summarization.py) than the log_history field of the training_state.json?
  • Might adding this functionality (evaluating during training before a training step has been taken) be a good idea? Seems like a pretty ubiquitous use case to me as you may want to plot performance over time and knowing the performance of the model (either randomly initialized or pre-trained) before the first train step is useful. Would be happy to take a crack at this if I could get some advice as to where to implement it.
3 Likes
4

A alternative approach, which may or may not solve the problem is to use a callback, like so:

class EvaluateFirstStepCallback(TrainerCallback):
    def on_step_end(self, args, state, control, **kwargs):
        if state.global_step == 1:
            control.should_evaluate = True

trainer.add_callback(EvaluateFirstStepCallback())

It does seem like it might be nice to have as a built-in TrainingArguments, given it mirrors logging_first_step pretty closely.

1 Like