Hi! I have a use case where I would like evaluation to happen at the beginning of training (before a training step has been taken) in addition to every n steps. I can easily get the latter using evaluation_strategy and eval_steps, but not sure how to get the former.
There is an argument, logging_first_step that sounds like it should do exactly what I need:
logging_first_step (bool, optional, defaults to False) — Whether to log and evaluate the first global_step or not.
But providing this argument does not lead to evaluation at the beginning of training like I would expect (tested with the run_summarization.py script). Does anyone have an idea how to get this behaviour from the HF Trainer?
You can just add a line trainer.evaluate() before the call to train.
Thanks for the quick response! I guess this solution works, but the results do not end up in the log_history field of the training_state.json file, which is how I am tracking performance over time. So I guess two questions:
- Is there a better way to track evaluation metrics over time (using the provided example scripts like
run_summarization.py) than the log_history field of the training_state.json?
- Might adding this functionality (evaluating during training before a training step has been taken) be a good idea? Seems like a pretty ubiquitous use case to me as you may want to plot performance over time and knowing the performance of the model (either randomly initialized or pre-trained) before the first train step is useful. Would be happy to take a crack at this if I could get some advice as to where to implement it.
1 Like