Transformers and Hyperparameter search using Optuna

[Petrina]

@The-Fanta

Steps with transformers are not epochs. In my understanding, 1 step = 1 batch
So if you have a train set with, say, 1000 samples, and you train in batches of 64 samples, then it will take you 16 steps (16 batches) to push the whole train set through training once, i.e. to do 1 epoch. In this example, 1 epoch = 16 steps. If you do the math for your script, you should find that, in your case, 500 steps = 1 epoch.

Ok yes, this clarifies the issue. The dataset I had would result in 1 epoch=500 steps, and because the default number of steps to log when logging_strategy="steps" was 500, I thought that this logs anyways every 500 steps. Thank you very much for this!

@The-Fanta

Not sure I understand which .json file you are referring to. What is the complete path?
Log fiiles I have are organized like this

Yes, you understood correctly, thank you for the clarifications on this too.

Also, for the trainer.state.log_history, just to underline that it contains only the log of the last trial. So, as you stated, I understand there is not an obvious way to get the logs for the wholes study.

Thank you very much once again for the help!