This thread should be useful. Using hyperparameter-search in Trainer
I’d break it into two steps. The first would be to train the model on the first dataset (with or without the hp search). The second would be do a hp search using the model from the last one. You can call .from_pretrained("trained_model_dir") to use the model trained in the first step.