Using hyperparameter-search in Trainer

🤗Transformers
97

EDIT: Seems like indeed (mentioned in one of your comments above) just passing the backend to wandb instead of ray worked;i.e

   trainer.hyperparameter_search(
                        direction="maximize",
                        backend="wandb",
                        n_trials=4,
                        keep_checkpoints_num=1,
                        scheduler=get_scheduler())

I tested the provided example and managed to run it successfully.

However I do not understand completely how you would integrate W&B Sweeps with a PopulationBasedTraining() for example.

Could you shed some light on that please?

Thank you.

*** I did read this article : Weights & Biases, but in this case I just see how the configuration for the different hyperparameter strategies is defined, not how to integrate it with wandb.config() and wandb.agent() like in the example you posted.

To be more precise:

from ray import tune
from ray.tune.schedulers import PopulationBasedTraining



def get_scheduler():
    #Creating the PBT scheduler
    scheduler = PopulationBasedTraining(
        mode = "max",
        metric='eval_f1',
        perturbation_interval=2,
        hyperparam_mutations={
            "weight_decay": tune.uniform(0.0, 0.3),
            "learning_rate": tune.uniform(1e-5, 5e-5),
            "per_device_train_batch_size": tune.choice([8, 16, 24, 32, 48]),
            "num_train_epochs": tune.choice([5]),
            "warmup_steps": tune.choice(range(0, 500))
        }
    )
    return scheduler


...




    trainer.hyperparameter_search(
                        direction="maximize",
                        backend="ray",
                        n_trials=4,
                        keep_checkpoints_num=1,
                        scheduler=get_scheduler())

How should I integrate W&B for Sweeps analysis?