Looking for hyperparameter tuning advices

Beginners
1

I’m finetuning a language model (vesteinn/ScandiBERT) on a multiclass text classification (pos, neg, neu) task and I wish to run a hyperparameter search to find the optimal hyperparameters. As I’m starting to read up on it I realize that HP tuning is a wild west with no consensus on which HP to tune on and which ranges to specify for these. I’m not an NLP expert so I might be missing some conceptual knowledge which is why I’m hoping that one of you would take a look at how I initiate my HP tuning using Optuna:
I looks like this:

!pip install -q optuna

def model_init():
    return model #model = AutoModelForSequenceClassification.from_pretrained(model, num_labels=3) (defined earlier)

training_args = TrainingArguments(output_dir=model_name, 
                                  evaluation_strategy = "epoch",
                                  save_strategy = "epoch", 
                                  num_train_epochs = epochs, 
                                  per_device_train_batch_size = batch_size,
                                  per_device_eval_batch_size = batch_size,
                                  learning_rate = learning_rate,
                                  weight_decay=0.01,
                                  load_best_model_at_end=True,
                                  report_to="wandb")

trainer = Trainer(
    model_init=model_init,
    args=training_args,
    train_dataset=tokenized_datasets["train"],
    eval_dataset=tokenized_datasets["test"],
    tokenizer=tokenizer,
    compute_metrics=compute_metrics
)

def my_hp_space(trial):
    return {
        "learning_rate": trial.suggest_float("learning_rate", 1e-6, 1e-4, log=True),
        "num_train_epochs": trial.suggest_int("num_train_epochs", 5, 15),
        #"seed": trial.suggest_int("seed", 1, 40),
        "per_device_train_batch_size": trial.suggest_categorical("per_device_train_batch_size", [4, 8, 16, 32, 64]),
        "weight_decay": trial.suggest_loguniform('weight_decay', 1e-4, 1e-2)
    }

import optuna
from optuna.samplers import TPESampler
sampler = optuna.samplers.TPESampler()
pruner = optuna.pruners.SuccessiveHalvingPruner()

best_run = trainer.hyperparameter_search(
    n_trials=100, 
    direction="maximize", 
    hp_space=my_hp_space, 
    backend = "optuna",
    sampler = sampler,
    pruner = pruner
    )

for n, v in best_run.hyperparameters.items():
    setattr(trainer.args, n, v) # for running the experiment with the best hyperparameters from the hyperparameters search

trainer.train() # argument trial can be used for hyperparameter search

My code does run but I’m uncertain of whether I’m missing anything important in my value definitions. Especially since the trial gets pruned from run 3 - 99. Here is a link to my notebook with all the code.Google Colab