Understanding how Ray "objective" works with the trainer

BramVanroy · April 1, 2022, 7:31am

After some further digging, I can confirm that my first post is correct. If you don’t specify a custom function for compute_objective, the sum of the values of compute_metrics is used. If you do not specify your own compute_metrics, the loss will be used. Note that this is important! You often want to minimize your loss but maximize secondary metrics (like F1, correlation scores, etc.), which is a crucial parameter in hyperparamter_search(direction=["minimize", "maximize"]).

github.com

huggingface/transformers/blob/bfeff6cc6a086b953b5f542d369b119e96b841cf/src/transformers/trainer_utils.py#L152-L170

      
        
            def default_compute_objective(metrics: Dict[str, float]) -> float:
                """
                The default objective to maximize/minimize when doing an hyperparameter search. It is the evaluation loss if no
                metrics are provided to the [`Trainer`], the sum of all metrics otherwise.
            
            
    Args:
                    metrics (`Dict[str, float]`): The metrics returned by the evaluate method.
            
            
    Return:
                    `float`: The objective to minimize or maximize
                """
                metrics = copy.deepcopy(metrics)
                loss = metrics.pop("eval_loss", None)
                _ = metrics.pop("epoch", None)
                # Remove speed metrics
                speed_metrics = [m for m in metrics.keys() if m.endswith("_runtime") or m.endswith("_per_second")]
                for sm in speed_metrics:
                    _ = metrics.pop(sm, None)
                return loss if len(metrics) == 0 else sum(metrics.values())

Topic		Replies	Views
Using hyperparameter-search in Trainer 🤗Transformers	101	37859	July 2, 2024
There is always something going wrong with hyper parameter tuning 🤗Transformers	4	1963	September 1, 2021
Inconsistency in hyperparameter search results 🤗Transformers	2	626	April 13, 2022
How to tune optimizer hyperparameters with Trainer.hyperparameter_search 🤗Transformers	0	831	March 6, 2023
About Hyperparameter Search with Ray Tune 🤗Transformers	2	17	March 7, 2025

Understanding how Ray "objective" works with the trainer

Related topics