How to get predictions after evaluation phase in a callback

Gladiator · April 25, 2022, 6:08am

I want to log prediction distribution after each evaluation phase to Weights & Biases. I thought of adding a callback to do this, but couldn’t find anything which allows me to access predictions in on_evaluate method to do so. Is there any way to achieve this or am I missing something ?

BramVanroy · April 25, 2022, 6:35am

If you want to get the predictions back, you should use Trainer.predict().

github.com

huggingface/transformers/blob/72728be3dbca26c70dddc8b724eb2c8d901e97dc/src/transformers/trainer.py#L2343-L2350

      
        
            def predict(
                self, test_dataset: Dataset, ignore_keys: Optional[List[str]] = None, metric_key_prefix: str = "test"
            ) -> PredictionOutput:
                """
                Run prediction and returns predictions and potential metrics.
            
            
    Depending on the dataset and your use case, your test dataset may contain labels. In that case, this method
                will also return metrics, like in `evaluate()`.

Topic		Replies	Views
How to log predictions from evaluation set after each Trainer validation to wandb? Beginners	2	920	March 14, 2024
Trainer.evaluate() vs trainer.predict() 🤗Transformers	6	33158	July 10, 2024
Callbacks for logging results to GPT2 🤗Transformers	1	442	February 16, 2022
Logging training accuracy using Trainer class 🤗Transformers	8	10298	December 2, 2021
Accessing model from a callback to predict between epochs Intermediate	1	1404	August 17, 2023

How to get predictions after evaluation phase in a callback

Related topics