Hi there, I am wondering, what would be the optimal solution to also report and log perplexity during the training loop via the Trainer API. How would the corresponding compute_metrics function look like. So far I tried without success since I am not 100% sure how the EvalPrediction output would lo…

Log Perplexity using Trainer

alihan October 9, 2021, 2:53pm 3

Hi, I am also looking very forward to see how it is possible to utilize perplexity during execution oc compute_metrixs() while finetuning pretrained models for MLM or CLM

1 Like

Topic		Replies	Views
Useful compute_metrics functions for perplexity 🤗Transformers	0	633	September 29, 2022
Compute Perplexity using compute_metrics in SFTTrainer Intermediate	1	944	January 22, 2025
Using perplexity as metric during training Beginners	0	1667	June 7, 2023
Custom model for Trainer 🤗Transformers	1	386	July 8, 2023
Trainer API to log both Training and Validation Metrics 🤗Transformers	2	1683	July 1, 2021

Log Perplexity using Trainer

Related topics