I have quantized a model using BitsAndBytes, and I want to evaluate this model on some benchmark tasks. For evaluation on other models, I use trainer.evaluate() to obtain the metrics. This is however different for quantized models. I use the following code: ... trainer = Trainer( model=model_q…

Evaluating with quantized model using Trainer

daminho August 10, 2024, 11:14am 2

You can try to use evaluate library from huggingface instead

Topic		Replies	Views
Using Trainer class + 4/8 bit quantised model for prediction 🤗Transformers	0	243	April 24, 2024
Resolving "Cannot Perform Fine-Tuning on Purely Quantized Models" Error in Falcon Model Training? 🤗Transformers	4	9239	May 9, 2025
Evaluation on Quantized model yields identical results across bit-precisions Beginners	1	112	June 20, 2024
Trainer in PEFT doesn't report evaluation metrics 🤗Transformers	4	535	June 17, 2025
Peft Model For SequenceClassification failing _is_peft_model Beginners	0	923	February 11, 2024