Using evaluate.evaluator on a PEFT model

Beginners
1

For an exercise I am experimenting with PEFT on GPT-2. One task is to evaluate the performance. For this I chose evaluate.evaluator, which worked fine on plain GPT-2. However, when trying to evaluate the fine-tuned model like this

task_evaluator = evaluator('text-classification')
eval_results = task_evaluator.compute(
    model_or_pipeline=lora_model,
    tokenizer=tokenizer,
    data=ds_test,
    input_column=dataset_textfield_name,
    metric=evaluate.combine(['accuracy', 'f1']),
    label_mapping=label2id
)

I got the following error:

AttributeError Traceback (most recent call last)
File ~/.local/lib/python3.10/site-packages/peft/peft_model.py:529, in PeftModel.getattr(self, name)
528 try:
→ 529 return super().getattr(name) # defer to nn.Module’s logic
530 except AttributeError:

File /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1614, in Module.getattr(self, name)
1613 return modules[name]
→ 1614 raise AttributeError(“‘{}’ object has no attribute ‘{}’”.format(
1615 type(self).name, name))

AttributeError: ‘PeftModelForSequenceClassification’ object has no attribute ‘task’

During handling of the above exception, another exception occurred:

AttributeError Traceback (most recent call last)
File ~/.local/lib/python3.10/site-packages/peft/tuners/lora/model.py:273, in LoraModel.getattr(self, name)
272 try:
→ 273 return super().getattr(name) # defer to nn.Module’s logic
274 except AttributeError:

File /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1614, in Module.getattr(self, name)
1613 return modules[name]
→ 1614 raise AttributeError(“‘{}’ object has no attribute ‘{}’”.format(
1615 type(self).name, name))

AttributeError: ‘LoraModel’ object has no attribute ‘task’

During handling of the above exception, another exception occurred:

AttributeError Traceback (most recent call last)
Cell In[19], line 1
----> 1 eval_results = task_evaluator.compute(
2 model_or_pipeline=lora_model,
3 tokenizer=tokenizer,
4 data=ds_test,
5 input_column=dataset_textfield_name,
6 metric=evaluate.combine([‘accuracy’, ‘f1’]),
7 label_mapping=label2id
8 )
9 print(eval_results)

File ~/.local/lib/python3.10/site-packages/evaluate/evaluator/text_classification.py:134, in TextClassificationEvaluator.compute(self, model_or_pipeline, data, subset, split, metric, tokenizer, feature_extractor, strategy, confidence_level, n_resamples, device, random_state, input_column, second_input_column, label_column, label_mapping)
130 data = self.load_data(data=data, subset=subset, split=split)
131 metric_inputs, pipe_inputs = self.prepare_data(
132 data=data, input_column=input_column, second_input_column=second_input_column, label_column=label_column
133 )
→ 134 pipe = self.prepare_pipeline(
135 model_or_pipeline=model_or_pipeline,
136 tokenizer=tokenizer,
137 feature_extractor=feature_extractor,
138 device=device,
139 )
140 metric = self.prepare_metric(metric)
142 # Compute predictions

File ~/.local/lib/python3.10/site-packages/evaluate/evaluator/base.py:474, in Evaluator.prepare_pipeline(self, model_or_pipeline, tokenizer, feature_extractor, device)
472 if tokenizer is not None and feature_extractor is not None:
473 logger.warning(“Ignoring the value of the preprocessor argument (tokenizer or feature_extractor).”)
→ 474 if (pipe.task != self.task) and not (self.task == “translation” and pipe.task.startswith(“translation”)):
475 raise ValueError(
476 f"Incompatible model_or_pipeline. Please specify model_or_pipeline compatible with the {self.task} task."
477 )
478 return pipe

File ~/.local/lib/python3.10/site-packages/peft/peft_model.py:531, in PeftModel.getattr(self, name)
529 return super().getattr(name) # defer to nn.Module’s logic
530 except AttributeError:
→ 531 return getattr(self.base_model, name)

File ~/.local/lib/python3.10/site-packages/peft/tuners/lora/model.py:275, in LoraModel.getattr(self, name)
273 return super().getattr(name) # defer to nn.Module’s logic
274 except AttributeError:
→ 275 return getattr(self.model, name)

File /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1614, in Module.getattr(self, name)
1612 if name in modules:
1613 return modules[name]
→ 1614 raise AttributeError(“‘{}’ object has no attribute ‘{}’”.format(
1615 type(self).name, name))

AttributeError: ‘GPT2ForSequenceClassification’ object has no attribute ‘task’

What should I do to use evaluate.evaluator for fine-tuned models?

2

@countermode Could you find the error? Have the same here…