Why Is My Fine-Tuned RoBERTa (Text classification) Model Only Predicting One Category/Class?

John6666 · March 24, 2025, 11:27am

the problem isnt that the LoRa values arent being saved and integrated correctly?

It seems that’s the case…
Usually, LoRA files are full of data.

But if a file is not created, that’s one thing, but what does it mean if there is a file but no content…?

Hmm…

github.com/huggingface/peft

Incorrect Saving Peft Models using HuggingFace Trainer

opened 12:30PM - 16 Feb 23 UTC

closed 03:03PM - 13 Apr 23 UTC

agemagician

solved

Hello, Thanks a lot for the great project. I am fine-tuning Flan-T5-XXL us…ing HuggingFace Seq2SeqTrainer and hyperparameter_search. However, the trainer doesn't store Peft models correctly because it is not a "PreTrainedModel" type. It stores the whole PyTorch model, including the Flan-T5-XXL, which is around 42 GB. I have dug into the code, and I made a hacky solution inside "trainer.py" for now: ``` def _save(self, output_dir: Optional[str] = None, state_dict=None): # If we are executing this function, we are the process zero, so we don't check for that. output_dir = output_dir if output_dir is not None else self.args.output_dir os.makedirs(output_dir, exist_ok=True) logger.info(f"Saving model checkpoint to {output_dir}") from peft.peft_model import PeftModelForSeq2SeqLM if isinstance(self.model, PeftModelForSeq2SeqLM): self.model.save_pretrained(output_dir, state_dict=state_dict) # Save a trained model and configuration using `save_pretrained()`. # They can then be reloaded using `from_pretrained()` elif not isinstance(self.model, PreTrainedModel): if isinstance(unwrap_model(self.model), PreTrainedModel): if state_dict is None: state_dict = self.model.state_dict() unwrap_model(self.model).save_pretrained(output_dir, state_dict=state_dict) else: logger.info("Trainer.model is not a `PreTrainedModel`, only saving its state dict.") if state_dict is None: state_dict = self.model.state_dict() torch.save(state_dict, os.path.join(output_dir, WEIGHTS_NAME)) else: self.model.save_pretrained(output_dir, state_dict=state_dict) if self.tokenizer is not None: self.tokenizer.save_pretrained(output_dir) # Good practice: save your training arguments together with the trained model torch.save(self.args, os.path.join(output_dir, TRAINING_ARGS_NAME)) ``` Do you have a better solution for saving the "Peft models" correctly using HuggingFace Seq2SeqTrainer and hyperparameter_search ?

Topic		Replies	Views
Fine-tuned pre-trained Roberta model on different labels 🤗Transformers	0	629	April 7, 2022
Fine-tuning Bert/Roberta for multi-label sentiment analysis Beginners	0	1568	November 8, 2021
Does anyone else observer RoBERTa fine-tuning instability? 🤗Transformers	8	3083	April 20, 2023
Is there a mechanism or feature in RoBERTa limiting the logits difference of its predictions? Models	0	312	August 6, 2021
Despite Low Training Loss, Model Can't Predict Training Set Correctly Beginners	0	847	October 26, 2022

Why Is My Fine-Tuned RoBERTa (Text classification) Model Only Predicting One Category/Class?

Related topics