Training with class weights

Hi everyone,

I am new to the huggingface library and I am fine-tuning a bert model for an imbalanced dataset with custom Trainer class and class weights:

from torch import nn
from transformers import Trainer

class CustomTrainer(Trainer):
    def compute_loss(self, model, inputs, return_outputs=False):
        labels = inputs.pop("labels")
        # forward pass
        outputs = model(**inputs)
        logits = outputs.get("logits")
        # compute custom loss (suppose one has 2 labels with different weights)
        loss_fct = nn.CrossEntropyLoss(weight=torch.tensor([8.0, 1.0], device=model.device))
        loss = loss_fct(logits.view(-1, self.model.config.num_labels), labels.view(-1))
        return (loss, outputs) if return_outputs else loss

However, I got same results while training with class weights [10.0, 1.0] and [9.0, 1.0]. Is that normal? Or is there something wrong with my initialization? (I am initializing the model with model_init option.)

Thanks for reading!

If your weights are similar to each other, I am not surprised that you get similar results.

Thanks for your reply!

The problem is the results are exactly the same for class weights like 8.0, 9.0 and 10.0.

Do you mean Hugging Face pipeline for text classification classes are exactly the same or that scores are exactly the same?

The resulting scores are exactly the same. I am working on a classification task so I looked at the classification report, and they are exactly the same for a various of class weights.

Maybe I did not initialize the model properly? I am using a jupyter notebook and I simply rerun the cells after changing the weights.

I would check if the CustomTrainer class is really used.