Multi-label token classification: "-100" special label

🀗Transformers
1

Hello!
I am implementing a HF-based model augmented with native Pytorch-code to classify tokens (not the document!) into one or more classes.

Now, based on single labels and using the AutoTokenizer and after aligning subwords with their labels, I get the following output, where the labels -100 correspond to the [CLS] and [SEP] special tokens as well as subwords starting with ## (not seen below). Apparently, the -100 labels tell the model to ignore the token when calculating loss.

{‘input_ids’: [2,
4759,
4683,
3],
‘token_type_ids’: [0,
0,
0,
0],
‘attention_mask’: [1,
1,
1,
1],
‘labels’: [-100,
3,
4,
-100]
}

However, with multi-label classification, my labels are lists of one-hot encodings as required for calculating nn.CEWithLogitsLoss():

‘labels’: [XXX,
[1, 0, 1],
[1, 1, 0],
[1, 0, 0],
XXX]]

What should I put here for XXX instead of the “-100” special label to tell the model to ignore special tokens as well as subword tokens?

1 Like
2

Perhaps you can try to create a vector consisting of -100 values?
So instead of XXX you can try [-100 -100 -100]

Let me know if you get some results.