Multi-label token classification

šŸ¤—Transformers
21

Hi @murdockthedude. Iā€™m using some sensitive (biomedical) data and my use case is actually a little more complicated than ā€˜justā€™ multi-label NER, so Iā€™d have to make up some dummy data and simplify my notebook a bit. Let me think more about that and how to make a shareable notebook.

In the mean time, the answer to your first two questions.

  1. The customer trainer is all thatā€™s needed, although you probably also want to implement a special compute_metrics function and use that when you instantiate the trainer so you can do early stopping.
  2. In my case I used BertForTokenClassification but using AutoModelForTokenClassification should be fine, I think.
1 Like
22

Hey @drussellmrichie, totally understand, thank you. Iā€™ll try to get a small notebook working too to see if I can tape this all together.

One question I have: Assuming I implement the custom trainer approach above, at inference time for multi label token classification, do you just take the individual output logits and run them through a sigmoid activation to get your final per-label-per-token probabilistic values (as opposed to single label which runs them through a softmax)? Or is it something more complex than that?

Thanks again!

23

@murdockthedude . I donā€™t think you even need to bother with sigmoid ā€“ you can just pass the logits through the sign function as in lambda x: 1 if x > 0 else 0. There are efficient TF and pytorch functions for that.

1 Like
24

sorry for the slow reply, this is great. Iā€™m working on getting things functioning but this approach seems promising thus far. Will update backā€¦

25

Hey, sorry for opening this old thread again, this looks pretty much exactly like what I am looking for at the moment. But how did you actually succeed in passing the one-hot encoded labels through to the Trainer?

Whenever I try this I am getting errors thrown by DataCollatorForTokenClassification that it expects the labels to be integers.

26

Can any one pls post your compute metrics function.

Help much appreciated.

27

@BunnyNoBugs @murdockthedude
I am working on morphological analysis problem where each token has multiple labels. Can you share sample notebook / working example so that I can understand and experiment with my problem ?

28

Can you share your code / sample working example?

29

Unfortunately I canā€™t. The code is in my old workplaceā€™s private laptop and git repo. :frowning:

However, we published a paper based on this work here: https://academic.oup.com/jamia/advance-article-abstract/doi/10.1093/jamia/ocad046/7099518

You might try emailing some of my coauthors, like Victor Ruiz. He might be willing to help you.

1 Like
30

You can take a look here. The code is a bit messy since it was done under time pressure in the end, but it gets the multi-token classification done, and we achieved good results with it.

1 Like