I am currently working on a project to fine-tune BERT models on a multi-class classification task with the goal to classify job ads into some broader categories like “doctors” or “sales” via
AutoModelForSequenceClassification (which works quite well ). Now I am wondering, whether it would be possible to add a second classification head “next” to the first one (not in sequence) to classify the minimum educational level that is required for the job. I imagine that each head is directly connected to the pooler output an then makes a prediction independent of the other’s prediction. I think my use-case is slightly different than a multi-label classification since both labels describe different aspects of the job ad.
Similar to this CV example: What is a multi-headed model? And what exactly is a ‘head’ in a model?
I hope it’s not total nonsense that I’m asking here