Combine BertForSequenceClassificaion with Additional Features

Hey,
I’m using BertForSequenceClassification + Pytorch Lightning-Flash for a text classification task. I want to add additional features besides the text (e.g. categorical features). From what I understand, I need to override BertForSequenceClassification “forward” method and change the final classification layer (at least) to include the CLS vector + features vector. However, I didn’t understand how I adapt the data loading procedure to this task - the text part is represented as input ids, and the rest supposed to be represented differently. Is there a simple way to combine text+features for Bert classification task? Thank you!

See this response where I explain how to modify BERT to add additional POS (part-of-speech) features to tokens to perform named-entity recognition.