Options for feature addition

🤗Transformers
1

Hi there,

I have a question about adding additional features to a transformer model I was hoping to get some advice on. I have a BERT model that classifies text into various categories. I would like to include additional numeric and categorical features into the model to improve performance. There are 3 ways I have in mind to do this, with the last one being where I was hoping for some advice.

  1. Use the transformer as a feature extractor and input to another algorithm alongside the additional features
  2. Add additional features as text: some text [SEP] feature_1 [SEP] feature_2
  3. Add additional features to the classification head while still fine tuning the model

Option 3 is the one that I’m unclear how to implement but it feels the most promising given the benefit of fine tuning the transformer based on the classification head.

Hoping somebody has encountered this before and can offer some advice. Thank you!

1 Like