Adding categorical and numerical values for bert training

🤗Transformers
#1

Hello, if we have a dataset with text, numerical and categorical values to be used for text classification. What options we have to use these additional (numerical and categorical) columns for classification? Here are the options which I can think of

option 1: Combine categorical values with text using [SEP]
option 2 : concatenate numerical/categorical data to the CLS embedding and pass it to linear layer

Any help on this is or tutorials is greatly appreciated.

Thanks

4 Likes
#2

great question and I have been looking for answers on the same for my academic project. I hope we will get some help here

1 Like