Hi @margootje123 ,
yes you can for instance first embed the text using BERT (get the last_hidden_state as this serves as a good text representation), and then concatenate that with additional custom features.
Regarding POS tags, you can use a one-hot encoding or a new nn.Embedding layer and provide those to the classification head.