How to include token-level prior into bert?

🤗Transformers
1

Is there a way to explicitly tell BERT to pay more attention to certain tokens given some prior information I have about the data?