Add per-word embedding from outer source to Bert embedding layer

[orko]

Hi
I have an outer source with per-word embedding,
I want to add it as an additional later for the bert embddings:
Such that
custom_embedding_layer = (word_embeddings)+(position_embeddings)+(outersource_embeddings)

I saw the answer in:

How to use additional input features for NER?

Actually no, because the pre-trained tokenizer only knows tokens, not tokens + POS tags. A better way to do this would be to create an additional input to the model (besides input_ids and token_type_ids) called pos_tag_ids, for which you can add an additional embedding layer (nn.Embedding). In that way, you can sum the embeddings of the tokens, token types and the POS tags. Let’s illustrate this for a pre-trained BERT model: We first have to modify the BertEmbeddings class. In short, we’ll add…

But I would rather create a wrapper class rather then change the source code of BertEmbeddings itself
How can I do it?
Thanks