ELMO Character encoder layer

Beginners
1

I think my model would benefit from the inclusion of character encoding similar to ELMO, i.e. as discussed in CharacterBERT (https://www.aclweb.org/anthology/2020.coling-main.609.pdf ) since I have loads of acronyms so I think added characters will help.

Is there a layer in the huggingface library I could use to replicate this configuration? I don’t see one, nor a hugginface ELMO but I could well have missed it. Otherwise I’ll try and use AllenNLP.

2

hey @david-waterworth, looking at the paper’s github repo (link), it seems that they just changed the embedding layer of the BERT implementation in transformers to accept character embeddings: character-bert/character_bert.py at 0519a6e22a2912bf0fdbbd49bd562cc2e5410bc7 · helboukkouri/character-bert · GitHub

you could try the same approach and see if that works!