I would like to create better search functionality for a domain specific language (DSL).
For this, I’m trying to finetune an encoder on the masked language modeling (MLM) objective as described here: Fine-tuning a masked language model - Hugging Face NLP Course. Similar to this question, except I’m leaving the tokenizer as is: Domain adaptation of Language Model and Tokenizer
I’ve tried two base-models: all-MiniLM-L6-v2 and roberta-base and finetuned them on about 30k samples from the DSL with 15% masking.
To evaluate the results, I use some semantically equivalent/similar pairs and see how well I can retrieve one from a line-up by encoding the other. Similar to InfoNCE loss, but with rankings instead of probabilities.
With both models I’ve tried, I find that as the MLM loss decreases (on eval, not just train), the actual metrics I care about get worse. In other words, roberta-base is better at searching for similar DSL pairs than a model that’s actually seen the DSL.
I realize I should also train on an InfoNCE style loss at some point (e.g. Losses — Sentence Transformers documentation), but there’s not enough data for that at the moment.
Shouldn’t there already be some improvement from training on MLM?
Would appreciate any thoughts/pointers/references!