Is there an implementation of the Psuedo Log Likelihood for bidirectional language models (i.e. Salazar et al. Masked Language Model Scoring) in transformers? The github repo in the linked paper uses transformers 3.3 and I’ve been unable to get it to work for 4.5.
1 Like
what kind of problems are you running into? presumably it’s due to a change in the API, so sharing what steps you’re taking and the error messages will help with the debugging 
Do you mean with the GitHub - awslabs/mlm-scoring: Python library & examples for Masked Language Model Scoring (ACL 2020) implementation? I’m assuming there’s not much I can do to try and get a 3rd party library which is specifically designed for transformers 3.3 to work with a transformer / tokeniser trained with version 4.5. Specifically my tokeniser is in the new single json file format and as far as I can see the 3.3 library is trying to load from the legacy format. The main issue is the setup.py of the mlm-scoring library requires ==3.3 rather than >=3.3 so installing it downgrades. I suppose I could try removing the version requirement and see what happens.
But ideally the metric would be available via a library which is more up to date. I’ll probably code it up myself altouhg it wont be overly efficient, you need to compute the MLM objective masking each token in order and then sum the log likelyhoods to compute PLL for a single sentance.
yes, i was wondering whether you could adapt their code to match the current transformers API.
can you point me to the line of code where this is done? i might be able to suggest a workaround this way 
Was this implemented in transformers or was there some solution for this? I am attempting to use this scoring technique in my project. Could you please share some details?
Hi, do you have some solutions? Could you share some experience?