Masked Language Model Scoring

Is there an implementation of the Psuedo Log Likelihood for bidirectional language models (i.e. Salazar et al. Masked Language Model Scoring) in transformers? The github repo in the linked paper uses transformers 3.3 and I’ve been unable to get it to work for 4.5.

what kind of problems are you running into? presumably it’s due to a change in the API, so sharing what steps you’re taking and the error messages will help with the debugging :slight_smile:

Do you mean with the GitHub - awslabs/mlm-scoring: Python library & examples for Masked Language Model Scoring (ACL 2020) implementation? I’m assuming there’s not much I can do to try and get a 3rd party library which is specifically designed for transformers 3.3 to work with a transformer / tokeniser trained with version 4.5. Specifically my tokeniser is in the new single json file format and as far as I can see the 3.3 library is trying to load from the legacy format. The main issue is the of the mlm-scoring library requires ==3.3 rather than >=3.3 so installing it downgrades. I suppose I could try removing the version requirement and see what happens.

But ideally the metric would be available via a library which is more up to date. I’ll probably code it up myself altouhg it wont be overly efficient, you need to compute the MLM objective masking each token in order and then sum the log likelyhoods to compute PLL for a single sentance.

yes, i was wondering whether you could adapt their code to match the current transformers API.

can you point me to the line of code where this is done? i might be able to suggest a workaround this way :slight_smile: