Pretraining RoBERTa (or another 'flavor') on permuted word orders

For the Community Week project I’d like to extend the work of the McGill researchers who, by inducing ‘order agnostic’ tokens via a ‘permuted’ pre-trained version of BERT, get high task scores on a wide range of benchmark tasks (GLUE, PAWS, etc.) due to the model learning to represent distributional priors rather than its ability to “discover the NLP pipeline”.

I’d like to replicate the authors’ work and perhaps extend it by looking at the ‘permuted model’s performance’ on other benchmarks / sets of NLP and ‘NLU’ tasks for which we have human-curated ‘gold standard’ performance measures.

To start, the authors pre-trained models on various permuted corpora that preserve sentence-level distributional information by randomly shuffling n-grams within the sentence where n is [1,4]. While the authors evaluated the ‘permuted’ models’ performance against ‘normally-trained’ Transformer models in a wide range of settings - GLUE, etc. - I’d be interested to see how the models perform against some of the human-curated benchmark datasets such as PAWS.

1 Like

Very cool project! I think we should try to keep the scope as small as possible → so maybe:

  1. PreTrain RoBERTa on permuted words
  2. Evaluate “permuted words” pretrained RoBERTa against “real” pretrained RoBERTa on PAWS?

What do you think @andreaparker ?

Anyone who would like to join? :slight_smile:

Hi @patrickvonplaten ! / Great to e-meet you!

Thanks for the scoping guidance; your proposed workflow and scope sounds great! I appreciate the assistance in constraining the scope to something a bit more manageable for the Community Week ‘timebox’.

Hopefully other folks will feel free to join in on this project as I’m all about ‘data science as a team sport’ :tada:, but I’ll keep an eye on the #flax-jax-community-week Slack channel for any updates regarding “single-person” projects in case permuted-word-order RoBERTa doesn’t pique anyone else’s interest. :slightly_smiling_face:

1 Like

Finalizing it!