NLI 2-sentence classification with GPT2, XLNet, etc.?

🤗Transformers
1

I’m doing research on NLI with 2-sentence classification. I have already successfully used BERT and its BertForSequenceClassification class to feed in two sentences in the form of an input string [CLS] sent1 [SEP] sent2 [SEP] and then perform classification.

I’d like to do the same with the other models available, such as GPT2, XLNet, and RoBERTa. However, I can’t seem to find any example code that takes two sentences for those models. Do they use the same type of input string as BERT? Can someone point me to the relevant webpages for more information?

Thank you for any help.

2

Surely there must be someone who’s done NLI with one of these other non-BERT models?

3

Hi @facehugger2020
XLNet and RoBERTa can take sequence pairs just like BERT does. For roberta its’s encoded likes <s>sent1 </s></s> sent2</s>. If you pass the sentence pair to the tokenizer it’ll automatically encode it like that.

You can use the run_glue.py script here to fine-tune XLNet, RoBERTa on NLI

WIth GPT-2, you can use GPT2Model class (without LM head), and use the final embeddings returned by the model and pass them to the classification layer. Here’s one example I found.

2 Likes