Hey @Neuroinformatica, if you already have labelled data then my suggestion would be to frame the problem as an entailment one, i.e. given a (question, passage) predict a boolean value for yes/no.
This is the approach taken in the BoolQ paper and fine-tuning models on this is pretty straight-forward, e.g.
from datasets import load_dataset
from transformers import AutoTokenizer, AutoModelForSequenceClassification
boolq = load_dataset("super_glue", "boolq")
model_ckpt = ...
tokenizer = AutoTokenizer.from_pretrained(model_ckpt)
boolq_enc = boolq.map(lambda x : tokenizer(x['question'], x['passage'], truncation="only_second"), batched=True)
model = AutoModelForSequenceClassification.from_pretrained(model_ckpt)
# fine-tune with Trainer or whatever method ...
I’ve fine-tuned a few BERT models this way on the Hub, e.g. here: lewtun/bert-large-uncased-wwm-finetuned-boolq · Hugging Face