This kind of problem is a task in the SuperGLUE benchmark.
Generally, the approach is to fine-tune a BERT-like model with question/context pairs with a SEP token (or equivalent) separating them. Accordingly, labels correspond to yes/no answers.
You can use BERT for sequence classification model to do so.
Do note though, that the objective of the BoolQ dataset is to give a yes/no answer to a question given a text that an answer may inferred from. It’s not expected to figure out whether that text is actually relevant to the question or contains the answer, so the BoolQ formulation may not be directly applicable for your task at hand.