Question-Answering/Text-generation/Summarizing: Fine-tune on multiple answers

Beginners
1

Hi all!

Looking to fine-tune a model for QA/Text-Generation (not sure how to frame this) and I’m wondering how to best prepare the dataset in a way that I can feed multiple answers to the same question?

My goal is to facilitate the creation of a unique answer to a given question that is based on the input answers. The answers are longer-form (2 to 3 sentences) and I want the model to output this kind of length too.

For now my idea is to fine-tune GPT-2 and look at this as a text generation problem (but I don’t know how GPT-2 would treat multiple answers to the same question - would it adjust the weight simply in favor of the used tokens in the answers?). Maybe creating a summary of the given answers would have the same effect.

What would be the best approach to this?

1 Like
2

Maybe @valhalla has some pointers?

3

hi @enrico!

I can think of two approaches

  1. text generation/text2text

Hear as you said you could feed the question and multiple answers to the model and ask it to generate the correct answer.

One way to prepare the dataset for this is to use some delimiter tokens to separate the answers. For ex.

Input text could be
context: context text question: question_text Answer1: answer1_text Answer2: answer2_text

and output text would be the answer text.

You could also delimit the answers using sep tokens if the models have them.

  1. Frame this as a multiple-choice problem

Here instead of generating correct answer text, you ask the model to identify the correct answer within given answers discriminatively.

You can use the *ForMultipleChoice models classes for this.

You can find multiple choice example script in examples/multiple-choice dir, here

4

Thank you @valhalla leaning towards the first idea. Will give this a try

5

Maybe you could try also this solution https://huggingface.co/qa/

1 Like
6

@cahya thanks a lot! Looks really interesting didn’t know about it!

7

@enrico please have a look also on the discussion we had about it Demo of Open Domain Long Form Question Answering

8

Hello can i use multiple context in a question and answer huggingface api?? I am actually trying to make a context that involves several paragraphs about all the past 46 US presidents and I cannot seem to put all those under a single context. Can you please help me ???

9

Hi there,

If I understood your situation, you have a context, questions and answers, but your context is too big to feed the model in one passage.

I suggest you to take a look on Hugging Face’s question answering example notebook. They manage to solve this problem splitting up the context in several parts, when necessary. During training, if the context split does not contain the answer, they point the answer to the CLS token. Finally, during evaluation, they search for the answer in all splits and consider the most likely answer.