Simple generative question answering with context

I want to build a simple example project using HuggingFace, where I ask a question and provide context (eg, a document) and get a generated answer. How do I go best about it? Are there any pre-trained models that I can use out of the box?

I found lots of examples about extractive question answering, where the answer is a substring from the given context, but that does not always lead to satisfying answers.
It seems like the “question-answering” task and related models only focus on extractive Q&A with BERT-like models (encoder only).

For generative models, I looked at “text2text-generation”, but these models seem to only answer questions without any given context. As a result, the answers often do not make sense.

Other than that, I only found much more complex approaches including document retrieval, which is more difficult than what I want to do in my beginners project. I already have a selected document and want to use that as context - no need to do document retrieval.

I found consciousAI/question-answering-generative-t5-v1-base-s-q-c · Hugging Face, which seems closest to what I want. But it only supports very short sequence lengths that do not fit an entire document (not even a short one).
How can I build something similar but with longer context? Possibly fine-tuned to a certain domain? But still as simple as possible without the need for excessive compute?

At risk of suggesting something you’ve tried - but could one of the smaller dialogue/instruction-tuned models like Llama3-8b or Phi-3-mini-instruct adapted to longer contexts work for answering the questions?

If you already know which information is relevant to the question, providing context for a question could be as simple as inserting the document as-is into the prompt.

1 Like

Good idea, I’ll give it a try!

Hi , I’m working on something similar. I want to create a dataset in .csv or .json format to train my own pre-trained model through fine-tuning. I did the following: I installed Ollama, downloaded Solar with Ollama, and then used Ollama in server mode to interact with the model locally from my application. My script divides the text by the number of tokens that the model supports with the corresponding prompt to generate questions and answers from the text in one of the two formats, which are stored in a .txt file, and then proceeds with the next block. The model returns the questions and answers sometimes organized, other times not, and sometimes with comments that it shouldn’t make. However, it generally works well, but I am failing to get the correct data structure and cannot find the right prompt. A model that gave me good responses in lmstudio was Orca, but its performance in server mode in that application was not good.