RAG Model performance does not match paper

krishanudb · February 5, 2021, 11:33am

Hi.
I am trying to reproduce the results obtained in the Retriever Augmented Generation paper for Question Answering on the Natural Questions (NQ) Dataset (Exact Match accuracy 44%).
However, I am not able to reproduce them. I am getting scores of around 40% but not more than that, in spite of trying with different document indexes and models.

Can someone kindly let me know which DPR model, DPR dataset and RAG dataset was used to obtain the 44% EM Accuracy on NQ dataset?

Thanks.

Topic		Replies	Views
RAG performance on WebQuestion dataset lower than expected Models	0	290	November 18, 2022
RAG (DPR+seq2seq) pre-trained example Models	0	372	December 12, 2022
How to increase the accuracy of answers from the model that has been fine tuned and uses the RAG and LangChain methods Models	0	939	December 1, 2023
RAG Model for QA Models	1	1284	December 30, 2023
How do I use the RagRetriever to retrieve documents? (What is the question_hidden_states variable and how do make it?) Beginners	1	409	March 18, 2024

RAG Model performance does not match paper

Related Topics