Use of Dataset Library as Retrieval System in QnA for Production Grade solution

🤗Datasets
1

Hi All,

I planning to build a Domain Specific Question and Answer Demo which would be a Production Grade Solution. Now for Document retrieval, there are two options that come to my mind:

  1. Datasets Library with Search index Facility using either FAISS or Elastic Search

  2. Haystack Library using either FAISS or Elastic Search

Earlier I used Datasets Library as a retrieval mechanism to make a small demo on a toy Dataset. But now QnA Dataset would be big and since it would be Production grade I’m not sure if Dataset Library is meant for this or not. Every other blog which I have visited is Haystack.

Since I have earlier used Hugginface Datasets so my inclination is towards it but since it would be production grade so I want to choose right so please help me to make the right choice.

Regards
Nabarun Barua

2

@sgugger: Can you please suggest…your opinion will help me a lot.