Oh and m=512 is a bit overkill.
Acoording to Patrick Lewis’ work on RAG (a model based on DPR), 128 looked more reasonable.
It led to the same performances with a significant reduction of the index size (from 140GB to around 70GB).
In the final RAG implementation, they used m=128 with an SQ8 quantization, and the final index takes 40GB.
1 Like