Huggingface datasets, faiss, sbert and cosine similarity

🤗Datasets
1

Hi

I’d like to setup a questions answering system using a pretrained sbert bi-encoder model. Can I use the huggingface datasets faiss functionality to compare the question vector with my encoded corpus? To my understanding sbert uses cosine similarity and faiss the dot product for vector similarity and I guess they are not compatible.

Or is there a better option in the huggingface datasets that I should use?

Thanks in advance!

2

You can use FAISS with the inner product metric. You just have to standardize your vectors first to make sure that the inner product is equal to the cosine similarity