FAISS and Elastic capabilities in IterableDataset

It seems that add_faiss_index functionality is not available in IterableDataset. My question is that if the dataset is big and not fit in memory, how can I leverage FAISS and Elastic capabilities?

Hi! Yes, IterableDataset doesn’t support vector similarity search, because, with it, you only have access to one example at a time. It seems that both Faiss and ElasticSearch support memory mapping, so we will probably add support for that to the Dataset class soon.

Some external resources that coud help:

1 Like