Facebook FAISS on Databricks

Intermediate
1

Been exploring this FAISS example on databricks, but I simply cannot get beyond indexing.

embeddings_dataset.add_faiss_index(column=“embeddings”)

leads to the following error.

ImportError: You must install Faiss to use FaissIndex. To do so you can run conda install -c pytorch faiss-cpu or conda install -c pytorch faiss-gpu. A community supported package is also available on pypi: pip install faiss-cpu or pip install faiss-gpu. Note that pip may not have the latest version of FAISS, and thus, some of the latest features and bug fixes may not be available.

No matter how I do it, I cannot manage to get FAISS working.

3 Likes
2

Were you able to find the solution? I am encountering the same issue in Snowflake.

1 Like
3

I (finally) got FAISS to work – at least for Chapter 5 of the LLM Course on Semantic search with FAISS. Crucially, I ran the embeddings_dataset.add_faiss_index(column=“embeddings”) cell, completed the Semantic search tutorial successfully and got the right answer. :100:

I tried everything I could find online, including Gemini (who gave up), but the solution that worked for me was from the Github README.md file for faiss_prebuilt. Here is the code:

!apt install libopenblas-base libomp-dev
!pip install faiss-cpu

I am running Python 3.11.13 in Colab, and I had to run this code to get datasets to work:

pip install -U datasets

The other methods I tried that failed were:

  1. !apt install -y libomp-dev
    !python -m pip install --upgrade faiss-gpu
  2. !pip install faiss-gpu-cu11==1.10.0
  3. !pip install faiss-cpu
  4. !pip install faiss-gpu
  5. !pip install faiss-cpu=1.7.4 mkl=2021 blas=1.0=mkl
  6. !wget https://anaconda.org/pytorch/faiss-cpu/1.2.1/download/linux-64/faiss-cpu-1.2.1-py36_cuda9.0.176_1.tar.bz2
    !tar xvjf faiss-cpu-1.2.1-py36_cuda9.0.176_1.tar.bz2
    !cp -r lib/python3.6/site-packages/* /usr/local/lib/python3.6/dist-packages/
    !pip install mkl

but NONE worked for me. I considered downgrading Python to 3.7 because the pypi.org FAISS project lists it as a programming language, but since I got it to work, I don’t have to! :partying_face:

1 Like