I have managed to use SentenceTransformer to use BERT and establish comparisons between texts and words here, with Google Colab. I am now trying to do it on my Mac, in the code I will deliver, before creating a Docker that could be run on all platforms.
My problem is that while I was able to download the library on Google Colab, it is not the same for MacOS. Indeed, I get:
(venv) remplacement@remplacements-MacBook-Pro SugiPerso % python3 -m pip install sentence-transformers
Collecting sentence-transformers
Using cached sentence-transformers-2.2.2.tar.gz (85 kB)
Preparing metadata (setup.py) ... done
Collecting transformers<5.0.0,>=4.6.0
Using cached transformers-4.27.4-py3-none-any.whl (6.8 MB)
Requirement already satisfied: tqdm in ./venv/lib/python3.8/site-packages (from sentence-transformers) (4.65.0)
Collecting torch>=1.6.0
Using cached torch-2.0.0-cp38-none-macosx_11_0_arm64.whl (55.8 MB)
Collecting torchvision
Using cached torchvision-0.15.1-cp38-cp38-macosx_11_0_arm64.whl (1.4 MB)
Requirement already satisfied: numpy in ./venv/lib/python3.8/site-packages (from sentence-transformers) (1.24.2)
Collecting scikit-learn
Using cached scikit-learn-1.2.2.tar.gz (7.3 MB)
Installing build dependencies ... error
error: subprocess-exited-with-error
× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> [75 lines of output]
Ignoring numpy: markers 'python_version == "3.10" and platform_system == "Windows" and platform_python_implementation != "PyPy"' don't match your environment
Collecting setuptools
Using cached setuptools-67.6.1-py3-none-any.whl (1.1 MB)
Collecting wheel
Using cached wheel-0.40.0-py3-none-any.whl (64 kB)
Collecting Cython>=0.29.24
Using cached Cython-0.29.33-py2.py3-none-any.whl (987 kB)
Collecting oldest-supported-numpy
Using cached oldest_supported_numpy-2022.11.19-py3-none-any.whl (4.9 kB)
Collecting scipy>=1.3.2
Using cached scipy-1.10.1.tar.gz (42.4 MB)
Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'done'
Installing backend dependencies: started
Installing backend dependencies: finished with status 'done'
Preparing metadata (pyproject.toml): started
Preparing metadata (pyproject.toml): finished with status 'error'
error: subprocess-exited-with-error
× Preparing metadata (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [41 lines of output]
The Meson build system
Version: 1.0.1
Source dir: /private/var/folders/9y/n84kv4sd3g3g957m35d_p3z40000gn/T/pip-install-06dhlrwi/scipy_d577af5217fe40cfaae75bed6ff2b162
Build dir: /private/var/folders/9y/n84kv4sd3g3g957m35d_p3z40000gn/T/pip-install-06dhlrwi/scipy_d577af5217fe40cfaae75bed6ff2b162/.mesonpy-in9wd6xp/build
Build type: native build
Project name: SciPy
Project version: 1.10.1
C compiler for the host machine: cc (clang 12.0.5 "Apple clang version 12.0.5 (clang-1205.0.22.11)")
C linker for the host machine: cc ld64 650.9
C++ compiler for the host machine: c++ (clang 12.0.5 "Apple clang version 12.0.5 (clang-1205.0.22.11)")
C++ linker for the host machine: c++ ld64 650.9
Cython compiler for the host machine: cython (cython 0.29.33)
Host machine cpu family: aarch64
Host machine cpu: aarch64
Compiler for C supports arguments -Wno-unused-but-set-variable: NO
Compiler for C supports arguments -Wno-unused-function: YES
Compiler for C supports arguments -Wno-conversion: YES
Compiler for C supports arguments -Wno-misleading-indentation: YES
Compiler for C supports arguments -Wno-incompatible-pointer-types: YES
Library m found: YES
Fortran compiler for the host machine: gfortran (gcc 12.2.0 "GNU Fortran (Homebrew GCC 12.2.0) 12.2.0")
Fortran linker for the host machine: gfortran ld64 650.9
Compiler for Fortran supports arguments -Wno-conversion: YES
Checking if "-Wl,--version-script" : links: NO
Program cython found: YES (/private/var/folders/9y/n84kv4sd3g3g957m35d_p3z40000gn/T/pip-build-env-8q2l3aja/overlay/bin/cython)
Program python found: YES (/Users/remplacement/Documents/Work/SugiPerso/venv/bin/python3)
Did not find pkg-config by name 'pkg-config'
Found Pkg-config: NO
Program pythran found: YES (/private/var/folders/9y/n84kv4sd3g3g957m35d_p3z40000gn/T/pip-build-env-8q2l3aja/overlay/bin/pythran)
Run-time dependency threads found: YES
Library npymath found: YES
Library npyrandom found: YES
Did not find CMake 'cmake'
Found CMake: NO
Run-time dependency openblas found: NO (tried pkgconfig, framework and cmake)
Run-time dependency openblas found: NO (tried framework)
../../scipy/meson.build:134:0: ERROR: Dependency lookup for OpenBLAS with method 'pkgconfig' failed: Pkg-config binary for machine 1 not found. Giving up.
A full log can be found at /private/var/folders/9y/n84kv4sd3g3g957m35d_p3z40000gn/T/pip-install-06dhlrwi/scipy_d577af5217fe40cfaae75bed6ff2b162/.mesonpy-in9wd6xp/build/meson-logs/meson-log.txt
+ meson setup --prefix=/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8 /private/var/folders/9y/n84kv4sd3g3g957m35d_p3z40000gn/T/pip-install-06dhlrwi/scipy_d577af5217fe40cfaae75bed6ff2b162 /private/var/folders/9y/n84kv4sd3g3g957m35d_p3z40000gn/T/pip-install-06dhlrwi/scipy_d577af5217fe40cfaae75bed6ff2b162/.mesonpy-in9wd6xp/build --native-file=/private/var/folders/9y/n84kv4sd3g3g957m35d_p3z40000gn/T/pip-install-06dhlrwi/scipy_d577af5217fe40cfaae75bed6ff2b162/.mesonpy-native-file.ini -Ddebug=false -Doptimization=2
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed
× Encountered error while generating package metadata.
╰─> See above for output.
note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error
× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.
It seems that the installation of scikit-learn has failed due to an error when compiling the compilation dependencies. Specifically, there is an error when preparing pyproject.toml metadata from scipy.
If it is useful, that is what I get with Google Colab:
Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/
Collecting sentence-transformers
Downloading sentence-transformers-2.2.2.tar.gz (85 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 86.0/86.0 KB 3.1 MB/s eta 0:00:00
Preparing metadata (setup.py) ... done
Collecting transformers<5.0.0,>=4.6.0
Downloading transformers-4.27.4-py3-none-any.whl (6.8 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.8/6.8 MB 25.8 MB/s eta 0:00:00
Requirement already satisfied: tqdm in /usr/local/lib/python3.9/dist-packages (from sentence-transformers) (4.65.0)
Requirement already satisfied: torch>=1.6.0 in /usr/local/lib/python3.9/dist-packages (from sentence-transformers) (1.13.1+cu116)
Requirement already satisfied: torchvision in /usr/local/lib/python3.9/dist-packages (from sentence-transformers) (0.14.1+cu116)
Requirement already satisfied: numpy in /usr/local/lib/python3.9/dist-packages (from sentence-transformers) (1.22.4)
Requirement already satisfied: scikit-learn in /usr/local/lib/python3.9/dist-packages (from sentence-transformers) (1.2.2)
Requirement already satisfied: scipy in /usr/local/lib/python3.9/dist-packages (from sentence-transformers) (1.10.1)
Requirement already satisfied: nltk in /usr/local/lib/python3.9/dist-packages (from sentence-transformers) (3.8.1)
Collecting sentencepiece
Downloading sentencepiece-0.1.97-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.3/1.3 MB 28.9 MB/s eta 0:00:00
Collecting huggingface-hub>=0.4.0
Downloading huggingface_hub-0.13.3-py3-none-any.whl (199 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 199.8/199.8 KB 20.0 MB/s eta 0:00:00
Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.9/dist-packages (from huggingface-hub>=0.4.0->sentence-transformers) (6.0)
Requirement already satisfied: packaging>=20.9 in /usr/local/lib/python3.9/dist-packages (from huggingface-hub>=0.4.0->sentence-transformers) (23.0)
Requirement already satisfied: requests in /usr/local/lib/python3.9/dist-packages (from huggingface-hub>=0.4.0->sentence-transformers) (2.27.1)
Requirement already satisfied: filelock in /usr/local/lib/python3.9/dist-packages (from huggingface-hub>=0.4.0->sentence-transformers) (3.10.7)
Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.9/dist-packages (from huggingface-hub>=0.4.0->sentence-transformers) (4.5.0)
Collecting tokenizers!=0.11.3,<0.14,>=0.11.1
Downloading tokenizers-0.13.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (7.6 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.6/7.6 MB 48.9 MB/s eta 0:00:00
Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.9/dist-packages (from transformers<5.0.0,>=4.6.0->sentence-transformers) (2022.10.31)
Requirement already satisfied: joblib in /usr/local/lib/python3.9/dist-packages (from nltk->sentence-transformers) (1.1.1)
Requirement already satisfied: click in /usr/local/lib/python3.9/dist-packages (from nltk->sentence-transformers) (8.1.3)
Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.9/dist-packages (from scikit-learn->sentence-transformers) (3.1.0)
Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /usr/local/lib/python3.9/dist-packages (from torchvision->sentence-transformers) (8.4.0)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.9/dist-packages (from requests->huggingface-hub>=0.4.0->sentence-transformers) (3.4)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.9/dist-packages (from requests->huggingface-hub>=0.4.0->sentence-transformers) (2022.12.7)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.9/dist-packages (from requests->huggingface-hub>=0.4.0->sentence-transformers) (1.26.15)
Requirement already satisfied: charset-normalizer~=2.0.0 in /usr/local/lib/python3.9/dist-packages (from requests->huggingface-hub>=0.4.0->sentence-transformers) (2.0.12)
Building wheels for collected packages: sentence-transformers
Building wheel for sentence-transformers (setup.py) ... done
Created wheel for sentence-transformers: filename=sentence_transformers-2.2.2-py3-none-any.whl size=125942 sha256=6374175b3d7f7989f81b48b1bd797203ecadd00d30b65bc4a7deacfdb004c9e5
Stored in directory: /root/.cache/pip/wheels/71/67/06/162a3760c40d74dd40bc855d527008d26341c2b0ecf3e8e11f
Successfully built sentence-transformers
Installing collected packages: tokenizers, sentencepiece, huggingface-hub, transformers, sentence-transformers
Successfully installed huggingface-hub-0.13.3 sentence-transformers-2.2.2 sentencepiece-0.1.97 tokenizers-0.13.2 transformers-4.27.4
I have a MacOS BigSure with an Apple M1 chip