Hi all
I am facing this issue:
‘ERROR: Failed building wheel for tokenizers’
‘ERROR: Could not build wheels for tokenizers, which is required to install pyproject.toml-based projects.’
Python version: 3.12
Pip version: 23.3.1
Have MS Visual Studio installed
Have MS Visual C++ 14.38 installed
System: Windows 11 64 bit
Command used: pip install transformers
Facing the same issue with installing gensim, sumy and tokenizers.
Here is the complete error:
Defaulting to user installation because normal site-packages is not writeable
Collecting transformers
Using cached transformers-4.36.1-py3-none-any.whl.metadata (126 kB)
Collecting filelock (from transformers)
Using cached filelock-3.13.1-py3-none-any.whl.metadata (2.8 kB)
Collecting huggingface-hub<1.0,>=0.19.3 (from transformers)
Using cached huggingface_hub-0.19.4-py3-none-any.whl.metadata (14 kB)
Requirement already satisfied: numpy>=1.17 in c:\users\malho\appdata\roaming\python\python312\site-packages (from transformers) (1.26.2)
Requirement already satisfied: packaging>=20.0 in c:\users\malho\appdata\roaming\python\python312\site-packages (from transformers) (23.2)
Collecting pyyaml>=5.1 (from transformers)
Using cached PyYAML-6.0.1-cp312-cp312-win_amd64.whl.metadata (2.1 kB)
Requirement already satisfied: regex!=2019.12.17 in c:\users\malho\appdata\roaming\python\python312\site-packages (from transformers) (2023.10.3)
Requirement already satisfied: requests in c:\users\malho\appdata\roaming\python\python312\site-packages (from transformers) (2.31.0)
Collecting tokenizers<0.19,>=0.14 (from transformers)
Using cached tokenizers-0.15.0.tar.gz (318 kB)
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... done
Collecting safetensors>=0.3.1 (from transformers)
Using cached safetensors-0.4.1.tar.gz (62 kB)
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: tqdm>=4.27 in c:\users\malho\appdata\roaming\python\python312\site-packages (from transformers) (4.66.1)
Collecting fsspec>=2023.5.0 (from huggingface-hub<1.0,>=0.19.3->transformers)
Using cached fsspec-2023.12.2-py3-none-any.whl.metadata (6.8 kB)
Requirement already satisfied: typing-extensions>=3.7.4.3 in c:\users\malho\appdata\roaming\python\python312\site-packages (from huggingface-hub<1.0,>=0.19.3->transformers) (4.9.0)
Requirement already satisfied: colorama in c:\users\malho\appdata\roaming\python\python312\site-packages (from tqdm>=4.27->transformers) (0.4.6)
Requirement already satisfied: charset-normalizer<4,>=2 in c:\users\malho\appdata\roaming\python\python312\site-packages (from requests->transformers) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in c:\users\malho\appdata\roaming\python\python312\site-packages (from requests->transformers) (3.6)
Requirement already satisfied: urllib3<3,>=1.21.1 in c:\users\malho\appdata\roaming\python\python312\site-packages (from requests->transformers) (2.1.0)
Requirement already satisfied: certifi>=2017.4.17 in c:\users\malho\appdata\roaming\python\python312\site-packages (from requests->transformers) (2023.11.17)
Using cached transformers-4.36.1-py3-none-any.whl (8.3 MB)
Using cached huggingface_hub-0.19.4-py3-none-any.whl (311 kB)
Using cached PyYAML-6.0.1-cp312-cp312-win_amd64.whl (138 kB)
Using cached filelock-3.13.1-py3-none-any.whl (11 kB)
Using cached fsspec-2023.12.2-py3-none-any.whl (168 kB)
Building wheels for collected packages: safetensors, tokenizers
Building wheel for safetensors (pyproject.toml) ... error
error: subprocess-exited-with-error
× Building wheel for safetensors (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [35 lines of output]
Running `maturin pep517 build-wheel -i C:\Program Files\Python312\python.exe --compatibility off`
ðŸ\x8d¹ Building a mixed python/rust project
🔗 Found pyo3 bindings
ðŸ\x90\x8d Found CPython 3.12 at C:\Program Files\Python312\python.exe
📡 Using build options features, bindings from pyproject.toml
Compiling proc-macro2 v1.0.70
Compiling unicode-ident v1.0.12
Compiling target-lexicon v0.12.12
Compiling autocfg v1.1.0
Compiling once_cell v1.18.0
Compiling windows_x86_64_msvc v0.48.5
Compiling syn v1.0.109
Compiling libc v0.2.150
Compiling serde v1.0.193
error: linker `link.exe` not found
|
= note: program not found
note: the msvc targets depend on the msvc linker but `link.exe` was not found
note: please ensure that Visual Studio 2017 or later, or Build Tools for Visual Studio were installed with the Visual C++ option.
note: VS Code is a different product, and is not sufficient.
error: could not compile `windows_x86_64_msvc` (build script) due to previous error
warning: build failed, waiting for other jobs to finish...
error: could not compile `proc-macro2` (build script) due to previous error
error: could not compile `syn` (build script) due to previous error
error: could not compile `libc` (build script) due to previous error
error: could not compile `serde` (build script) due to previous error
error: could not compile `target-lexicon` (build script) due to previous error
💥 maturin failed
Caused by: Failed to build a native library through cargo
Caused by: Cargo build finished with "exit code: 101": `"cargo" "rustc" "--features" "pyo3/extension-module" "--message-format" "json-render-diagnostics" "--manifest-path" "C:\\Users\\malho\\AppData\\Local\\Temp\\pip-install-ue0afhiv\\safetensors_8421124225bb4540a576b10554c18b9c\\bindings\\python\\Cargo.toml" "--release" "--lib"`
Error: command ['maturin', 'pep517', 'build-wheel', '-i', 'C:\\Program Files\\Python312\\python.exe', '--compatibility', 'off'] returned non-zero exit status 1
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for safetensors
Building wheel for tokenizers (pyproject.toml) ... error
error: subprocess-exited-with-error
× Building wheel for tokenizers (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [34 lines of output]
Running `maturin pep517 build-wheel -i C:\Program Files\Python312\python.exe --compatibility off`
ðŸ\x8d¹ Building a mixed python/rust project
🔗 Found pyo3 bindings
ðŸ\x90\x8d Found CPython 3.12 at C:\Program Files\Python312\python.exe
📡 Using build options features, bindings from pyproject.toml
Compiling autocfg v1.1.0
Compiling proc-macro2 v1.0.69
Compiling unicode-ident v1.0.12
Compiling windows_x86_64_msvc v0.48.5
Compiling cfg-if v1.0.0
Compiling syn v1.0.109
Compiling target-lexicon v0.12.12
Compiling scopeguard v1.2.0
Compiling libc v0.2.150
error: linker `link.exe` not found
|
= note: program not found
note: the msvc targets depend on the msvc linker but `link.exe` was not found
note: please ensure that Visual Studio 2017 or later, or Build Tools for Visual Studio were installed with the Visual C++ option.
note: VS Code is a different product, and is not sufficient.
error: could not compile `windows_x86_64_msvc` (build script) due to previous error
warning: build failed, waiting for other jobs to finish...
error: could not compile `proc-macro2` (build script) due to previous error
error: could not compile `syn` (build script) due to previous error
error: could not compile `libc` (build script) due to previous error
error: could not compile `target-lexicon` (build script) due to previous error
💥 maturin failed
Caused by: Failed to build a native library through cargo
Caused by: Cargo build finished with "exit code: 101": `"cargo" "rustc" "--features" "pyo3/extension-module" "--message-format" "json-render-diagnostics" "--manifest-path" "C:\\Users\\malho\\AppData\\Local\\Temp\\pip-install-ue0afhiv\\tokenizers_89153c55d3464e4fb4f83393492f9b0b\\bindings\\python\\Cargo.toml" "--release" "--lib"`
Error: command ['maturin', 'pep517', 'build-wheel', '-i', 'C:\\Program Files\\Python312\\python.exe', '--compatibility', 'off'] returned non-zero exit status 1
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for tokenizers
Failed to build safetensors tokenizers
ERROR: Could not build wheels for safetensors, tokenizers, which is required to install pyproject.toml-based projects