ERROR: Failed building wheel for tokenizers

Hi all

I am facing this issue:
‘ERROR: Failed building wheel for tokenizers’
‘ERROR: Could not build wheels for tokenizers, which is required to install pyproject.toml-based projects.’

Python version: 3.12
Pip version: 23.3.1
Have MS Visual Studio installed
Have MS Visual C++ 14.38 installed
System: Windows 11 64 bit

Command used: pip install transformers

Facing the same issue with installing gensim, sumy and tokenizers.

Here is the complete error:

Defaulting to user installation because normal site-packages is not writeable
Collecting transformers
  Using cached transformers-4.36.1-py3-none-any.whl.metadata (126 kB)
Collecting filelock (from transformers)
  Using cached filelock-3.13.1-py3-none-any.whl.metadata (2.8 kB)
Collecting huggingface-hub<1.0,>=0.19.3 (from transformers)
  Using cached huggingface_hub-0.19.4-py3-none-any.whl.metadata (14 kB)
Requirement already satisfied: numpy>=1.17 in c:\users\malho\appdata\roaming\python\python312\site-packages (from transformers) (1.26.2)
Requirement already satisfied: packaging>=20.0 in c:\users\malho\appdata\roaming\python\python312\site-packages (from transformers) (23.2)
Collecting pyyaml>=5.1 (from transformers)
  Using cached PyYAML-6.0.1-cp312-cp312-win_amd64.whl.metadata (2.1 kB)
Requirement already satisfied: regex!=2019.12.17 in c:\users\malho\appdata\roaming\python\python312\site-packages (from transformers) (2023.10.3)
Requirement already satisfied: requests in c:\users\malho\appdata\roaming\python\python312\site-packages (from transformers) (2.31.0)
Collecting tokenizers<0.19,>=0.14 (from transformers)
  Using cached tokenizers-0.15.0.tar.gz (318 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting safetensors>=0.3.1 (from transformers)
  Using cached safetensors-0.4.1.tar.gz (62 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: tqdm>=4.27 in c:\users\malho\appdata\roaming\python\python312\site-packages (from transformers) (4.66.1)
Collecting fsspec>=2023.5.0 (from huggingface-hub<1.0,>=0.19.3->transformers)
  Using cached fsspec-2023.12.2-py3-none-any.whl.metadata (6.8 kB)
Requirement already satisfied: typing-extensions>=3.7.4.3 in c:\users\malho\appdata\roaming\python\python312\site-packages (from huggingface-hub<1.0,>=0.19.3->transformers) (4.9.0)
Requirement already satisfied: colorama in c:\users\malho\appdata\roaming\python\python312\site-packages (from tqdm>=4.27->transformers) (0.4.6)
Requirement already satisfied: charset-normalizer<4,>=2 in c:\users\malho\appdata\roaming\python\python312\site-packages (from requests->transformers) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in c:\users\malho\appdata\roaming\python\python312\site-packages (from requests->transformers) (3.6)
Requirement already satisfied: urllib3<3,>=1.21.1 in c:\users\malho\appdata\roaming\python\python312\site-packages (from requests->transformers) (2.1.0)
Requirement already satisfied: certifi>=2017.4.17 in c:\users\malho\appdata\roaming\python\python312\site-packages (from requests->transformers) (2023.11.17)
Using cached transformers-4.36.1-py3-none-any.whl (8.3 MB)
Using cached huggingface_hub-0.19.4-py3-none-any.whl (311 kB)
Using cached PyYAML-6.0.1-cp312-cp312-win_amd64.whl (138 kB)
Using cached filelock-3.13.1-py3-none-any.whl (11 kB)
Using cached fsspec-2023.12.2-py3-none-any.whl (168 kB)
Building wheels for collected packages: safetensors, tokenizers
  Building wheel for safetensors (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building wheel for safetensors (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [35 lines of output]
      Running `maturin pep517 build-wheel -i C:\Program Files\Python312\python.exe --compatibility off`
      ðŸ\x8d¹ Building a mixed python/rust project
      🔗 Found pyo3 bindings
      ðŸ\x90\x8d Found CPython 3.12 at C:\Program Files\Python312\python.exe
      📡 Using build options features, bindings from pyproject.toml
         Compiling proc-macro2 v1.0.70
         Compiling unicode-ident v1.0.12
         Compiling target-lexicon v0.12.12
         Compiling autocfg v1.1.0
         Compiling once_cell v1.18.0
         Compiling windows_x86_64_msvc v0.48.5
         Compiling syn v1.0.109
         Compiling libc v0.2.150
         Compiling serde v1.0.193
      error: linker `link.exe` not found
        |
        = note: program not found

      note: the msvc targets depend on the msvc linker but `link.exe` was not found

      note: please ensure that Visual Studio 2017 or later, or Build Tools for Visual Studio were installed with the Visual C++ option.

      note: VS Code is a different product, and is not sufficient.

      error: could not compile `windows_x86_64_msvc` (build script) due to previous error
      warning: build failed, waiting for other jobs to finish...
      error: could not compile `proc-macro2` (build script) due to previous error
      error: could not compile `syn` (build script) due to previous error
      error: could not compile `libc` (build script) due to previous error
      error: could not compile `serde` (build script) due to previous error
      error: could not compile `target-lexicon` (build script) due to previous error
      💥 maturin failed
        Caused by: Failed to build a native library through cargo
        Caused by: Cargo build finished with "exit code: 101": `"cargo" "rustc" "--features" "pyo3/extension-module" "--message-format" "json-render-diagnostics" "--manifest-path" "C:\\Users\\malho\\AppData\\Local\\Temp\\pip-install-ue0afhiv\\safetensors_8421124225bb4540a576b10554c18b9c\\bindings\\python\\Cargo.toml" "--release" "--lib"`
      Error: command ['maturin', 'pep517', 'build-wheel', '-i', 'C:\\Program Files\\Python312\\python.exe', '--compatibility', 'off'] returned non-zero exit status 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for safetensors
  Building wheel for tokenizers (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building wheel for tokenizers (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [34 lines of output]
      Running `maturin pep517 build-wheel -i C:\Program Files\Python312\python.exe --compatibility off`
      ðŸ\x8d¹ Building a mixed python/rust project
      🔗 Found pyo3 bindings
      ðŸ\x90\x8d Found CPython 3.12 at C:\Program Files\Python312\python.exe
      📡 Using build options features, bindings from pyproject.toml
         Compiling autocfg v1.1.0
         Compiling proc-macro2 v1.0.69
         Compiling unicode-ident v1.0.12
         Compiling windows_x86_64_msvc v0.48.5
         Compiling cfg-if v1.0.0
         Compiling syn v1.0.109
         Compiling target-lexicon v0.12.12
         Compiling scopeguard v1.2.0
         Compiling libc v0.2.150
      error: linker `link.exe` not found
        |
        = note: program not found

      note: the msvc targets depend on the msvc linker but `link.exe` was not found

      note: please ensure that Visual Studio 2017 or later, or Build Tools for Visual Studio were installed with the Visual C++ option.

      note: VS Code is a different product, and is not sufficient.

      error: could not compile `windows_x86_64_msvc` (build script) due to previous error
      warning: build failed, waiting for other jobs to finish...
      error: could not compile `proc-macro2` (build script) due to previous error
      error: could not compile `syn` (build script) due to previous error
      error: could not compile `libc` (build script) due to previous error
      error: could not compile `target-lexicon` (build script) due to previous error
      💥 maturin failed
        Caused by: Failed to build a native library through cargo
        Caused by: Cargo build finished with "exit code: 101": `"cargo" "rustc" "--features" "pyo3/extension-module" "--message-format" "json-render-diagnostics" "--manifest-path" "C:\\Users\\malho\\AppData\\Local\\Temp\\pip-install-ue0afhiv\\tokenizers_89153c55d3464e4fb4f83393492f9b0b\\bindings\\python\\Cargo.toml" "--release" "--lib"`
      Error: command ['maturin', 'pep517', 'build-wheel', '-i', 'C:\\Program Files\\Python312\\python.exe', '--compatibility', 'off'] returned non-zero exit status 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for tokenizers
Failed to build safetensors tokenizers
ERROR: Could not build wheels for safetensors, tokenizers, which is required to install pyproject.toml-based projects
1 Like