I am trying to use AutoModelForTokenClassification from transformers. It is importing correctly but I’m getting the following runtime error when executing
model = AutoModelForTokenClassification.from_pretrained("distilbert-base-uncased", num_labels=len(label_list))
Error:
"name": "RuntimeError", "message": "Failed to import transformers.models.camembert.configuration_camembert because of the following error (look up to see its traceback):\ndlopen(/Users/etan/Projects/Proj1/venv/lib/python3.9/site-packages/tokenizers/tokenizers.cpython-39-darwin.so, 0x0002): tried: '/Users/etan/Projects/Proj1/venv/lib/python3.9/site-packages/tokenizers/tokenizers.cpython-39-darwin.so' (**mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e')**), '/usr/local/lib/tokenizers.cpython-39-darwin.so' (no such file), '/usr/lib/tokenizers.cpython-39-darwin.so' (no such file)"
Looks like it has something to do with the fact that I have an x86_64 architecture, but when I execute the command exec arch -arm64 zsh
it exits my virtual environment and applies the arch to my global env. What should I do? For reference, I have a 2021 MacBook Pro with M1 chip running MacOS Monterey (12.2.1).