Getting error while loading model from local path : Exception: expected value at line 1 column 1

mahmutc · August 20, 2024, 9:36am

You might have some issues with this process. Can you please check the size of config.json, tokenizer.json and tokenizer.model:

github.com/huggingface/transformers

Exception: expected value at line 1 column 1

opened 05:21AM - 27 Mar 23 UTC

closed 03:02PM - 30 May 23 UTC

wccccp

### System Info - `transformers` version: 4.28.0.dev0 - Platform: Linux-5.4.0-…144-generic-x86_64-with-glibc2.31 - Python version: 3.9.16 - Huggingface_hub version: 0.13.2 - PyTorch version (GPU?): 2.0.0+cu117 (True) - Tensorflow version (GPU?): not installed (NA) - Flax version (CPU?/GPU?/TPU?): not installed (NA) - Jax version: not installed - JaxLib version: not installed - Using GPU in script?: <fill in> - Using distributed or parallel set-up in script?: <fill in> ### Who can help? @ArthurZucker @sgugger @gante ### Information - [X] The official example scripts - [ ] My own modified scripts ### Tasks - [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...) - [ ] My own task or dataset (give details below) ### Reproduction File "/mnt1/wcp/BEELE/BELLE-main/generate_instruction.py", line 28, in tokenizer = AutoTokenizer.from_pretrained(checkpoint) File "/home/appuser/miniconda3/envs/wcppy39/lib/python3.9/site-packages/transformers/models/auto/tokenization_auto.py", line 679, in from_pretrained return tokenizer_class.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs) File "/home/appuser/miniconda3/envs/wcppy39/lib/python3.9/site-packages/transformers/tokenization_utils_base.py", line 1804, in from_pretrained return cls._from_pretrained( File "/home/appuser/miniconda3/envs/wcppy39/lib/python3.9/site-packages/transformers/tokenization_utils_base.py", line 1958, in _from_pretrained tokenizer = cls(*init_inputs, **init_kwargs) File "/home/appuser/miniconda3/envs/wcppy39/lib/python3.9/site-packages/transformers/models/bloom/tokenization_bloom_fast.py", line 118, in init super().init( File "/home/appuser/miniconda3/envs/wcppy39/lib/python3.9/site-packages/transformers/tokenization_utils_fast.py", line 111, in init fast_tokenizer = TokenizerFast.from_file(fast_tokenizer_file) Exception: expected value at line 1 column 1 ### Expected behavior i hope the file is run

Topic		Replies	Views
OSError When Trying to Load Model from Local Disk (Offline) Models	2	3436	August 28, 2024
.from_pretrained($local_path) downloading already fine-tuned model instead of loading the model locally Beginners	0	469	August 22, 2024
Can't load tokenizer for 'rukaiyaaaah/fine-tuned'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name Beginners	0	672	November 6, 2023
Transformer pipeline load local pipeline 🤗Transformers	8	9062	March 4, 2024
Download and load fine-tuned model locally (VS Code) Beginners	3	4600	January 24, 2025

Getting error while loading model from local path : Exception: expected value at line 1 column 1

Related topics