Inference Endpoint Deployment Error

Beginners
1

i have put the key and value in secret evn. I have also been granted acces to the base model as well . But i still get this error

[Server message]Endpoint failed to start
See details
Exit code: 3. Reason: ile "/app/huggingface_inference_toolkit/handler.py", line 22, in __init__
    self.pipeline = get_pipeline(
                    ^^^^^^^^^^^^^
  File "/app/huggingface_inference_toolkit/utils.py", line 252, in get_pipeline
    hf_pipeline = pipeline(task=task, model=model_dir, device=device, **kwargs)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/transformers/pipelines/__init__.py", line 849, in pipeline
    config = AutoConfig.from_pretrained(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/transformers/models/auto/configuration_auto.py", line 1054, in from_pretrained
    config_dict, unused_kwargs = PretrainedConfig.get_config_dict(pretrained_model_name_or_path, **kwargs)
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/transformers/configuration_utils.py", line 591, in get_config_dict
    config_dict, kwargs = cls._get_config_dict(pretrained_model_name_or_path, **kwargs)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/transformers/configuration_utils.py", line 650, in _get_config_dict
    resolved_config_file = cached_file(
                           ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/transformers/utils/hub.py", line 421, in cached_file
    raise EnvironmentError(
OSError: You are trying to access a gated repo.
Make sure to have access to it at https://huggingface.co/meta-llama/Llama-2-7b-hf.
401 Client Error. (Request ID: Root=1-67a47ce3-3bf42d400bdb981152276ec0;1523df3a-9f78-4109-843b-c4bdb4270ee8)

Cannot access gated repo for url https://huggingface.co/meta-llama/Llama-2-7b-hf/resolve/main/config.json.
Access to model meta-llama/Llama-2-7b-hf is restricted. You must have access to it and be authenticated to access it. Please log in.

Application startup failed. Exiting.

https://huggingface.co/RichardLu/Llama2_7B_ABSA_Lap14

Above link is the model i am trying to deploy. I have also created custom handler as well. Please help

1 Like
2

The error content is clearly related to the token. Since you can access the base model during training, it seems that the token is not being passed correctly or the wrong token is being passed when using the endpoint.

The most forceful and reliable method is to pass it directly to the function, and the next best method is to use login().

1 Like
3

This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.