Gated Model access error during training on SageMaker

Beginners
1

I am trying to run a training job with my own data on SageMaker using HugginFace estimator.

  1. I have accepted T&C on the model page,
  2. I do a hugging face login
from huggingface_hub import notebook_login
notebook_login()
  1. Run fit method
    git_config = {'repo': 'https://github.com/huggingface/transformers.git','branch': 'v4.26.0'}
huggingface_estimator = HuggingFace(
entry_point='run_clm.py',
source_dir='./examples/pytorch/language-modeling',
instance_type='ml.p3.2xlarge',
instance_count=1,
role=role,
git_config=git_config,
transformers_version='4.26.0',
pytorch_version='1.13.1',
py_version='py39',
hyperparameters = hyperparameters
)

huggingface_estimator.fit()

But i get access token error.

HTTPError: 401 Client Error: Unauthorized for url:
Cannot access gated URL repo for URL https://huggingface.co/bigcode/starcoder/resolve/main/config.json.

OSError: bigcode/starcoder is not a local folder and is not a valid model identifier listed on ‘Models - Hugging Face’ . If this is a private repository, make sure to pass a token having permission to this repo with use_auth_token or log in with huggingface-cli login and pass use_auth_token=True.

I have tried to pass the auth token in HuggingFace estimator but has not worked, tried to pass it as hyperparameter as a hub_token but that didnt work.
‘hub_token’: HfFolder.get_token()

How can i successfully run the fit method?

2

Same problem with LLAMA model even I have a PRO subscription.