How to use llm (access fail)

Beginners
1

meta-llama/Meta-Llama-3.1-8B-Instruct

I follow the tutorial and it show the error.

‘’’
Cannot access gated repo for url https://huggingface.co/meta-llama/Meta-Llama-3.1-8B-Instruct/resolve/main/config.json.
Access to model meta-llama/Meta-Llama-3.1-8B-Instruct is restricted. You must be authenticated to access it.
‘’’

Q2.
I download the Meta-Llama-3.1-8B-Instruct by git clone
How to load it with the code?
I dont see any solution?

pipeline = transformers.pipeline(
    "text-generation",
    model=model_id,
    model_kwargs={"torch_dtype": torch.bfloat16},
    device_map="auto",
)
2

hi @alice86
You need to agree the license agreement to access this model. You can find it in the model card.

FAQ

You will need this one:

3

Q1
I already accept the license.
And the official don’t show any method to download it.

It just direct use

pipeline = transformers.pipeline(
    "text-generation",
    model=model_id,
    model_kwargs={"torch_dtype": torch.bfloat16},
    device_map="auto",
)

Q2
It is different question.
If I download the model and how to load model without token? (already download model)
Because I already download the model.

I expect the result is
python test.py => it will automatically download model with token

python test.py -m ./model/llama3 => It will use the model that I already download without any token.

In fact, I dont see any code for python test.py -m ./model/llama3

4

hi @alice86
About your first question:
Did you create an access token? Did you add a permission for relevant repository?

You can check from Hugging Face – The AI community building the future..
(Edit permissions → Repositories permissions)

You need to run something like this:

pipeline = transformers.pipeline(
    "text-generation",
    model=model_id,
    model_kwargs={"torch_dtype": torch.bfloat16},
    device_map="auto",
    token="hf_xxxxxxx",
)
5

About your second question:
You need to call the pipeline with model='./path_for_local_model':

pipeline = transformers.pipeline(
    "text-generation",
    model='./path_for_local_model',
    model_kwargs={"torch_dtype": torch.bfloat16},
    device_map="auto",
)