Llama 2 repeats its prompt as output without answering the prompt

🤗Transformers
1

I followed the Huggingface instructions (Llama 2 is here - get it on Hugging Face) but I cannot get a proper output from Llama 2. It keeps repeating the prompt without giving an output. Inputting a simple prompt without system prompt does work but for the purpose I’ll be using the model, I need to give a system prompt.

This is my code:

tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf", use_fast = True) 
base_model = "meta-llama/Llama-2-7b-hf"

model = AutoModelForCausalLM.from_pretrained(base_model, low_cpu_mem_usage=True, return_dict=True, torch_dtype=torch.float16, device_map = "auto")

prompt = """ <s>[INST] <<SYS>>

You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe. Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature.

If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information.

<</SYS>>

There's a llama in my garden 😱 What should I do? [/INST] """

import transformers

pipeline = transformers.pipeline(
    "text-generation",
    model=model,
    tokenizer=tokenizer, 
    temperature=0.1
)


sequences = pipeline(
    prompt,
    do_sample=True,
    top_k=10,
    num_return_sequences=1,
    max_length=500,
    return_full_text=False,
    eos_token_id=tokenizer.eos_token_id)

for seq in sequences:
    print(f"{seq['generated_text']}")

What am I doing wrong?

2

For reference, this is what I get as an output when I run this code:

[INST] <<SYS>>
You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe.  Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature.

If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information.
<</SYS>>

There's a llama in my garden 😱 What should I do? [/INST] 

[INST] <<SYS>>
You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe.  Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature.

If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information.
<</SYS>>

There's a llama in my garden 😱 What should I do? [/INST] 

[INST] <<SYS>>
You are a helpful, respectful and honest assistant. Always answer
3

Try to set repetition_penalry=1.2 or ‘top_p=0.95’, or ‘temperature=0.8’.
Experiment with these parameters. Try a combination of them.

1 Like
4

Use
return_full_text=False