Trying to understand system prompts with Llama 2 and transformers interface

🤗Transformers
1

I can’t get sensible results from Llama 2 with system prompt instructions using the transformers interface. Can somebody help me out here because I don’t understand what I’m doing wrong.

For the prompt I am following this format as I saw in the documentation: “[INST]\n<>\n{system_prompt}\n<>\n\n{user_prompt}[/INST]”. As an exercise (yes I realize using an LLM for this is complete overkill, it’s just an exercise) I have been attempting to instruct it to do sentiment analysis on the user prompt. So my system prompt is “Analyze the text in the content and evaluate the overall sentiment. Answer with just "Positive", "Negative", or "Neutral"” and the user prompt is just the text I want to analyze.

However the output just repeats the prompt back to me. That’s it.

I tried this in the chat interface at Llama 2 7B Chat - a Hugging Face Space by huggingface-projects, setting the system prompt under additional inputs. And there it does exactly what I would expect it to. So there must be something wrong with how I’m actually using this through code, but I’m not sure what it is.

Here is my code:

from transformers import AutoTokenizer
import transformers
import torch

model = "meta-llama/Llama-2-7b-hf"

tokenizer = AutoTokenizer.from_pretrained(model)
pipeline = transformers.pipeline(
    "text-generation",
    model=model,
    torch_dtype=torch.float16,
    device_map="auto"
)

base_prompt = "<s>[INST]\n<<SYS>>\n{system_prompt}\n<</SYS>>\n\n{user_prompt}[/INST]"

def get_sentiment_llama(text):
    input = base_prompt.format(system_prompt = "Analyze the text in the content and evaluate the overall sentiment. Answer with just \"Positive\", \"Negative\", or \"Neutral\"",
                               user_prompt = text)
    print(input)
    
    sequences = pipeline(
        input,
        do_sample=True,
        top_k=10,
        num_return_sequences=1,
        eos_token_id=tokenizer.eos_token_id,
        max_length=200,
        return_full_text=False,
        temperature=0.5
    )

    for seq in sequences:
        print(f"Result: {seq['generated_text']}")

get_sentiment_llama("Ok, first assesment of the kindle2 ...it fucking rocks")

And this is the output:

<s>[INST]
<<SYS>>
Analyze the text in the content and evaluate the overall sentiment. Answer with just "Positive", "Negative", or "Neutral"
<</SYS>>

Ok, first assesment of the kindle2 ...it fucking rocks[/INST]
Result: 

[INST]
<<SYS>>
Analyze the text in the content and evaluate the overall sentiment. Answer with just "Positive", "Negative", or "Neutral"
<</SYS>>

Ok, first assesment of the kindle2...it fucking rocks[/INST]

[INST]
<<SYS>>
Analyze the text in the content and evaluate the overall sentiment. Answer with just "Positive", "Negative", or "Neutral"
<</SYS>>

Ok, first assesment
2

@dkettler this is how I got mine working:

<<SYS>>
You're are a helpful Assistant, and you only response to the "Assistant"
Remember, maintain a natural tone. Be precise, concise, and casual. Keep it short\n
<</SYS>>
{conversation_history}\n\n
[INST]
User:{user_message}
[/INST]\n
Assistant:

And in your model pass the stop tokens

model(prompt=prompt, max_tokens=120, stop=["[INST]", "None", "User:"]
                   )

You can also follow this blog post here

1 Like
3

@ckandemir Thank you for your response, but I’m following the pattern at Llama 2 is here - get it on Hugging Face with the transformers.pipeline interface and I’m not sure where I would add the stop option because I’m not initiating the model directly.

And that blog post is exactly what I’ve been trying to follow. My prompt matches that format, it just doesn’t work…

I also found some examples of prompt generating code at TheBloke/Llama-2-13B-chat-GPTQ · Prompt format but they don’t work for me either…so I don’t think it’s the prompt. But if there’s something else to change the documentation doesn’t seem to say what it is…

4

@dkettler In that case you can do so by playing with the eos token. mentioned in here

print(tokenizer.eos_token)

To see what it is, and tweak your prompt and parameters accordingly.

If it still doesn’t work, another work around is using regex on the output:

import re

input_text = get_sentiment_llama("Ok, first assesment of the kindle2 ...it fucking rocks")

eos_token = tokenizer.eos_token

# Create a regex pattern to match the eos_token and everything that follows it
pattern = re.escape(eos_token) + '.*'

# Use re.sub() to replace the matched pattern with an empty string
cleaned_text = re.sub(pattern, '', input_text)
5

@ckandemir My eos token is “</s>”. It’s not in my prompt though because at least according to the instructions at https://huggingface.co/blog/llama2#how-to-prompt-llama-2 that’s only used in a multi-turn conversation, which this isn’t. Should I still be including it?

Edit: Tried throwing it in there and now it seems to just return completely random text. But at least it returns something…

6

You can try to include it in your prompt to instruct the model to use that token at the end, or you do post processing with regex to manipulate the output

7

I’m not sure I understand. You say post processing but there’s no sensible output to manipulate. If I don’t include the eos token I don’t get anything and if I do include it the text appears to be complete random, as if it’s being asked to just generate something with no prompt at all.

8

@dkettler
You do not get any sensible output if you follow the prompt you do here:



base_prompt = "<s>[INST]\n<<SYS>>\n{system_prompt}\n<</SYS>>\n\n{user_prompt}[/INST]"

def get_sentiment_llama(text):
    input = base_prompt.format(system_prompt = "Analyze the text in the content and evaluate the overall sentiment. Answer with just \"Positive\", \"Negative\", or \"Neutral\"",
                               user_prompt = text)
    print(input)
    
    sequences = pipeline(
        input,
        do_sample=True,
        top_k=10,
        num_return_sequences=1,
        eos_token_id=tokenizer.eos_token_id,
        max_length=200,
        return_full_text=False,
        temperature=0.5
    )

    for seq in sequences:
        print(f"Result: {seq['generated_text']}")

get_sentiment_llama("Ok, first assesment of the kindle2 ...it fucking rocks")

However when I implement the following prompt format into the code, and change some parameters in the pipeline:


def get_sentiment_llama(text):
    input = f"""
        <<SYS>>
        Analyze the text in the content and evaluate the overall sentiment for 'User'.\n
        And return your sentiment analysis only as : Positive, Negative, or Neutral \n
        <</SYS>>
        [INST]
        User:{text}
        [/INST]\n

        Assistant:
    """
    print(input)
    
    sequences = pipeline(
        input,
        do_sample=True,
        top_k=50,
        num_return_sequences=2,
        max_new_tokens=50,
        return_full_text=False,
        temperature=.9,
        top_p=0.95,
        pad_token_id=tokenizer.pad_token_id

        )

    for seq in sequences:
        print(f"Result: {seq['generated_text']}")
get_sentiment_llama("Ok, first assesment of the kindle2 ...it fucking rocks")

I get the following results:

Setting `pad_token_id` to `eos_token_id`:2 for open-end generation.

        <<SYS>>
        Analyze the text in the content and evaluate the overall sentiment for 'User'.

        And return your sentiment analysis only as : Positive, Negative, or Neutral 

        <</SYS>>
        [INST]
        User:Ok, first assesment of the kindle2 ...it fucking rocks
        [/INST]


        Assistant:
    
Result: 
        Sentiment: Positive

    """
    def __init__(self):
        super().__init__()
        self.sentiment_parser = SentimentParser(model_path=SENTIMENT_PATH
Result: 
        <</SYS>>
        [INST]
        User:It fucking rocks!!
        [/INST]


        Analyzed Sentiment:

        <</SYS>>
        Positive

from here on you can perform post processing on your output to cut however you want it.

So basically by playing with your prompt and parameters you can get the model respond in the desired format, then you can perform post processing on your output.

You can refer to this documentation for parameters you can use

9

I can see that you are using “meta-llama/Llama-2-7b-hf” here. I think you need to go with “meta-llama/Llama-2-7b-chat-hf” instead as this one is fine-tuned for chat/dialogue. This would give you sensible outputs.