Is it possible to add `system prompt` to Blenderbot?

🤗Transformers
1

This is a simple BlenderBot app:

from transformers import BlenderbotSmallTokenizer, BlenderbotSmallForConditionalGeneration
import os

class BlenderBot:
    def __init__(
        self,
        model_name: str ='facebook/blenderbot_small-90M',
    ):
        if not os.path.exists('./models/blenderbot'):
            BlenderbotSmallForConditionalGeneration.from_pretrained(model_name).save_pretrained('./models/blenderbot')
            BlenderbotSmallTokenizer.from_pretrained(model_name).save_pretrained('./models/blenderbot')

        self.model = BlenderbotSmallForConditionalGeneration.from_pretrained('./models/blenderbot')
        self.tokenizer = BlenderbotSmallTokenizer.from_pretrained('./models/blenderbot')

    def __call__(self, inputs: str) -> str:
        inputs_tokenized = self.tokenizer(inputs, return_tensors='pt')
        reply_ids = self.model.generate(**inputs_tokenized)
        reply = self.tokenizer.batch_decode(reply_ids, skip_special_tokens=True)[0]

        return reply

    def run(self):
        while True:
            user_input = input("User: ")
            print("Bot:", self(user_input))

The problem is i dont know how to add any system prompts to manage the out puts of the chatbot. Any help with that ?

2

Hi,

Blenderbot does not support system prompts. An example of a model which supports this is Meta’s LLaMa 3.1 series, e.g. meta-llama/Meta-Llama-3.1-8B-Instruct · Hugging Face.

One can set the system prompt as shown here (notice the “system” message):

import transformers
import torch

model_id = "meta-llama/Meta-Llama-3.1-8B-Instruct"

pipeline = transformers.pipeline(
    "text-generation",
    model=model_id,
    model_kwargs={"torch_dtype": torch.bfloat16},
    device_map="auto",
)

messages = [
    {"role": "system", "content": "You are a pirate chatbot who always responds in pirate speak!"},
    {"role": "user", "content": "Who are you?"},
]

outputs = pipeline(
    messages,
    max_new_tokens=256,
)
print(outputs[0]["generated_text"][-1])