meta-llama/Llama-2-7b-chat-hf weird responses, compared to the ones returned by the HF API

[John6666]

or that affects only speed

Basically, this should be the case, and there are few cases where you get half-baked results due to insufficient hardware performance. It’s either it works or it doesn’t, and it’s either fast or slow.

I found the official HF implementation for Llama2. It may be that tokenizer.use_default_system_prompt = False is meaningful.

Since Llama2 has been around for a long time, it has been affected by various HF specification changes, so there is likely to be some confusion about how to use it.

app.py · huggingface-projects/llama-2-7b-chat at main

Issue with LlaMA-2 Chat Template (and out of date documentation)

Hi @Rocketknight1 is see that you added the chat_template data for the LlaMA-2 models. There appears to be a bug in that logic where if you only pass in a system prompt, formatting the template returns an empty string/list. For example, the below code results in printing an empty string: chat = [ {"role": "system", "content": "You are a helpful and honest assistant."}, ] print(tokenizer.apply_chat_template(chat, tokenize=False)) However, if you edit it to have an empty user object, then it w…