Hi, when I try to run a model I get model_type is not defined, and that it should be of a certain list. I am using provided code in the model card:
model_id = "utter-project/EuroLLM-9B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)
messages = [
{
"role": "system",
"content": "You are EuroLLM --- an AI assistant specialized in European languages that provides safe, educational and helpful answers.",
},
{
"role": "user", "content": "What is the capital of Portugal? How would you describe it?"
},
]
inputs = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True, return_tensors="pt")
outputs = model.generate(inputs, max_new_tokens=1024)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Therefore, I have downloaded model locally, now I am able to run it, here is my setup:
from huggingface_hub import snapshot_download
from transformers import LlamaTokenizer, LlamaForCausalLM
import torch
DOWNLOAD_MODEL_LOCALLY = False
if DOWNLOAD_MODEL_LOCALLY:
local_path = snapshot_download(
repo_id="utter-project/EuroLLM-9B-Instruct",
local_dir="./EuroLLM-9B-Instruct",
local_dir_use_symlinks=False, # ensure full copy
)
model_path = "./EuroLLM-9B-Instruct"
tokenizer = LlamaTokenizer.from_pretrained(model_path, use_fast=False)
tokenizer.pad_token_id = tokenizer.eos_token_id
model = LlamaForCausalLM.from_pretrained(
model_path,
trust_remote_code=True,
device_map="auto",
torch_dtype=torch.bfloat16,
)
messages = [
{"role": "system", "content": "You are EuroLLM --- an AI assistant specialized in European languages that provides safe, educational and helpful answers."},
{"role": "user", "content": "What is the capital of the Netherlands? Tell me something about it."}
]
# Generate chat-formatted input instaed of prompt and inputs -v0, kind of working
inputs = tokenizer.apply_chat_template(
messages,
tokenize=True,
add_generation_prompt=True,
return_tensors="pt"
).to(model.device)
# # Safe pad fallback
# if tokenizer.pad_token_id is None:
# tokenizer.pad_token_id = tokenizer.eos_token_id
# Generate
outputs = model.generate(
input_ids=inputs,
max_new_tokens=512,
do_sample=False,
pad_token_id=2,
eos_token_id=4
)
# Decode
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Although I am getting output such as :
<|im_start|> system
You are EuroLLM --- an AI assistant specialized in European languages that provides safe, educational and helpful answers.
<|im_start|> user
What is the capital of the Netherlands? Tell me something about it.
<|im_start|> assistant
ونssss
Is it something I am doing wrong or the model itself is so bad, I assume the first. Could someone help me running the model correctly?