ValueError: model.embed_tokens.weight doesn't have any device set

🤗Transformers
1

Hi, I am having this error and I don’t know what it means. Can someone explain?
This code triggers error:
model = LlamaForCausalLM.from_pretrained(
base_model,
#load_in_8bit=True,
torch_dtype=torch.float16,
device_map=device_map,
quantization_config=quantization_config,
)
device_map = {
“transformer.word_embeddings”: 0,
“transformer.word_embeddings_layernorm”: 0,
“lm_head”: “cpu”,
“transformer.h”: 0,
“transformer.ln_f”: 0,
}

quantization_config = BitsAndBytesConfig(llm_int8_enable_fp32_cpu_offload=True)

1 Like
2

I am also facing the same error. Please help with the fix

3

Same here. Any new?

4

while using device map add “model.embed_tokens”. In my case I have set everything to “cpu” since I don’t have any GPUs, you can set your’s as required.

you may need to add “model.layers” and “model.norm” also

device_map = {
“transformer.word_embeddings”: “cpu”,
“transformer.word_embeddings_layernorm”: “cpu”,
“lm_head”: “cpu”,
“transformer.h”: “cpu”,
“transformer.ln_f”: “cpu”,
“model.embed_tokens”: “cpu”,
“model.layers”:“cpu”,
“model.norm”:“cpu”
}

5

In llama2-70b, we have encountered the same issue.

Does anyone know why this error occurs in the device_map, or perhaps have a clear solution?
If anyone has detailed documentation regarding llama2’s tokenizer, sharing it would be greatly appreciated.