I trained llama-7b-chat-hf using the following command as I saw in the video
autotrain llm --train --project_name peft-tune-llama-credit-card-fraud-v2 --model meta-llama/Llama-2-7b-chat-hf --data_path . --use_peft --use_int4 --learning_rate 2e-4 --train_batch_size 6 --num_train_epochs 10 --trainer sft
On the same Ubunutu EC2 instance that I used to train the model, I am trying to load the model. The model is not pushed to the hub. I am running into issues as follows:
import torch
import transformers
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer, LlamaTokenizer, StoppingCriteria, StoppingCriteriaList, TextIteratorStreamer
model=‘/home/ubuntu/LLM/meta-llama/peft-tune-llama-2-7b-chat-hf-credit-card-fraud-v2’
model_name = “meta-llama/Llama-2-7b-chat-hf”
adapters_name = {‘privacy/repo_name’: ‘/home/ubuntu/LLM/meta-llama/peft-tune-llama-2-7b-chat-hf-credit-card-fraud-v2’}
print(f"Starting to load the model {model} into memory")
m = AutoModelForCausalLM.from_pretrained(
model_name,
load_in_4bit=True,
torch_dtype=torch.bfloat16,
device_map={“”: 0}
)
m = PeftModel.from_pretrained(m, model)
m = m.merge_and_unload()
tok = LlamaTokenizer.from_pretrained(model)
tok.bos_token_id = 1
stop_token_ids = [0]
However, getting an exception as follows
Traceback (most recent call last):
File “/home/ubuntu/LLM/meta-llama/model_loader.py”, line 19, in
m = PeftModel.from_pretrained(m, model)
File “/home/ubuntu/LLM/meta-llama/llama-train/lib/python3.10/site-packages/peft/peft_model.py”, line 278, in from_pretrained
model.load_adapter(model_id, adapter_name, is_trainable=is_trainable, **kwargs)
File “/home/ubuntu/LLM/meta-llama/llama-train/lib/python3.10/site-packages/peft/peft_model.py”, line 554, in load_adapter
adapters_weights = load_peft_weights(model_id, device=torch_device, **hf_hub_download_kwargs)
File “/home/ubuntu/LLM/meta-llama/llama-train/lib/python3.10/site-packages/peft/utils/save_and_load.py”, line 171, in load_peft_weights
has_remote_safetensors_file = hub_file_exists(
File “/home/ubuntu/LLM/meta-llama/llama-train/lib/python3.10/site-packages/peft/utils/hub_utils.py”, line 24, in hub_file_exists
url = hf_hub_url(repo_id=repo_id, filename=filename, repo_type=repo_type, revision=revision)
File “/home/ubuntu/LLM/meta-llama/llama-train/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py”, line 110, in _inner_fn
validate_repo_id(arg_value)
File “/home/ubuntu/LLM/meta-llama/llama-train/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py”, line 158, in validate_repo_id
raise HFValidationError(
Question:
How do I fix this error, load the model and use it for a query?
IF this is all wrong, can you provide a sample code to load and use the local saved model, without going to hub?