LoRA Adapter Loading Issue with Llama 3.1 8B - Missing Keys Warning

John6666 · March 31, 2025, 11:12am

I think this is probably a warning related to this specification.

Adapter name conflict with tuner prefix leads to unclear warning during model loading

opened 02:05PM - 03 Dec 24 UTC

closed 10:52AM - 06 Jan 25 UTC

### System Info - peft=0.13.3.dev0 - transformers=4.46.3 - Python=3.12.6 … ### Who can help? @BenjaminBossan @stevhliu ### Information - [ ] The official example scripts - [X] My own modified scripts ### Tasks - [ ] An officially supported task in the `examples` folder - [X] My own task or dataset (give details below) ### Reproduction ```python import peft import torch from pathlib import Path from transformers import AutoModelForCausalLM, AutoTokenizer from peft import LoraConfig, TaskType from tempfile import TemporaryDirectory def get_adapter_state_dict(model, adapter_name): adapter_state_dict = model.state_dict() adapter_weights = {key: value for key, value in adapter_state_dict.items() if adapter_name in key} return adapter_weights def test_adapter_name(): MODEL_ID = "openai-community/gpt2" ADAPTER_NAME = "lora" # Load the base model base_model = AutoModelForCausalLM.from_pretrained(MODEL_ID) # Define LoraConfig without base_model_name_or_path peft_config = LoraConfig( task_type=TaskType.CAUSAL_LM, inference_mode=False, r=8 ) # Get the PEFT model with the adapter name peft_model = peft.get_peft_model(base_model, peft_config, adapter_name=ADAPTER_NAME) # Saving and loading the model with TemporaryDirectory() as temp_dir: temp_dir = Path(temp_dir) adapter_weights = get_adapter_state_dict(peft_model, ADAPTER_NAME) peft_model.save_pretrained(temp_dir) loaded_peft_model = peft.PeftModel.from_pretrained(model=base_model, model_id= temp_dir / ADAPTER_NAME, adapter_name=ADAPTER_NAME) loaded_adapter_weights = get_adapter_state_dict(loaded_peft_model, ADAPTER_NAME) # Assertion fails due to adapter weights been newly intialized for key in adapter_weights: assert torch.isclose(adapter_weights[key], loaded_adapter_weights[key]).all() ``` ### Expected behavior A clear error or warning message indicating that the adapter_name and tuner_prefix should not be the same.

github.com/tloen/alpaca-lora

There were missing keys in the checkpoint model loaded

opened 12:47PM - 13 Apr 23 UTC

closed 01:16AM - 19 Apr 23 UTC

Tungsong

**I trained Alpaca-LoRA model with params:** base_model: /usr/local/dbbd/model/…llama-7b-hf data_path: alpaca_data.json output_dir: ./lora-alpaca batch_size: 128 micro_batch_size: 4 num_epochs: 2 learning_rate: 0.0001 cutoff_len: 512 val_set_size: 2000 lora_r: 8 lora_alpha: 16 lora_dropout: 0.05 lora_target_modules: ['q_proj', 'v_proj'] train_on_inputs: True group_by_length: True wandb_project: wandb_run_name: wandb_watch: wandb_log_model: resume_from_checkpoint: False prompt template: alpaca **But I get a wrong adapter_model.bin which is only 443 bytes, and some output like:** **There were missing keys in the checkpoint model loaded**: ['base_model.model.model.embed_tokens.weight', 'base_model.model.model.layers.0.self_attn.q_proj.weight', 'base_model.model.model.layers.0.self_attn.q_proj.lora_A.default.weight', '.model.layers.0.self_attn.q_proj.lora_B.default.weight', 'base_model.model.model.layers.0.self_attn.k_proj.weight', 'base_model.model.model.layers.0.self_attn.v_proj.weight', 'base_model.model.model.layers.0.self_attn.v_proj.lora_A.defaue_model.model.model.layers.0.self_attn.v_proj.lora_B.default.weight', 'base_model.model.model.layers.0.self_attn.o_proj.weight', 'base_model.model.model.layers.0.self_attn.rotary_emb.inv_freq', 'base_model.model.model.layers.0.mlp.gate_pse_model.model.model.layers.0.mlp.down_proj.weight', 'base_model.model.model.layers.0.mlp.up_proj.weight', 'base_model.model.model.layers.0.input_layernorm.weight', 'base_model.model.model.layers.0.post_attention_layernorm.weight', 'baseel.layers.1.self_attn.q_proj.weight', 'base_model.model.model.layers.1.self_attn.q_proj.lora_A.default.weight', 'base_model.model.model.layers.1.self_attn.q_proj.lora_B.default.weight', 'base_model.model.model.layers.1.self_attn.k_proj.wdel.model.model.layers.1.self_attn.v_proj.weight', 'base_model.model.model.layers.1.self_attn.v_proj.lora_A.default.weight', 'base_model.model.model.layers.1.self_attn.v_proj.lora_B.default.weight', 'base_model.model.model.layers.1.self_ht', 'base_model.model.model.layers.1.self_attn.rotary_emb.inv_freq', 'base_model.model.model.layers.1.mlp.gate_proj.weight', 'base_model.model.model.layers.1.mlp.down_proj.weight', 'base_model.model.model.layers.1.mlp.up_proj.weight', Does anyone know why this is

Topic		Replies	Views
Lora: missing adapter keys while loading the checkpoint Intermediate	2	635	January 6, 2025
Using LoRA Adapters Beginners	0	2102	January 24, 2024
`get_peft_model` or `model.add_adapter` Beginners	2	1044	February 17, 2025
Loading an LoRA adapter trained on quantized model on a non-quantized model Intermediate	0	1340	November 7, 2023
Issue with LoRA Adapter Loading on Multiple GPUs during Fine-Tuning with Accelerate and SFTTrainer 🤗Accelerate	3	835	September 18, 2024

LoRA Adapter Loading Issue with Llama 3.1 8B - Missing Keys Warning

Related topics