Lora: missing adapter keys while loading the checkpoint

Intermediate
1

Hi,

When I load a checkpoint during inference using “PeftModel.from_pretrained”, I see missing adapter keys which are weights from {qproj,k_proj, v_proj, o_proj, gate_proj, up_proj,down_proj} of lora_A and lora_B from all layers. This seems to affect the inference. Googling this does not give me a good resolution. Can someone point me in the right direction ?

Thanks
Mohan

2 Likes
2

From what I can tell, If i push the model to the repository and later do this:

from peft import AutoPeftModelForCausalLM
model = AutoPeftModelForCausalLM.from_pretrained()

I don’t get any warnings. The adapter-config.json in the repository points to the base model from which it was trained on. But using the above code to load from the checkpoint itself did not work. I am not sure why.

Thanks
Mohan

1 Like
3

Hi Mohan,

The issue you’re describing typically occurs when the LoRA adapter weights were not saved correctly during training or when there’s a mismatch between the base model and the adapter configuration.

Here are a few things you can check to troubleshoot this issue:

  1. Verify LoRA Weights During Training

  2. Check Base Model Compatibility

  3. Adapter Configuration
    Verify that the adapter configuration (e.g., PeftConfig) is correctly set up. The PeftModel.from_pretrained() function depends on this configuration to properly initialize and load the adapter weights. Check the adapter type, rank, and any additional parameters.

  4. Use ignore_mismatched_keys=True
    When loading the model, you can set the ignore_mismatched_keys=True flag to avoid stopping the process due to missing keys. However, this is more of a workaround than a fix. Use this cautiously, as it may lead to incomplete or incorrect inference results.

  5. Log the Missing Keys
    Use the from_pretrained() function with verbose logging to identify the specific missing keys. For example:

    model = PeftModel.from_pretrained(
    model_name_or_path,
    adapter_name_or_path,
    output_loading_info=True
)

    This will return a dictionary with details about the missing and unexpected keys.

  6. Re-Export the LoRA Adapter
    If the problem persists, re-export the trained LoRA adapter weights. Ensure you save them using save_pretrained() on the PeftModel after training:

    model.save_pretrained("path_to_adapter")

If none of these resolve the issue, you might need to share more details about the training setup, base model, and adapter configuration for a more specific diagnosis.

Let me know how it goes!

Best,
Alan

2 Likes