adobby
March 12, 2025, 5:07am
1
I have trained an embedding model using LoRA and have some questions regarding how to load the trained LoRA adapter.
There are two different ways to load the LoRA adapter:
direct_adapter = SentenceTransformer(lora_adapter_path)
(2) Adding to a base model
model = SentenceTransformer(base_model)
model.load_adapter(lora_adapter_path)
I would like to know if there are any performance differences between these two methods and the reason for using one method over the other.
1 Like
I think the former may be slightly faster. If you apply the merge_and_unload() function mentioned in the following issue to the latter, it should become a merged state similar to the former.
Once merged, there are no particular drawbacks other than the fact that it cannot be separated.
opened 12:56AM - 25 Feb 25 UTC
closed 09:34PM - 27 Feb 25 UTC
I am working on fine tuning a 7B model and due to the size, we trained it with l… ora- by following the guidance (https://sbert.net/examples/training/peft/README.html)
```python
peft_config = LoraConfig(
task_type=TaskType.FEATURE_EXTRACTION,
inference_mode=False,
r=8,
lora_alpha=32,
lora_dropout=0.1,
)
model.add_adapter(peft_config)
```
Training works great and we are looking for some guidances to merge the lora layer with the base model and saved.
What we have tried:
1. `model.save_pretrained("")` => only save the lora layer
2. using `peft` library: this doesn't seem to work correctly, as the inference result is the same as the base model.
```
model.save_pretrained(tmp_path)
base_model = SentenceTransformer(model_name_or_path=model_path)
adapter_model = PeftModel.from_pretrained(base_model, adapter_tmp_path)
merged_model = adapter_model.merge_and_unload()
merged_model.config = transformers.AutoConfig.from_pretrained(model_path)
merged_model.save_pretrained(path)
```
We are reaching out for insights about how to merge the sentence transformer trained peft model with the base model. Thanks!