I have a custom module (a linear layer as a projector).
In the old version of transformers (4.31.0), I can fine-tuning this module and freeze the backbone LLM.
However, when I update the transformers to a new version (4.41.0), I am not allowed to do that, and the program raises the error:
ValueError: You cannot perform fine-tuning on purely quantized models.
I do not need to use LoRA to train a tiny module, how can I do the Full fine-tuning on my tiny module?
Thanks
Seems newer Transformersâ spec. For smaller models, I think the simplest approach is de-quantization â fine-tuning â quantization (if needed).
Youâre running into a deliberate new restriction in recent transformers versions, not a bug in your code.
At a high level:
transformers>=4.38 (including 4.41.0), the Trainer refuses to train any model that is marked as quantized (is_quantized=True) unless itâs wrapped as a PEFT/adapter model.(GitHub)requires_grad=True, so even âonly my tiny projector is trainable, the backbone is frozenâ is blocked.So the behavior you had in 4.31.0 was âallowed by omissionâ; the new versions added an explicit guard.
Below:
The error text comes from Trainer.__init__ in transformers:
âYou cannot perform fine-tuning on purely quantized models. Please attach trainable adapters on top of the quantized model to correctly perform fine-tuning. Please see: https://huggingface.co/docs/transformers/peft for more detailsâ(GitHub)
Internally, recent Trainer code does roughly this:(GitHub)
# simplified sketch of what happens inside Trainer.__init__
def _is_peft_model(model):
# True if model is a PEFT PeftModel
...
_is_quantized_and_base_model = getattr(model, "is_quantized", False) and not getattr(
model, "_hf_peft_config_loaded", False
)
# model already loaded here
if _is_quantized_and_base_model and not _is_peft_model(model):
raise ValueError(
"You cannot perform fine-tuning on purely quantized models. "
"Please attach trainable adapters on top of the quantized model..."
)
Key points:
model.is_quantized == True is set when you load with a quantization config (e.g. bitsandbytes 4-bit/8-bit, AWQ, etc.).(Hugging Face)_hf_peft_config_loaded is set when the model has PEFT adapters attached (e.g. LoRA, other PEFT methods).(GitHub)_is_peft_model(model) checks if this is actually a PeftModel.If the model is quantized and not a PEFT model, Trainer raises the ValueError before it cares which parameters are frozen.
Thatâs why âfrozen quantized backbone + tiny trainable projectorâ still hits the error.
In 4.31.0, this quantization guard either didnât exist or was much weaker.
People could do exactly what you did: load a model with load_in_4bit / quantization_config, freeze everything except a small head or projector, and call Trainer. It âworkedâ, but it was never a formally supported path.
Starting around late 2023, the HF team explicitly documented that pure 4-bit/8-bit training is not supported, only PEFT adapters on top of quantized weights are supported:
So:
ValueError.Your code is not âwrongâ; the library policy changed.
This is the simplest way to get full gradient training of your tiny projector while staying fully supported and still freezing the LLM.
Idea:
During training
Trainer normally â only projector parameters have gradients.After training
This completely avoids the âpurely quantized modelâ check, because the model isnât quantized while training.
import torch
import torch.nn as nn
from transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments, Trainer
model_name = "your-llm-here"
# 1. load in bf16/fp16 (not quantized)
backbone = AutoModelForCausalLM.from_pretrained(
model_name,
torch_dtype=torch.bfloat16, # or torch.float16 / torch.float32
device_map="auto",
)
# 2. freeze backbone
for p in backbone.parameters():
p.requires_grad = False
# 3. add your projector
hidden_size = backbone.config.hidden_size
proj_dim = 256 # example
backbone.projector = nn.Linear(hidden_size, proj_dim)
# by default, projector params have requires_grad=True
# 4. define a wrapper that uses projector in forward
class BackboneWithProjector(nn.Module):
def __init__(self, model):
super().__init__()
self.model = model
def forward(self, input_ids, attention_mask=None, labels=None):
# example: take last hidden state as features
outputs = self.model(
input_ids=input_ids,
attention_mask=attention_mask,
output_hidden_states=True,
)
hidden = outputs.hidden_states[-1] # [batch, seq, hidden]
features = hidden[:, -1, :] # [batch, hidden]
logits = self.model.projector(features) # [batch, proj_dim]
# you can plug in your task-specific loss here, or
# return logits and compute loss in Trainer's compute_metrics
return {"logits": logits}
model = BackboneWithProjector(backbone).to("cuda")
# 5. training as usual with Trainer
training_args = TrainingArguments(
output_dir="./out",
per_device_train_batch_size=4,
num_train_epochs=3,
fp16=True, # if using fp16
)
trainer = Trainer(
model=model,
args=training_args,
train_dataset=train_dataset, # your dataset
eval_dataset=eval_dataset,
)
trainer.train()
Important points:
is_quantized flag â no error.After training finishes, you can quantize the backbone for inference following the standard quantization docs.(Hugging Face)
This is the cleanest and most future-proof solution if you can afford to train with an unquantized backbone (bf16/fp16).
If you really want the memory savings of a quantized backbone during training and still do not want LoRA/adapters, the supported direction is:
Trainer, because Trainer is where the âpurely quantized modelâ check lives.(Hugging Face Forums)Conceptually, this is similar to how QLoRA works: gradients go through a frozen quantized base into small extra weights, except youâre writing the loop yourself and your extra weights are your projector instead of LoRA matrices.(Hugging Face)
import torch
import torch.nn as nn
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
model_name = "your-llm-here"
# 1. load quantized backbone
quant_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16,
bnb_4bit_use_double_quant=True,
)
backbone = AutoModelForCausalLM.from_pretrained(
model_name,
quantization_config=quant_config,
device_map="auto",
)
# 2. freeze backbone
for p in backbone.parameters():
p.requires_grad = False
device = next(backbone.parameters()).device
# 3. projector in full precision
hidden_size = backbone.config.hidden_size
proj_dim = 256 # example
projector = nn.Linear(hidden_size, proj_dim).to(device)
optimizer = torch.optim.AdamW(projector.parameters(), lr=1e-4)
loss_fn = nn.CrossEntropyLoss() # or any loss you need
backbone.eval() # backbone as feature extractor
for epoch in range(num_epochs):
for batch in train_dataloader:
input_ids = batch["input_ids"].to(device)
attention_mask = batch["attention_mask"].to(device)
labels = batch["labels"].to(device)
with torch.no_grad():
outputs = backbone(
input_ids=input_ids,
attention_mask=attention_mask,
output_hidden_states=True,
)
hidden = outputs.hidden_states[-1] # [B, T, H]
features = hidden[:, -1, :] # [B, H] (example pooling)
logits = projector(features) # [B, proj_dim]
loss = loss_fn(logits, labels)
optimizer.zero_grad()
loss.backward() # gradients only in projector
optimizer.step()
Again, this gives you:
Trainer and thus no ValueError.Caveats:
But if the constraint is âno LoRA, but Iâm OK writing my own loopâ, this fits that requirement.
Even though you donât want LoRA, itâs useful to understand why the library keeps talking about âadaptersâ.
HFâs recommended pattern for quantized training is:(Hugging Face)
Trainer or SFTTrainer on the resulting PeftModel.This pattern:
The Trainer guard you hit is simply enforcing: if is_quantized=True and this is not a PEFT model, then stop.
So:
Trainer sees a PeftModel.(Hugging Face)People sometimes bypass the guard when theyâre sure theyâre only training new full-precision modules and not touching quantized base weights. Typical hacks (seen in various issues and projects)
fossies.org)
Manually flip flags on the model before passing it to Trainer:
model.is_quantized = Falsesetattr(model, "_hf_peft_config_loaded", True)This tricks _is_quantized_and_base_model into being false.
Wrap the quantized model inside your own nn.Module shell that does not expose is_quantized at all, and pass that wrapper to Trainer.
Pin Transformers to â¤4.31.0 or another pre-guard version so the error never fires.(CSDN)
These approaches:
is_quantized.If you use them, make absolutely sure:
requires_grad=True.These links reinforce the explanations above and are good to skim:
Transformers GitHub issue â âIssue with Fine-tuning LLM for Classificationâ
Shows the exact same error when trying to fine-tune a quantized Mistral model with Trainer; includes the code path in trainer.py that raises it.(GitHub)
HF forum â âResolving âCannot Perform Fine-Tuning on Purely Quantized Modelsââ
Reproduces the error with Falcon, and the answer says clearly: you must attach adapters (PEFT/LoRA) or avoid fine-tuning a purely quantized model.(Hugging Face Forums)
StackOverflow â âValueError: You cannot perform fine-tuning on purely quantized modelsâ
Q&A that explains the guard and suggests PEFT or custom loops as alternatives.(Stack Overflow)
HF blog â 4-bit transformers with bitsandbytes and QLoRA
Explicitly: âIt is not possible to perform pure 4bit training on these models⌠you can train these models by leveraging parameter efficient fine tuning methods (PEFT).â(Hugging Face)
PEFT quantization guide
Explains that quantized models are not usually further trained; PEFT lets you train extra parameters on top of a frozen quantized base.(Hugging Face)
Accelerate quantization guide â fine-tune a quantized model
Repeats the same point for 4-bit and 8-bit: pure training is not supported; use PEFT adapters.(Hugging Face)
Overview of quantization schemes in Transformers
Summarizes bitsandbytes, GPTQ, AWQ, etc., and notes limitations of training on quantized weights.(Hugging Face)
QLoRA / 4-bit fine-tuning tutorials and repos (e.g., official QLoRA repo)
Show the canonical âfrozen 4-bit base + LoRA adaptersâ training loop.(Hugging Face)
The cause of your error is the new Trainer guard that forbids training any quantized base model unless it is wrapped as a PEFT model. It doesnât care that only your projector has gradients.
To get full gradient training of your custom projector without LoRA:
Trainer; then quantize for inference.LoRA/PEFT is the officially endorsed way to fine-tune on quantized models; your non-LoRA setup is slightly off the main road, so you either stay unquantized while training or bypass Trainer for the projector-only optimization.