Seems some mistakes?
Your results don’t reflect training because you (1) didn’t train or infer with Mistral’s chat template, (2) computed loss on the whole string instead of only the assistant spans, (3) passed a fresh empty LoraConfig() to SFTTrainer instead of your tuned one, and (4) likely tested the base model without loading your LoRA adapter. One more factor: 100 samples is very small for steering a 7B instruct model unless you intentionally overfit. (Hugging Face)
What to fix, concretely
-
Use the Mistral chat template for both training and inference.
Mistral-Instruct expects [INST] ... [/INST]. Build training texts with tokenizer.apply_chat_template(...). Do the same at inference. If you skip this, the model sees the wrong control tokens and ignores your patterns. Docs show the exact format and an apply_chat_template example. (Accessed Oct 13, 2025.) (Hugging Face)
-
Train on the assistant’s tokens only.
Otherwise the model learns to copy the prompt. In TRL you can either:
- set
assistant_only_loss=True (newer TRL), or
- use
DataCollatorForCompletionOnlyLM(response_template='[/INST] ') to mask labels before the assistant span. (Mind the space after ].) (Accessed Oct 13, 2025.) (Hugging Face)
-
Pass the correct LoRA config.
You created peft_config = LoraConfig(...) and then threw it away by calling peft_config=LoraConfig(). Pass the one you defined.
-
Load the adapter at inference.
Saving only the adapter means the base model’s behavior is unchanged until you attach the adapter with PeftModel.from_pretrained(...) or model.load_adapter(...). (Hugging Face)
-
Small dataset ⇒ overfit on purpose or add data.
100 chats rarely shift a strong 7B instruct model. Either run many epochs with a higher adapter LR to overfit as a sanity check or scale to thousands of samples. QLoRA helps with VRAM, not data needs. (Accessed Oct 13, 2025.) (arXiv)
Minimal corrected script (QLoRA + proper masking + correct LoRA + template)
# docs:
# - Mistral template: https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1 # [INST] ... [/INST]
# - Chat templates: https://huggingface.co/docs/transformers/en/chat_templating
# - TRL SFTTrainer + loss masking: https://huggingface.co/docs/trl/en/sft_trainer
# - Load LoRA adapter for inference: https://huggingface.co/docs/transformers/en/peft
from datasets import load_dataset
from transformers import (AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig,
TrainingArguments)
from trl import SFTTrainer, DataCollatorForCompletionOnlyLM, SFTConfig
from peft import LoraConfig, PeftModel
import torch
model_id = "mistralai/Mistral-7B-Instruct-v0.1"
tok = AutoTokenizer.from_pretrained(model_id, use_fast=True)
# Map your JSONL {system, user, assistant} into a single "text" column using the chat template
def row_to_text(ex):
msgs = []
if ex.get("system"):
msgs.append({"role": "system", "content": ex["system"]})
msgs.append({"role": "user", "content": ex["user"]})
msgs.append({"role": "assistant", "content": ex["assistant"]})
return {"text": tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=False)}
ds = load_dataset("json", data_files="dataset.jsonl", split="train")
ds = ds.map(row_to_text, remove_columns=ds.column_names)
bnb4 = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_use_double_quant=True, # QLoRA trick
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16,
)
base = AutoModelForCausalLM.from_pretrained(model_id, quantization_config=bnb4, device_map="auto")
peft_cfg = LoraConfig(
r=16, lora_alpha=32, lora_dropout=0.05, bias="none", task_type="CAUSAL_LM"
)
# Mask labels so loss is computed only on the assistant completion
collator = DataCollatorForCompletionOnlyLM(response_template="[/INST] ", tokenizer=tok)
# Intentionally overfit small data to verify signal
train_args = TrainingArguments(
output_dir="out",
per_device_train_batch_size=1,
gradient_accumulation_steps=16,
num_train_epochs=6,
learning_rate=5e-5, # higher LR for adapters on tiny data
bf16=True,
logging_steps=5,
save_strategy="epoch",
)
sft_cfg = SFTConfig( # if your TRL is recent, you may set assistant_only_loss=True instead of the collator
dataset_text_field="text",
packing=False,
)
trainer = SFTTrainer(
model=base,
processing_class=tok,
train_dataset=ds,
data_collator=collator,
peft_config=peft_cfg, # <-- use the correct config
args=train_args,
**sft_cfg.to_dict(),
)
trainer.train()
# Save adapter + tokenizer together
trainer.model.save_pretrained("mistral-sft-adapter")
tok.save_pretrained("mistral-sft-adapter")
# -------- Inference (attach adapter and use SAME template) --------
from peft import PeftModel
base = AutoModelForCausalLM.from_pretrained(model_id, quantization_config=bnb4, device_map="auto")
model = PeftModel.from_pretrained(base, "mistral-sft-adapter").eval()
messages = [
{"role": "system", "content": "You are assistant."},
{"role": "user", "content": "Ask me the *exact* question you trained on, to sanity-check."}
]
prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tok(prompt, return_tensors="pt").to(model.device)
with torch.no_grad():
out = model.generate(**inputs, max_new_tokens=256, do_sample=True, temperature=0.7, top_p=0.9)
print(tok.decode(out[0], skip_special_tokens=True))
Why each element matters:
- Template ensures
[INST] ... [/INST] framing matches pretraining. (Hugging Face)
- Masking ensures cross-entropy hits only assistant tokens. Use
assistant_only_loss=True (new TRL) or the collator. (Hugging Face)
- QLoRA config fits 7B on 12 GB VRAM with NF4 and bf16 compute. (Hugging Face)
- Adapter loading is required at inference or your outputs come from the base model. (Hugging Face)