Issues when fine tuning Llama-3.2-11B-Vision

[mgertner]

My goal is to fine tune meta-llama/Llama-3.2-11B-Vision so that it can recognize specific brand names in images. I have a large training set (~30,000 examples) with both positive and negative examples.

I successfully fine tuned the model using LoRA with an A100 GPU and 4-bit quantization. I load it by loading the base model then applying the fine tuned weights using PeftModel

I’ve run into two problems:

1. When I use the model for inference, I get correct output but with weird repetition. For example, it might output something like:

What brand is represented in this image? Respond with only the official brand name, or 'no brand' if none is present. <OCR/> SOME BRAND NAME. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

SOME BRAND NAME is the correct output but it outputs that weird <OCR/> tag before it and a row of dashes afterwards.

2. I tried setting repetition_penalty=1.2 in the generate() call but then I get a CUDA crash:

RuntimeError Traceback (most recent call last)

[<ipython-input-9-89460788cfae>](https://i5y0lzsary-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab_20250505-060110_RC00_754880401#) in <cell line: 0>() 3 print("Max input ID:", inputs["input_ids"].max().item()) 4 print("Tokenizer vocab size:", tokenizer.vocab_size) ----> 5 generated_ids = model.generate( 6 **inputs, 7 max_new_tokens=100,

---
5 frames
---

[/usr/local/lib/python3.11/dist-packages/transformers/generation/logits_process.py](https://i5y0lzsary-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab_20250505-060110_RC00_754880401#) in __call__(self, input_ids, scores) 338 339 # if score < 0 then repetition penalty has to be multiplied to reduce the token probabilities --> 340 score = torch.where(score < 0, score * self.penalty, score / self.penalty) 341 342 scores_processed = scores.scatter(1, input_ids, score)

RuntimeError: CUDA error: device-side assert triggered CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1 Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Does anyone have any idea what might be causing the weird tokens and repetition in the output? And why CUDA crashing when I try to use repetition_penalty?

[mgertner]

One other data point. For some images I get output like:

THE BEST BRAND. <OCR/> THE BEST BRAND. BRAND. BRAND. BRAND. BRAND. BRAND. BRAND. BRAND. BRAND. BRAND. BRAND. BRAND. BRAND. BRAND. BRAND. BRAND. BRAND. BRAND. BRAND. BRAND.

Once again, the output (“THE BEST BRAND”) is correct but the tag is unexpected as is the repetition after outputting the answer.

[mgertner]

I should also mention that my research suggests that that CUDA crash is caused by the model using more tokens than the tokenizers knows about:

inputs["input_ids"].max().item() = 128256
tokenizer.vocab_size = 128000

However I’m not sure why this is causing it to fail or how to fix it.

[John6666]

Errors related to repetition_penalty seems to be constraints in Llava or Llama Vision models.

llama 3.2, inference error with "repetition_penalty" in generation_config

repetition penalty

[John6666]

And perhaps:

Llama-2 7B-hf repeats context of question directly from input prompt, cuts off with newlines

I am not sure I understand your question well or not. If you want the answer from Llama 2 to not include the prompt you provide, you can use return_full_text=False sequences = pipeline( myPrompt, do_sample=True, num_return_sequences=1, eos_token_id=tokenizer.eos_token_id, max_length=4096, # max lenght of output, default=4096 return_full_text=False, # to not repeat the question, set to False top_k=10, # default=10 # top_p=0.5, # default=0.9 temperature=0.6, # …

you can use return_full_text=False