Hello guys,
Undergrad student here. Recently came across this paper called “VAPO: End-to-end Slide-Enhanced Speech Recognition with Omni-modal Large Language Models”. The gist is that OLLMs, when it comes to transcribing audio to text using slide image as context, the model hallucinates. The output text contains hallucinated text in the slide which was never spoken. They solved this problem by utilizing a reinforcement learning paradigm where the model extracts visual priors in a ‘think’ block and then transcribes the remaining text in the ‘answer’ block.
Now, I am an undergrad student in CS and although I have worked with small machine learning projects, I find this problem to be interesting. From what I understand, the paper says they designed a ‘temporally decoupled policy’ which according to my brief understanding means that in normal OLLMs or even the qwen omni architecture, all the different types of tokens are flattened and then concatenated which causes this problem and so the authors had to look for this ‘temporally decoupled policy’.
I have been deliberating about the qwen 2.5 3B omni architecture for quite some time. Is it possible to apply the idea of the authors architecturally? Is there a way to make sure that visual tokens do not interfere with audio tokens somehow? Perhaps to capture the embeddings from the visual encoder before it passes to the thinker architecture block? Or perhaps making changes to the thinker block itself?
I wasn’t sure whether this would belong in the research section or even in the beginner section but I have to start from somewhere.
Really interesting problem to pick as an undergrad — and you’ve already named the mechanism.
Before touching the architecture: the VAPO authors didn’t rewire the model first, they changed what it was trained to do. So test the architecture hypothesis cheaply on stock Qwen2.5-Omni-3B: run your slide+audio case once, then re-run with the visual tokens ablated (zeroed). If the hallucinated slide text disappears, the interference lives in the fusion — the flatten-then-concatenate seam — not in the encoder. The visual encoder is innocent; the thinker is letting vision leak into the transcription path.
Then the cheapest architectural gate sits at that same seam: mask the visual token positions during the audio forward pass, or fuse the visual branch late instead of flattening it in early. No encoder surgery needed.
Measure it by comparing thinker-block embeddings on audio-only tokens before and after the gate — if drift drops and the transcript stays clean, you’ve got your mechanism. The RL approach is the heavy hammer; the gate is the experiment that tells you whether you need it.
Holy shit, thanks, I was thinking really hard about the problem and I came across this very interesting paper serendipitously called ‘Seeing is Understanding’ where they unlock the causal attention masks of decoder only multimodal LLMs so that image tokens can bidirectionally attend to the text query tokens. The problem they found is the one which you totally grasped, the whole ‘flatten-then-cocatenate’.
Infact, I have been fiddling with the qwen 2.5 Omni 3B model in Colab and seeing how the tokens are organized and the mask works. Lots to learn but I am getting there. My hypothesis is what if both audio and visual tokens attend to each other or the media tokens attend to the prompt, will it decrease the interference? That is one experiment I had in mind. The next is as you suggested, masking the visual tokens during the transcription, perhaps only transcript tokens can see the audio and the prompt or the late fusion where I can process audio and prompt tokens first to get a clean transcription then bring the visual at a later stage.
Nice find — sharper aim: let the slide inform the thinking, never the writing.
Their door starts closed — the image gets to see the question. Yours starts open — the transcript reading the slide. (Honest note: their unlock ran during finetuning, so flipping it at inference is a weaker move.)
Two switches:
Ablate first: zero the visual span on the stock model. If the hallucinated text survives, the doors aren’t the problem.
If it clears: mask transcript-rows × visual-cols to -inf. Control: the reverse — does visual grounding help or feed it?
Same slide+audio pair. Check clean-audio WER before and after: a gate that fixes the transcript but mangles audio is a trade, not a fix.