NVIDIA released Nemotron 3.5 Lightning two weeks ago: a 30B-A3B hybrid
Mamba MoE agent model with 1M context, text only. Tonight I released
Lightning-Omni: the same model reading images and understanding speech,
with no training involved.
The trick is a geometry coincidence that is not really a coincidence.
Lightning shares the exact backbone shape of NVIDIA’s Nemotron-3-Nano-Omni
(nemotron_h, hidden 2688, 52 layers, same 131072 vocabulary). The Omni’s
projectors were trained by NVIDIA to translate C-RADIO vision features and
Parakeet audio features into precisely that embedding geometry. So the
towers and projector attach to Lightning zero-shot: download, load, works.
What I measured before claiming anything:
- Image understanding: matches the donor Omni model exactly on my gates
(reads text and shapes correctly at temp 0). - Audio: on a 38-word test narration, the donor scores 2.6 percent word
error rate, the zero-shot assembly 7.9 percent. The extra errors are
rare proper nouns, including one the donor also misses. Single-clip
gate, stated at that scope. - Video: wired and functional but EXPERIMENTAL. The donor reads my test
clip perfectly through the same binary and projector; zero-shot
Lightning often reports a black screen. The gap is measurably
Lightning-specific; my working hypothesis is drift on the temporal
patch pathway, since static images are unaffected. A Lightning-tuned
projector (v1.1) is planned and will test that hypothesis. - Decode speed is unchanged by the grafting: the projector runs once at
encode. Both siblings decode in the 60 to 69 tok/s band at Q4 on a DGX
Spark.
Every quantization passes text, image, and audio gates before upload; the
graded matrix ships in the repository and grows as tiers land:
Runtime: image, text and audio run on stock llama.cpp with these files.
Video needs my fork (the Nemotron video graph is not upstream yet), with
prebuilt arm64 CUDA binaries:
This is also a small existence proof for a general pattern: within an
architecture family, perception can be a detachable organ. Any future
nemotron_h checkpoint with this geometry can be given the same eyes and
ears in an evening. Questions welcome, and if you try it on your own
hardware I would love to see results.