I would like to propose a broader discussion about two problems that I think are becoming increasingly connected:
- Why do we keep training and distributing increasingly large models as mostly separate products?
- Why is the AI ecosystem still so dependent on general-purpose GPUs?
I am not a hardware engineer or an AI researcher. I am approaching this from the perspective of someone experimenting with local models and repeatedly running into the same limits: model size, VRAM, memory bandwidth, training cost, and hardware availability.
The current direction often seems to be:
4B → new 8B model
8B → new 30B model
30B → 100B+
100B+ → 300B+
and then the hardware solution is usually:
use a larger GPU,
use more GPUs,
or distribute the model across more accelerators.
I wonder whether we should seriously explore another direction.
1. Grow the model instead of repeatedly starting over
In my previous post, “Proposal: Progressive Knowledge Transfer Between Qwen Models,” I suggested keeping a small model fixed in size while allowing it to learn progressively from increasingly capable teacher models.
For example:
4B student
→ learn from 8B/9B teacher
→ improved 4B
→ learn from 12B/14B teacher
→ improved 4B
→ learn from a larger teacher
The student remains 4B.
That is one idea.
But there is another, separate possibility:
4B
→ expand to 8B
→ continue training
→ expand to 12B
→ continue training
→ expand again only when necessary
Here the model itself grows.
Instead of training every model size as an almost independent object, we could attempt to preserve as much of the already learned representation as possible while adding capacity.
Conceptually:
Stage 1:
Train a useful 4B model.
Stage 2:
Expand it to approximately 8B.
Reuse the trained 4B parameters where possible.
Initialize new layers, dimensions, heads, experts, or other capacity using function-preserving or minimally disruptive methods.
Then continue training instead of restarting.
Stage 3:
Expand the resulting model to approximately 12B.
Again preserve previous knowledge and add capacity rather than replacing everything.
Then continue:
4B → 8B → 12B → 20B → …
only when additional capacity actually produces useful gains.
This is not entirely new as a research direction.
Ideas such as Net2Net, progressive Transformer growth, bert2BERT, function-preserving expansion, depth up-scaling, and more recent model-growth research have already explored related mechanisms.
My question is different:
Why is progressive growth not a more visible first-class workflow in the modern open LLM ecosystem?
Why does a Transformers user normally download a completely different checkpoint for every size instead of having a standardized way to grow a model?
2. Fixed-size knowledge transfer and model growth could complement each other
These two ideas do not have to compete.
A possible pipeline might be:
4B base model
→ selectively learn useful capabilities from a larger teacher
→ improved 4B
Then, when the 4B model approaches its practical capacity limit:
improved 4B
→ structurally expand to 8B
→ continue training
→ selectively learn additional information from a larger teacher
Then:
improved 8B
→ expand to 12B
→ continue
This gives us two different tools:
Knowledge transfer
Improve the information contained within the existing capacity.
Model growth
Add capacity when the smaller model can no longer absorb useful improvements.
This could potentially tell us something very useful that simple parameter scaling does not:
At what point does a given model size actually become saturated?
Maybe a 4B model can absorb much more useful capability than we currently assume.
Maybe it cannot.
Either result would be valuable.
3. The hardware problem
This connects to another problem.
Transformers has become extraordinarily good at supporting new architectures.
That flexibility is one of its greatest strengths.
But there may be an unintended consequence:
Whenever a model introduces a new architectural variation, the software ecosystem adapts to it, and the hardware is expected to follow.
Eventually the answer becomes:
“Run it on GPUs.”
If it does not fit:
“Use a larger GPU.”
If that still does not fit:
“Use multiple GPUs.”
This cannot be the only long-term hardware strategy.
Cryptocurrency mining went through a familiar transition:
CPU
→ GPU
→ FPGA
→ ASIC
Once the workload became sufficiently important and sufficiently stable, specialized hardware became economically worthwhile.
AI is obviously much more complicated than SHA-256 mining. We cannot hard-wire one Transformer architecture forever.
But AI workloads also contain a large amount of recurring computation.
Examples include:
- matrix multiplication / GEMM
- INT4 / INT8 / FP8 / BF16 operations
- quantization and dequantization
- RMSNorm
- RoPE
- attention
- GQA / MQA
- KV-cache operations
- SwiGLU / related MLP operations
- embedding lookup
- MoE routing
- grouped expert matrix multiplication
Transformers is already moving toward separating some model semantics from their optimized implementations.
There are now common interfaces for attention implementations, expert backends, device backends, optimized kernels, and inference engines.
Could this be taken one step further?
4. An open “AI Hardware Profile” for Transformers
I am not suggesting that Hugging Face manufacture chips.
I am suggesting that Transformers may be in a unique position to help define a stable software target for chip manufacturers.
For example, imagine an optional specification such as:
“Transformers AI Core Profile”
A model compatible with this profile would use a documented common set of operations that accelerator vendors could implement extremely efficiently.
There could be several levels:
Core Profile
- common dense Transformer operations
- common quantization formats
- standard KV-cache behavior
- standard normalization and positional operations
MoE Profile
- routing
- grouped GEMM
- expert memory handling
Multimodal Profile
- vision/audio operations commonly required by VLM and Omni-style models
Experimental extensions could still exist.
Nothing would prevent researchers from introducing a new operation.
Unsupported operations could fall back to programmable compute.
The point would not be to restrict research.
The point would be to create a stable optimization target.
5. Then ASIC vendors would know what to build
A hardware company currently has a difficult problem.
If it builds an accelerator specifically for today’s model architecture, a new architecture may appear before the hardware reaches the market.
That strongly favors programmable GPUs.
But suppose several major model families and Transformers agreed on a sufficiently stable common execution profile.
Then a manufacturer could build:
AI compute ASIC
- programmable fallback units
- very large model memory
- extremely high memory bandwidth
- high-speed accelerator-to-accelerator interconnect
without needing graphics hardware at all.
It would not need to render games.
It would not need to be a GPU.
It would exist specifically to run AI models.
Consumer and workstation versions could prioritize large usable model memory — potentially hundreds of GB or eventually even ~1 TB-class systems — rather than forcing local AI users to treat GPU VRAM as the fundamental limit of model size.
The important constraint would of course still be memory bandwidth, not merely memory capacity.
But at least the entire machine could be designed around model execution from the beginning.
6. Transformers already contains some of the building blocks
This is why I think Hugging Face is an interesting place to discuss this.
Transformers already has abstractions for:
- multiple accelerator types
- interchangeable attention implementations
- expert/MoE backends
- optimized kernels
- inference-engine compatibility
- quantization
- distributed execution
These abstractions already separate, to some extent, the model definition from the exact hardware implementation.
Could those abstractions eventually become the foundation of a documented hardware execution profile?
Instead of every accelerator vendor implementing dozens of special cases independently, there could be a predictable target.
7. Efficiency should become a first-class benchmark
There is also a cultural problem in AI development.
Model size is very visible.
A 300B or 500B model sounds impressive immediately.
Efficiency is much harder to advertise.
I think model releases would become more informative if we compared not only benchmark scores and parameter counts, but also metrics such as:
- tokens per joule
- tokens per second per dollar of hardware
- minimum usable memory
- memory bandwidth requirements
- inference energy per token
- training compute required to reach a given score
- performance after INT8 / INT4 quantization
- performance retained after model growth
- consumer/workstation accessibility
Then a 12B model achieving almost the same practical result as a 100B model at a fraction of the compute cost would be recognized as a major engineering achievement rather than merely “the smaller model.”
8. A concrete experiment
A community experiment does not need to begin with a new ASIC.
The software question can be tested first.
For example:
Experiment A — Progressive model growth
Train or start from a 4B-class model.
Expand:
4B → 8B
using an existing function-preserving or model-growth technique.
Measure:
- training compute required after expansion
- convergence speed
- retained capabilities
- newly acquired capabilities
- catastrophic forgetting
- final performance compared with an independently trained 8B model
Then:
8B → 12B
and repeat the measurements.
The important comparison would be:
train 12B from scratch
versus
train 4B → grow to 8B → grow to 12B.
Experiment B — Progressive teacher transfer
Keep the model at 4B.
4B learns selectively from 8B/9B
→ then 12B/14B
→ then a larger teacher
Measure when additional teachers stop producing meaningful improvements.
Experiment C — Combine them
Start with progressive knowledge transfer.
When the small model reaches a measurable capacity ceiling, grow the architecture and continue.
This might produce an interesting training strategy:
learn efficiently
→ saturate capacity
→ grow
→ preserve knowledge
→ continue learning
rather than:
make a much larger model
→ train enormous model
→ compress it afterward.
9. The question for Hugging Face and model developers
I think there are several questions worth discussing:
Could Transformers expose model-growth operations as a standardized research interface?
Could compatible model families publish information describing how their architectures can be expanded in depth, width, heads, FFN dimensions, or experts?
Could Transformers define an optional hardware-friendly core execution profile?
Could Qwen, Llama, Gemma, and other model developers target such a profile without giving up architectural experimentation?
Could accelerator manufacturers implement that profile directly in ASICs or NPUs?
And could efficiency become as visible in the ecosystem as parameter count?
I do not think larger models should disappear.
Large models are useful as frontier systems, teachers, research platforms, and sources of capabilities that smaller models may not contain.
But I question whether indefinitely increasing model size while assuming GPUs will absorb every architectural and memory requirement should remain the default path.
Maybe it is time for the software, model, and hardware sides of the AI ecosystem to discuss the problem together.
Instead of asking only:
“How much larger can the next model be?”
perhaps we should also ask:
“How much of the previous model can we preserve?”
“How efficiently can we grow it?”
“What operations should AI hardware be designed around?”
and:
“At what point should AI stop being primarily a GPU workload?”