🧠 I built a novel triple-hybrid LLM (Mamba + Attention + 32-expert MoE) from scratch for ~$50 — Titan v1 complete, Titan v2 first cycle done, expanding dataset now

Show and Tell
1

Hi HuggingFace community :waving_hand:

I’m Mateusz — independent ML researcher from UK, operating under Project Inkblot. By day I work as a Manufacturing Controller, this project is something I’ve been building in my spare time, driven purely by passion and curiosity. Today I want to share something I’ve been working on for months: a full pre-trained language model designed, implemented, and trained entirely from scratch, combining Mamba SSM, Multi-Head Attention, and fine-grained 32-expert MoE in a single decoder-only architecture — to my knowledge the first to do so under 1B parameters.

:microscope: What is HybridMoE Titan v1?

A 450M parameter decoder-only language model that unifies three fundamentally different sequence processing paradigms in a single architecture:

Component Role
:cyclone: Mamba SSM (12 layers) Linear-time local context, O(T) compute
:telescope: Multi-Head Attention + RoPE (4 layers) Global context capture
:high_voltage: Fine-Grained MoE 32 routed + 2 shared experts, top-2 routing

This is not a wrapper, fine-tune, or modification of any existing model. Every component — architecture, training loop, data pipeline, checkpointing — written from scratch in PyTorch.

:building_construction: What’s actually novel here

1. Fixed-Shape Expert Dispatch

Every standard MoE uses torch.nonzero() which creates variable-length tensors incompatible with gradient checkpointing. Titan solves this with a fixed-shape dispatch that works with full gradient checkpointing across all 16 layers:

VRAM with gradient checkpointing: 5.84 GB
Without:                         >24 GB

This is the difference between requiring an A100 cluster and training on a single L4 GPU.

2. Loss-Free EMA Load Balancing

Standard MoE (Switch, Mixtral) adds auxiliary balance loss that competes with the LM objective. Titan uses EMA-tracked router logit biases — zero auxiliary loss, zero competing gradients:

Dead experts:     0 / 32  (entire training run)
usage_std:        0.019 – 0.031  (ideal uniform = 0.031)
Max expert load:  14.7%

Textbook routing health. No dead experts, no collapse.

3. Jamba-Style 1:3 Layer Interleaving

Layer  0:  [Attention+RoPE] + [MoE: 32 routed + 2 shared]
Layer  1:  [Mamba SSM]      + [SwiGLU Dense FFN]
Layer  2:  [Mamba SSM]      + [SwiGLU Dense FFN]
Layer  3:  [Mamba SSM]      + [SwiGLU Dense FFN]
Layer  4:  [Attention+RoPE] + [MoE: 32 routed + 2 shared]
...

Mamba handles local patterns in O(T). Attention captures global context every 4 layers. MoE provides specialized capacity exactly where global reasoning happens.

4. Single-Line Scalability

The entire model is parameterized through one config. Scaling from 57M to 1.3B+ parameters requires changing one integer:

config = HybridModelConfig(num_layers=8)   # Titan Tiny    57.7M
config = HybridModelConfig(num_layers=16)  # Titan v1     450.4M
config = HybridModelConfig(num_layers=24)  # Titan Mid    690.0M
config = HybridModelConfig(num_layers=48)  # Titan Large  ~1.3B

The 1:3 interleaving, expert counts, RoPE — everything propagates automatically.

:bar_chart: Training Results (Titan v1)

Config Value
Hardware Single NVIDIA L4 24GB
Instance AWS g6.2xlarge
Precision bfloat16
Effective batch 64 sequences (~65K tokens)
Peak throughput ~6,200 tok/s
Total tokens trained ~4.2B
Total cost ~$50

Validation Perplexity Progression

Step Val Loss PPL Notes
12,000 2.38 10.81 50-shard val set
20,000 3.47 32.31 :up_right_arrow: Expanded to 200 shards (Polish added)
26,000 3.38 29.26 Monotonic improvement resumes
42,850 3.30 ~27.5 Final checkpoint :white_check_mark:

The PPL increase at step 20K is from validation set expansion (50→200 shards, adding Polish text) — not model degradation. Within the 200-shard regime loss decreases monotonically.

:rocket: Titan v2 — First Training Cycle Complete :white_check_mark:

Titan v2 completed its first full pre-training cycle (3 epochs) with major improvements across every component:

Aspect v1 v2
Parameters 450.4M 464.5M
Tokenizer GPT-2 BPE 50K Custom BPE 64K (PL + EN + bio + code)
Polish support Fragmented (ą→2 tokens) Native single-token
Optimizer AdamW AdamW(Tested and Stable)
Training Single GPU Single GPU + gradient checkpointing
Data pipeline Runtime loading Pre-tokenized binary shards
Data quality Unknown Full corruption audit — 0 binary artifacts

Titan v2 Pre-Training — Completed Cycle

Start:          Fresh init, custom 64K tokenizer
Epoch 0:        PPL  87.87  ← cold start, new vocabulary
Epoch 1:        PPL  61.37  ← −26.5 points in one epoch
Epoch 2:        PPL  57.50  ← best_val_loss = 4.0523  ✅ FINAL

Total tokens:   ~1.8B  (3 epochs × ~0.6B per epoch)
Total steps:    27,981
Hardware:       AWS g6.2xlarge (L4 24GB) + benchmarking on A100
Duration:       6 days  (Jun 11 → Jun 17 2026)
Cost:           ~$40

Four-domain corpus: 45% FineWeb-Edu (English) + 18% Polish (articles + Wolne Lektury) + 20% biomedical (PubMed/clinical) + 17% Python/code.

Currently: Building an expanded ~4.56B token dataset for the next training cycle, targeting 9B total tokens — Chinchilla-optimal scale for 464M parameters.

:link: Links

The model weights, full architecture code, training scripts, and paper are all public on HuggingFace. Fully open source, MIT license.

:handshake: Looking for arXiv Endorsement

The paper is pending submission to arXiv cs.LG. As a first-time independent submitter I need one endorsement from an established arXiv author in cs.LG or cs.AI.

If you’re endorsed for cs.LG and find this work interesting — please reach out via HuggingFace messages. One endorsement is all I need.

The paper covers: architecture design, fixed-shape dispatch, EMA load balancing, training dynamics, and scaling analysis.

:red_question_mark: Happy to discuss

  • Why 1:3 interleaving over 1:1 or 1:7?
  • Fixed-shape dispatch implementation details
  • Training stability on a single GPU without loss spikes
  • The $50 compute budget — what’s possible and what isn’t
  • Roadmap to Titan Standard (~1B) and beyond

Built from scratch, one GPU, one researcher. Project Inkblot :fountain_pen:

2

Nice work! There is so much happening in the space right now. It is amazing. I wish I had time to interact with everyone working on cool projects such as this, while working on my own.

I am very interested in the details that you outlined and will engage when I get time. If no one else chimes in, do you want to expose a little bit on the topics you outlined, So I or others can engage easier.

I find with my projects I don’t get to actually play around with them much because I am always moving on to the next idea etc. Is it the same for yourself? or have you intereacted with it, used it day to day etc? If so, how does it feel/perform etc?

3

So sorry if answer will be to long but honestly was to many surprises on this project and dont know where to start answer hehe Let’s do from question about interacting with it — yes, quite a lot actually! Just ran a fresh inference test tonight across all four domains the model was trained on. Here’s the honest picture: What works already (after just 1.8B tokens, no SFT): Polish diacritics render perfectly — ą ę ć ł ń ó ś ż ź all clean, including Polish quotation marks „ ". This was the whole point of building a custom 64K tokenizer — GPT-2 fragmented Polish characters, Titan v2 doesn’t. Domain awareness is there — medical prompts produce real terminology (RNA-seq, differential gene expression, chromosome references, Fig. 2 citation format). It knows it’s in a scientific paper. Python syntax surface is correct — def, docstrings, imports, for loops — the structure is right even if the logic isn’t. What doesn’t work yet — and this is expected: Semantic loops (“future of the future of the future”) — classic undertrained behaviour, fixable partially with repetition_penalty without retraining. No real algorithmic reasoning in code — Fibonacci with no Fibonacci logic, factorial accepting a string. Pattern matching, not understanding. Polish text has an archaic 19th-century flavour with was bit funny— that’s the Wolne Lektury (classic Polish literature) dataset leaving its fingerprint. What’s next: First SFT attempt already happened on the 1.8B checkpoint — curious results, data collected. But the real milestone is tomorrow: Titan v2 hits Chinchilla optimum (~9.3B total tokens for 464M parameters) completing epochs 3 and 4 on the expanded dataset. That’s the point where the model should have seen enough data to actually justify its parameter count.
After that — a proper SFT run on a Chinchilla-trained base. The hypothesis is that instruction tuning on top of an undertrained model mostly amplifies existing loops, while SFT on a properly trained base should produce something meaningfully different. Tomorrow we find out if that hypothesis holds.
Either way, it’s data. That’s the fun part of doing this from scratch — every stage teaches you something the papers don’t mention

BEST REGARDS :grinning_face:

Mati83moni

4

This is a great answer, and honestly the honesty is the best part. “Pattern matching, not understanding,” “Fibonacci with no Fibonacci logic,” catching the 19th-century flavour as a Wolne Lektury fingerprint - that clear-eyed read on your own model is rarer than the model itself. Most people show the cherry-picked Polish output and stop.

A few things you clearly got right, worth pointing out: the routing health is genuinely good - 0 dead experts across a whole run with usage_std basically at the uniform floor is not easy, and the loss-free EMA-bias balancing is the right modern call (same family as the aux-loss-free routing the bigger labs moved to). And the fixed-shape dispatch to keep gradient checkpointing alive is the real unlock - that’s what turns “needs a cluster” into “$50 on an L4,” and it’s a more useful contribution than the architecture novelty itself.

A few things I’d reach for next, mostly so the numbers tell you the truth:
Report bits-per-byte, not PPL - especially v1 vs v2. Your v1 (~27.5, GPT-2 50K) and v2 (~57.5, custom 64K) perplexities aren’t comparable: a bigger vocab mechanically raises PPL for the same underlying quality, so v2 is almost certainly not a regression - it just can’t be read off raw PPL. BPB (or bits-per-char) is tokenizer-invariant and is the one cross-version curve you can actually trust. Reviewers will ask for this too.

Break the eval out per domain. With a 45/18/20/17 mix, one aggregate number hides which domains are learning. Per-domain BPB across checkpoints shows whether Polish, bio, and code are each improving or whether FineWeb is carrying it - and that directly informs your next data-mix decision.

The ablation that’s worth the most. The whole premise is the triple hybrid, so the figure you (and reviewers) want is: does Mamba+Attn+MoE actually beat Mamba-only and Attn+MoE-only at matched params and tokens? Even at your 57M config that’s a clean, cheap experiment - and it’s the difference between “I combined three things” and “here’s evidence the combination earns its keep.”

Your SFT hypothesis - make it a controlled pair. “SFT amplifies loops on an undertrained base but does something real on a Chinchilla base” is a genuinely good, testable claim. To attribute it cleanly, hold the SFT data + recipe identical and vary only the base (1.8B checkpoint vs the 9.3B one). Same SFT, two bases, side by side - that’s a paper figure on its own.

One gentle note on “Chinchilla-optimal.” ~20 tokens/param is compute-optimal for training a model that size - but for a small model you actually want to use, the modern move (LLaMA-style) is to train well past it, often 100–200+ tokens/param, because the model’s cheap to run and you’re buying capability, not minimizing training FLOPs. So 9B is a sensible checkpoint, not a ceiling - if the loops persist at 9B, more tokens (not just SFT) is a lever.

And one tiny scoping thing, as a friend: “first to combine Mamba+Attention+MoE” will get poked - Jamba’s right there, and you even call the interleaving Jamba-style. Your actual novelty - the fixed-shape dispatch, the loss-free balancing recipe, doing it from scratch under 1B for $50 - is the stronger, more defensible claim. Lead with that.

To your other question - yeah, exactly the same here: always three ideas ahead, rarely sit with my own models. The few times I force myself to actually use the thing, it’s humbling and clarifying in equal measure - you find the bug the metrics were hiding. Your inference-test-across-four-domains habit is already giving you that. Keep posting the honest read; it’s the most important part of the thread.

5

Quick clarification on the PPL numbers — I think there’s a misread in the post. The PPL ~57.5 for Titan v2 isn’t a final result — it’s a snapshot from Epoch 2 of an ongoing run. Every model starting cold with a brand new 64K vocabulary begins around PPL 100+ (Titan v2 opened at PPL 87.87 at Epoch 0, which is actually quite clean for a cold start). The post was showing the trajectory, not the destination.

Here’s where things actually stand as of today:

Cycle 1 (original dataset, ~1.8B tokens):

Epoch 0: PPL 87.87 ← cold start, new vocabulary

Epoch 1: PPL 61.37 ← −26.5 points in one epoch

Epoch 2: PPL 57.50 ← Cycle 1 complete

Cycle 2 (expanded dataset, 3.78B tokens/epoch):

Step 45,000: val_loss 3.8319 → PPL ~46.1 ← current BEST

Step 47,000: autoresume checkpoint — still training

Still actively converging. Target is ~9.3B total tokens (Chinchilla optimum for 464M params).

On all your technical points — you’re right across the board, and I’m taking each one seriously:

BPB instead of PPL — absolutely, and this is now on my list before the paper submission. The v1 (50K vocab) vs v2 (64K vocab) comparison in raw PPL is misleading and I should have flagged it myself. BPB is the right cross-tokenizer metric and reviewers will ask for it.

Per-domain eval — planned for the next checkpoint report. The 45/18/20/17 mix is a hypothesis I want to validate, not a result I’ve confirmed. Per-domain BPB across checkpoints will tell me whether FineWeb is carrying everything or whether Polish and bio are genuinely learning.

The ablation — this is the one I keep circling back to. Mamba-only vs Attn+MoE-only vs triple-hybrid at matched params and tokens, run at 57M scale. It’s cheap and it’s the figure the architecture claim actually rests on. I’ll run it.

SFT as a controlled pair — same SFT data, same recipe, two bases: 1.8B checkpoint vs the 9.3B one. That’s a clean figure and I’ll run it exactly that way.

On the Chinchilla framing — noted. 9B is a checkpoint, not a ceiling. LLaMA proved empirically that for small models you train well past compute-optimal because inference is cheap and you’re buying capability, not minimizing FLOPs. If the loops persist past 9B, more tokens is the lever.

On the novelty — fair and appreciated. Jamba is right there and I cite it. The defensible claim is the fixed-shape dispatch + loss-free EMA balancing + doing it from scratch under 1B for ~$50. Leading with the engineering contribution rather than the architectural combination is the stronger move for the paper abstract.

​I’ll post a full checkpoint update this weekend—I’m taking some time off from work to focus entirely on BPB numbers and the per-domain breakdown. Would genuinely love your eyes on the Titan v2 model card if you have a moment — it has the full training config, architecture details, and I just updated it with the corrected training progress.

This is the most useful single comment this project has received.

Thank you!!! :sparkling_heart:

Mati83moni

6

Hi Mateusz, this is a really useful build log. The parts that stood out are the repeated training cycles, the autoresume checkpoint at step 47,000, and keeping the work viable on L4/A100-level budget instead of a cluster.

I’m Rahul, founder of VaultLayer. VaultLayer is live now and helps individual builders run training/fine-tuning jobs on reliable, available, and affordable GPUs with monitoring, checkpointing, and recovery around the run.

If you want to compare it against the current AWS/L4 workflow for Titan v2 checkpoints or the 57M ablation runs, happy to share details.

Rahul
Founder, VaultLayer

7

I am not into the technicalities, but I am very interested in the possibilities that arise from smaller models that can be easily trained because IMO this enables the creation of specialised models for specific tasks.

My own interest is agentic coding, and it frustrates me to heck that to get decent results for a specialised task I need to use huge frontier models that know 50 human languages, the whole general knowledge of wikipiedia and capable of writing erotic poetry in iambic pentameter but they can’t write decent code first time.

Software Engineering is possibly a unique use case - it is probably the single most common use case, and it is also the one with the biggest base of: academic knowledge/best practice, practical knowledge/best practice and training data (github etc).

What I would like to see is a small model that has been trained to understand:

  • The most common human languages (or possibly just English as a starter since it is the most common language used by software engineers)
  • Modern Software Engineering fundamentals & best practices (programming language independent) i.e. UML, what makes good code, design patterns etc.
  • Programming languages themselves as Experts
  • Millions of examples of good quality code from Github
  • Dual free-text / structured responses approach so you can chat but it can also work in a structured way with harnesses specifically designed to work with it.

I know this is probably not the area of interest for OP, but I did want to chip in to see if there was any interest in this.

8

All the examples you cite interest me and are some of the things I have been thinking about. I have this idea in my head, floating around to, something along the lines of taking Gemma 4 MTP heads and finetuning or training to create only python for example.

I have created a platform currently that grafts the heads on to the model but uses a custom driver and passes them through an interceptor. It basically allows you to use them as small “individual” models that are then injected back into latent space.

I have many, many things I plan to try using this setup. for example creating small python to do actual real calculation or compute, tool calls, The possabilities are endless really.

I have previous systems that I exclusively use small finetuned models for everything combined with a small finetuned router model that sends requests off to the right model. I have used function gemma 270m and qwen3 0.5b. I finetune them for specific tasks such as grammar, conversational flavour, or looking for missing ] } ; brackets, (common mistakes etc that small local models drop alot during coding)

There is quite alot you can do with finetung/training tiny models and they work extremly well and extremly fast. Anything that I do in this area I will be posting on this board somewhere.

9

@KnackAU Obviously anything you decide to do for yourself is automatically OK - but this isn’t my vision which is much much much bigger than this.

My vision is for the AI industry to:

  1. Stop forcing everyone to use frontier models which are massively expensive to run and deplete the worlds resources whilst causing climate change.
  2. Create a new more flexible architecture for AI which enable specialised SLMs easily to be created which can run on consumer hardware to perform specific functions to high quality at low cost.

If I want to make a shelf for my home that requires me to sand it down to a fine finish and drill a few holes in my wall, I don’t have to go and rent time on a massive mobile drilling/grinding/sanding/milling machine that can do everything from drill an oil well to sand planks and everything else. I want to get out my own local sander and drill and do some sanding and make some holes. YES, I both own and use two specialised tools that are designed to do a specific job and do it well.

Obviously if I have a highly specialised requirement that is far from common and no one has created a consumer tool for it, then I might need to hire an expensive more generalised and flexible tool - but for common tasks, I should be able to buy an inexpensive tool that has economies of scale.

So, what I envisage in AI is the ability for people to buy (or download for free) an off the shelf SLM that is specifically designed for coding in English, adapt it for Python or PHP by adding a language module (that contains the AST rules) and set it going (like having a drill with a both a chuck and a sanding attachment).

Equally obviously the AI industry isn’t interested in doing this when all their investment is in monolithic frontier models which are hugely expensive to train - like most consumerism this needs to be driven by smaller specialised groups creating a solution that everyone wants.

However I don’t want a tool thrown together by a neighbour made from bits of wood and metal he had laying around either which doesn’t fit the drill holder or sanding attachment I already own - I want something that is supported and has a variety of attachments etc.

In other words, I want the SLM to be a community effort by a team of like-minded people who get together to produce something of high quality which is supported and has a future.

(Aside: I honestly don’t understand why there is so little team-work in the AI space to create an entire solution that can survive and be enhanced for decades, but instead individuals create a single specialised part for themselves that doesn’t get synergy and eventually disappears as if it never existed. OPEN SOURCE is well established in other areas of IT - so why not in the AI space.)

10

@ProtopiaUk I have designed and created such a system, it uses a blockchain style mechanism and a decentralized transformer mechanism that allows for distributed training and inference.

But, you asked very specific questions and I answered with very specific ways in which I had played around with those subjects and as they also pertained to the thread I thought them relevant. If you want to talk about other such things maybe we should create another thread or you can jump over to one of mine if you like.

in response to your aside:

It is a very peculiar phenomonen. I think it is because everything is happening so fast and everybody wants to be the one who comes up with the next breakthough, or because AI’s create this intelligent mirror that hypes everything up, and tells you that you are special and that you have made a truly world shattering breakthrough and that you should create a paper about it and tell the world and protect it because it is yours only.

My main goal in this area is to attempt to change this slowly, by attempting to nudge people towards working together, by attempting to show how you benefit when your work is looked at by everyone openly and honestly. By attempting to show that it is ok for your idea to be pulled apart, improved on, etc. I try to create my projects along the same lines, not as a hey look at this amazing thing I think I have done, but rather as a “hey I am attempting stuff and I am failing many, many times on the way, but I present those failures to you, so maybe you can get something out of it, or build on it and take it further”

11

@KnackAU Ray - I probably should have gone and looked at your contributions here to see what you had done, but I failed to do this earlier for which I apologise - however, honestly wasn’t pointing at you when I was commenting generally about a lack of teamworking.

I guess both you and @Mati83moni Mateusz essentially are doing fundamental experimental research - you have synergistic ideas on how to improve some basics and you do experiments to see how it works out. These are the essential building blocks for useable products and they are typically done by individuals. The biggest exception to this I can think of is CERN - but historically Faraday, Curie, Einstein, and thousands of other scientists worked this way, mostly taking other people’s ideas and building upon them by coming up with something new, but occasionally doing something completely new. Products, however, are typically engineered through team efforts - often mostly commercial and driven by a single CEO, but in recent decades also organically through team-based open-source efforts.

But, if you take a look on Reddit there are literally tens (or possibly hundreds) of people who have vibe coded a memory system for AI. Some of these might gain traction, but most will fall by the wayside, because it doesn’t gain many users, or because the author loses interest. In a scientific analogy these are like new life-forms coming into being - some succeeding in surviving but not evolving, and most dying off. Now imagine what the outcome might have been if these same developers had come together to work on a single memory project as a team - we would have synergy where ideas spark off one another, we would have evolution, we would have longevity, we would have a much much much better chance of achieving critical mass / virality. Now imagine extending that concept to an entire AI ecosystem - harnesses, memory, language models, infrastructure pipelines …

I would imagine my own situation is a good example of the consequences - I have a history of making significant contributions to several open source projects over the past 20 years or so - but I have not yet managed to start such a project myself. I have an idea for an open source project (synergising the ideas but not the code) from two other open source projects I have contributed to) but since getting the idea 6+ years ago I haven’t had the time to start coding it, even though my shower-time thinking has taken the ideas forward. More recently I started to think about doing AI Agentic Coding to get it off the ground, but discovered just how fragmented and the agentic coding world and AI world is. I get that it is so new and fast moving, but nevertheless the lack of use of e.g. UML as part of this and the unstructured free-text approach is astounding - not building on the wealth of academic and practical knowledge as a foundation of agentic coding is an IMO amateur vibe approach.

12

@ProtopiaUk I am pretty sure we are both on the same page. I did not take any offense by the way. But as I say, the only conclusion to the problem that I have come up with is to engage and interact with others who are doing the same thing that I am interested in, or that I am doing. I don’t try to push my idea’s, rather I try to engage in a way that perhaps gets them past a bump in the road or suggest ways to falsify their idea’s, experiments they can use to isolate if the results they are seeing are signal or cross-talk etc.

I started to look at it this way - Why would anyone use other people’s or other organisations products in a world in which they can create their own? This is why you see all these projects spring up and go nowhere, people not working together etc.

My solution is to take a step back and look at this from a higher perspective. Rather than try and get a team together to work on something niche in such a fast moving enviroment, why not engage with the people doing the stuff you want to see pushed forward and help them where you can. Show them the correct methods to move forward etc if they are unsure. Maybe I am just a lazy person in general, but I figure why do all the work if I can let others do the work for me?

So to that end. I simply work on what I am interested on working on and publish it, write about it. engage with people working on things that I can contribute to, by hopefully providing them with the methods to sucessfully push their project forward or let them know that their is interest in what they are doing and thet people can see them.

I think one of the biggest things people are finding out now is that using agentic coding and AI research is so fast and so powerfull, that a single human or even groups of humans simply can not keep up at the same pace. The key (in my mind) is to take a step back. Remember that just because you can do things fast doesn’t mean that you should. Sometimes you need to take your time, go back over what you are doing and refocus, because it is very easy to stray from your original path and if you are using agents for everything you will just push them down that path as well (or they will drag you down it).

So, anyway.. That is my thoughts on the matter. Just do your own thing. Provide it and the steps taken to get their so others can benefit from it and engage with the people on the same wave length or whom are doing something you want pushed forward, in any small way that you can. I think of this as kind of a meta group, rather then the previous methods of joining opensource projects and all the gate keeping or directionality that comes with that, you can work together, without actually working together and you can benefit from others working on similar projects, rather than have your project be the “ONE” that people focus on.

It’s a new world and you have to play by new rules if you want to contribute to it.

13

My personal interpretation of the lack of teamwork is:

  1. American rugged individualism - don’t rely on others when you can do it yourself.
  2. Desire for monetisation - also a trait that originated in the USA
  3. Selfishness - not just about hoarding for monetisation, this term can apply equally to believing that you are the only person experiencing this problem.
  4. Lack of more generalised knowledge - they install e.g. Claude Code and stay focused entirely within the Clause ecosystem, without much (if any) knowledge about how things work. So when they experience a problem and don’t already know of a solution, they assume that there isn’t one.

To a large extent I think that this happening here is simply a specific example of a more general growing societal problem of selfishness, greed, lack of empathy and compassion, short-termism and focus only on the immediate problem without looking at the generalised issue.

P.S. The American culture of rugged individualism supposedly derives from the Wild West - and it probably does but only in the sense of the cowboy movies and not the real-life “wild” west where people worked very much in teams to create communities necessary for survival.

14

Hmmm. What we are seeing right now is a huge number of AI adopters pulling back due to the costs or because the quality isn’t there. In other words its a bubble - and given that Anthropic and OpenAI are subsidising stuff to a massive degree, once full pricing kicks in after IPOs a lot of demand is going to evapourate.

I don’t think that this means that AI is dead - far from it - but it may mean that dominance by a few giant IS Corporates is going to fade away.

I think that the other problem people are going to face is one of code quality. There is a well understood architectural issue in human-developed IT whereby you start with a nice clean architecture and then bolt on new functions and bolt further stuff onto these bolt-ons etc. I suspect (without having done any research or collected any evidence - in other words a pure guess based on decades of IT experiences) that even if the code quality is good, the underlying architecture may not be so good. In other words, AI coding may result in large scale technical debt. To prevent this you need not only diligent code reviews, but also senior reviews of the underlying architecture - assuming that A) there is an underlying architecture conjured up by the AI and B) the architecture is sufficiently documented to allow someone to review it. In other words, quality AI coding is for the diligent and NOT for the lazy.

15

I agree with alot of what you say too, It’s all these things and alot more, far more than we can fit in replies. It is also extremely nuanced.

I agree 100% on the technical debt, the review etc and think you are spot on.

While I can see your angle, the only thing I would push back on is calling this an American thing. This is a human thing and while it might be interesting, intellectually to to look at or speculate at the causes, IMHO it is much smarter to simply accept reality and look at the solutions. Everyone needs to deal with what the world brings and roll with it in their own way. But as I said, we are cluttering up this poor guys thread now with all this :slight_smile:

16

Yes - it is more of a human thing, but it is most obvious in the USA though also noticeable in the UK, and in both countries it started getting worse at the same time (Thatcher / Reagan).

But also yes - we cannot fix society or really even influence it - but we might perhaps be able to make a difference in a much smaller area of AI.

17

:brain: Weekend Research Update — SFT Experiments, BPB Metric & New Pretraining BEST

Project Inkblot | Mateusz Piesiak | 26–28 June 2026

A quick personal note before diving in — I genuinely wish I had more time to participate actively in the discussion here. Life has been quite full lately: day job as a Manufacturing Controller, family, kids, the household, and everything else that comes with being a solo researcher doing this entirely on the side. I read every comment carefully and they matter a lot to me — I just do not always get to respond as quickly as I would like. This weekend report is my way of making up for the silence. Thank you for your patience.

:bar_chart: Pretraining Progress — New BEST Checkpoint

Since my last post the pretraining has continued on AWS g6.2xlarge (L4 24GB). Here is the full checkpoint history:

Step Val Loss PPL Notes
30,000 4.0508 57.5 Cycle 2 start (expanded dataset)
45,000 3.8319 46.1 Previous BEST
50,000 3.7447 42.3
53,950 3.7447 42.3 Autoresume STOP
55,000 3.6835 39.8
60,000 3.6361 37.9
65,000 3.5968 ~36.5 :trophy: New BEST

The model continues to converge steadily. At step 65,000 we have processed approximately 4.26B tokens (65,000 steps × batch 4 × grad_accum 16 × seq_len 1024). Chinchilla optimum for 464M params is ~9.3B tokens so we are at 46% of the way.

:microscope: Weekend SFT Experiments

This weekend I ran three SFT experiments. Here is exactly what happened, including the raw outputs.

Experiment 1 — Colab T4, 120 examples, freeze 10/16 layers

I started with the step 50,000 checkpoint (PPL 42.3) and a small dataset of 120 ChatML examples across 5 domains. Froze the embedding and layers 0–9, training only the top 6 layers. Used fp16 with gradient checkpointing on a T4 (15GB VRAM), 7 epochs at LR 3e-6.

The training curve looked promising on paper:

Ep 1/7  val=7.49  PPL=1796
Ep 2/7  val=5.79  PPL=326
Ep 3/7  val=4.57  PPL=96
Ep 4/7  val=3.77  PPL=43
Ep 5/7  val=3.35  PPL=28
Ep 6/7  val=3.15  PPL=23
Ep 7/7  val=3.07  PPL=21.6  ← saved

But the actual generation told a very different story:

Prompt: "Co to jest szlak mTOR i jaką rolę odgrywa w nowotworach?"

Output: 123466678910101011121314151718181819206070909090899090909190909094909090
        CCC C C C G G G H G G G M G G G A G G G I G G G I G G G I G G G I...

PPL 21.6 after 7 epochs was pure memorisation, not learning. The model reproduced numerical patterns (TPM values 1, 2, 3… 100, 200, 500, 1000) from the RNA-seq training examples. 120 examples × 7 epochs = 742 forward passes on the same data. Classic overfitting on a tiny dataset.

Experiment 2 — Colab T4, 120 examples, full unfreeze

Same 120 examples but this time all 16 layers unfrozen, 2 epochs.

Ep 1/2  val=7.49  PPL=1796
Ep 2/2  val=2.68  PPL=14.5  ← saved (interrupted)

Output:

Prompt: "Explain the mechanism of action of tyrosine kinase inhibitors in CML"

Output: /. / W W W W W W W W W W W W W W W W W W W W W W W W W W W W W W W W
        W W W W W W W W W W W W W W W W W W W W W W W W W W W W W W W W W W...

Catastrophic forgetting in 88 seconds. Training all 464M parameters on 120 examples erased 5.2B tokens of pretraining knowledge. The model replaced everything with a loop on a single token.

Experiment 3 — AWS L4, 4300 examples, structural MoE-aware freeze

Based on KnackAU’s earlier comment and my own architecture inspection, I moved from sequential layer freezing to structural freezing — freezing by component type rather than by layer index:

for name, param in model.named_parameters():
    freeze = (
        'token_emb' in name or
        any(f'layers.{i}.' in name for i in range(4)) or
        'router' in name or
        'routed_experts' in name or
        'shared_experts' in name or
        'shared_gate' in name
    )
    param.requires_grad = not freeze

The key decision here was freezing the MoE routers and experts. Why? Because the Expert Sparsity Analysis (done earlier this weekend) showed that the router learned domain-specific expert allocation with zero supervision:

Medical vs Python   overlap = 0.00  ← perfect domain specialisation
Medical vs Polish   overlap = 0.15  ← clear specialisation
Python  vs English  overlap = 0.11  ← clear specialisation

This emerged purely from pretraining on the mixed 4-domain corpus. SFT should not destroy this.

The split ended up at 278.6M frozen (embedding + first 4 layers + all MoE routing and experts) and 185.9M trainable (Attention QKV, Mamba SSM, Dense/SwiGLU, lm_head) — about 40% of the model.

The dataset was much larger this time: 4,300 ChatML examples total. 3,500 clinical reasoning examples with step-by-step analysis, 800 examples from MedDialog/MedQA/PubMedQA/ChatDoctor/iCliniq, and 50 of my own Polish medical examples covering RNA-seq omics and drug mechanisms.

Training curve:

Val BEFORE  9.92   PPL=20,257
Ep 1/3      train=0.46  val=0.060  PPL=1.1
Ep 2/3      train=0.04  val=0.042  PPL=1.0  ← BEST saved

Val loss 0.04 looked incredible but it was an artefact of data leakage. When building the dataset I mixed train_final.csv and val.csv into one pool and then split 95/5 randomly. The model was being evaluated on examples it had already seen during training. The actual generalisation is unknown.

The output confirmed the problem:

Prompt: "What is the mTOR signaling pathway and its role in cancer?"

Output: Guthrie struct char ******* Tibpectral vagus neutrophonsai Monaster
        sappveston CoQ provinandelion oracle alp powiatowszeostridochastic...

Root cause is the same as Experiments 1 and 2 — base model PPL 39.8 is simply too weak for SFT to produce anything useful. This confirms KnackAU’s point that Chinchilla is a checkpoint, not a ceiling. The SFT pipeline, dataset, and freeze strategy are all correct. The base model just needs more pretraining first.

:straight_ruler: Evaluation Results (step 50,000, before SFT)

MCQ Log-Likelihood

I ran a proper MCQ eval with full-text scoring and a BPE fix (the previous version had a merge bug where single-word continuations produced cont=0 tokens — fixed by stripping trailing space from prompt and including it with the continuation, which is the standard lm-evaluation-harness approach).

🟢 Easy    (5 questions)  3/5 (60%)  Paris ✅  print() ✅  Warszawa ✅
🟡 Medium  (5 questions)  2/5 (40%)  mTOR ✅  RNA-seq ✅
🔴 Hard    (5 questions)  0/5 (0%)   expected for base model pre-SFT

Tokenizer Compression — Titan 64K vs Mistral-7B vs GPT-2

Polish General    Titan 4.04 c/t  |  Mistral 2.47 (+67%)  |  GPT-2 1.93 (+110%)
Polish Medical    Titan 3.25 c/t  |  Mistral 2.30 (+41%)  |  GPT-2 2.07 (+57%)
English Tech      Titan 5.19 c/t  |  Mistral 4.84 (+7%)   |  GPT-2 5.05 (+3%)
Python Code       Titan 2.52 c/t  |  Mistral 2.77 (-9%)   |  GPT-2 2.22 (+14%)

The 64K Polish-optimised tokenizer gives +67% better compression for Polish text vs Mistral. This directly reduces compute cost for Polish-language inference.

Expert Sparsity — Router Specialisation across 4 MoE layers (L0/L4/L8/L12)

Medical (top-5)  E24:24  E29:23  E16:21  E23:21  E3:21
Python  (top-5)  E9:52   E28:50  E30:48  E27:47  E10:47
Polish  (top-5)  E24:26  E4:25   E30:23  E29:21  E22:21
English (top-5)  E23:18  E3:18   E30:15  E13:14  E29:14

Domain specialisation (IoU of top-5 expert sets)
  Medical  vs Python   overlap = 0.00  ✅ Perfect specialisation
  Medical  vs Polish   overlap = 0.15  ✅ Clear specialisation
  Python   vs English  overlap = 0.11  ✅ Clear specialisation
  Polish   vs English  overlap = 0.25  ✅ Clear specialisation

The router learned domain-specific expert allocation with zero supervision — this emerged purely from pretraining on the mixed 4-domain corpus.

:triangular_ruler: BPB Evaluator — responding to KnackAU’s feedback

As KnackAU correctly pointed out, PPL is tokenizer-dependent and cannot fairly compare Titan v1 (50K vocab, PPL ~27.5) with Titan v2 (64K vocab, PPL ~36.5).

Bits Per Byte (BPB) is tokenizer-invariant:

BPB = total_NLL_nats / (ln(2) × n_bytes_UTF8)
    = log₂(PPL) / avg_bytes_per_token

Implementation:

@torch.no_grad()
def compute_bpb(model, tok, text: str) -> dict:
    n_bytes  = len(text.encode('utf-8'))
    ids      = tok.encode(text, add_special_tokens=False)

    input_ids = torch.tensor([ids]).to(DEVICE)
    out = model(input_ids)
    logits = (out[0] if isinstance(out, tuple) else out)[0]

    log_probs = F.log_softmax(logits[:-1].float(), dim=-1)
    targets   = input_ids[0, 1:]
    nll_per_token = -log_probs[range(len(targets)), targets]

    total_nll = nll_per_token.sum().item()
    bpb = total_nll / (math.log(2) * n_bytes)
    return {"bpb": bpb, "ppl": math.exp(nll_per_token.mean().item())}

The evaluator tests across 5 domains (English General, Polish General, Medical English, Medical Polish, Python Code). BPB results for v1 vs v2 comparison will be published once v2 reaches the target PPL.

Interpretation scale:

< 1.0    excellent (better than human for English)
1.0–1.5  good
1.5–2.0  average
> 2.0    undertrained

:package: SFT Dataset — Ready for Next Run

The dataset for the next SFT attempt is built and waiting:

sft_master_v2.jsonl — 4,300 high-quality ChatML examples (7.6MB)

  • 3,500 medical examples from train_final.csv — clinical reasoning with step-by-step analysis, system prompt “You are an expert medical AI assistant”
  • 800 examples from val.csv — MedDialog, MedQA, PubMedQA, ChatDoctor, iCliniq, system prompt “You are an empathetic AI health companion”
  • 50 of my own Polish medical examples — RNA-seq omics analysis, drug mechanism explanations (PL/EN), patient education in Polish

Validation split is 197 examples in a separate file (sft_val_v2.jsonl) with no mixing with train. Lesson learned.

:triangular_ruler: SFT Architecture for Next Run

For the next SFT run (once pretraining reaches PPL ~30-32) the plan is the same structural freeze that was validated in Experiment 3:

# Structural freeze — preserves MoE domain specialisation
freeze = (
    'token_emb' in name or
    any(f'layers.{i}.' in name for i in range(4)) or
    'router' in name or
    'routed_experts' in name or
    'shared_experts' in name or
    'shared_gate' in name
)
# Frozen    278.6M (embedding + 4 layers + all MoE routing)
# Trainable 185.9M (Attention, Mamba, Dense, lm_head)

Target config is LR 3e-6, 3 epochs, GRAD_ACCUM 4, fp32 on L4.

:world_map: What’s Next

Current   step 65,000  |  PPL 36.5  |  4.26B tokens
Target    step ~92,000  |  PPL ~33-34  |  ~6B tokens (early stop considered)
          step ~164,490  |  PPL ~30-32  |  9.3B tokens (Chinchilla optimum)

After pretraining reaches the target:

  1. BPB eval — v1 vs v2 cross-tokenizer comparison
  2. SFT with sft_master_v2.jsonl (4,300 examples, structural freeze)
  3. Per-domain eval post-SFT
  4. Architecture ablation at 57M scale (Mamba-only vs Attn+MoE vs triple hybrid)

:light_bulb: Key Lessons This Weekend

  1. SFT requires a competent base. PPL > 35 means garbage output regardless of dataset quality or freeze strategy. This is not a bug — it is physics.

  2. MoE expert routing is precious. Freeze routers and experts during SFT. The domain specialisation (Medical vs Python overlap = 0.00) took billions of tokens to learn and can be destroyed in minutes.

  3. Data leakage is subtle. Mixing sources before splitting produces optimistically low val loss that means nothing. Val set must be built separately before any train data is assembled.

  4. BPB > PPL for cross-model comparison. Larger vocab mechanically inflates PPL. The models are more comparable than the raw PPL numbers suggest.
    *Update comming soon :grin:

  5. 120 examples is not enough for 464M params. Minimum viable SFT for this model size is ~500-1000 high-quality instruction pairs. The 4,300-example dataset should work.

So yee that happen on weekend guys have nice reading hehe :slight_smile: and yee
Any Question Just Message me Happy to Answer For all Question
@ProtopiaUk @KnackAU @vl-dev @John6666

18

@KnackAU I just realized we’re on the exact same wavelength! I’ve been meaning to circle back to something you mentioned in your earlier post — that idea about taking Gemma MTP heads and finetuning them for specific tasks like Python generation. That really stuck with me because I’ve actually been down a very similar rabbit hole, just from a slightly different angle.

A while back I took Google’s FunctionGemma-270M and fine-tuned it specifically for on-device mobile function calling — things like send_email(), create_contact(), create_calendar_event(), show_map(), the kind of everyday phone actions people do constantly. Used LoRA, trained it on a Colab T4 in about 30 minutes, and managed to push accuracy from ~58% (base model) to nearly 85%. The whole thing quantizes down to 272MB INT8 and runs in 1-3 seconds on a phone with zero cloud calls.

Here’s the repo if you’re curious: Mati83moni/functiongemma-270m-it-mobile-actions · Hugging Face

What caught my eye even more is your Aiden project — a physical device that watches the screen over HDMI and controls the phone via USB HID. That’s genuinely clever because it sidesteps all the jailbreak/root/ADB problems. And it made me think — something like my functiongemma could potentially serve as a lightweight brain for exactly that kind of setup. You have the hardware layer that sees and controls the phone, and this model could be the part that takes a natural language command like “send email to my boss with today’s report” and translates it into the right function call, all running locally on the device itself.

I see a lot of overlap between what we’re both chasing — small specialized models that actually do useful things on real hardware instead of burning $50/day on API calls to frontier models. Your head-grafting platform sounds fascinating too, especially the interceptor concept. Would love to hear more about how that works in practice.

Also completely agree with what you said about engaging with people doing similar things rather than trying to push ideas — that’s exactly the spirit. Different approaches to the same problem space, and I think we’re both finding that these small Gemma-class models are way more capable than people give them credit for when you train them right for a specific job.

Would be great to collaborate on something down the line :call_me_hand:

19

We do very much align in our thinking. I to used FunctionGemma for mobile, I also finetuned it for use in grammar correction, as a small model router, it is extremely useful and I love playing around with small focused models. I used Termux and Shizuku to get around most things. I agree with you that the Aiden project is a great idea, but..

I wish I could say that the Aiden project was mine. But that is actually NatalieY’s project. I just discussed it with her on her post. I have considered similar projects myself and I have plenty of MCU’s to play with, I can very easily whip up some esp32 or rp pico code if you want to do something opensource or you can see if NatalieY wants to do something or check out her code.

On the topic of the Gemma MTP heads. I have posted in my thread, but it is flagged and will probably remain so for days if things progress as they have previously. I will post some of it here for you and just let you know that it has progressed to working extremely well, passing a lot of trials and triubulations. I am now working on the “Telepathy” side which is model to different arch model latent space injection and I have that working pretty well, tidying and finishing it all up now. All reciepts, tests and code is on my repo under the mtp-draft-transcode branch if you want to check any of it out.

The Latent Interceptor framework:

Draft body = the shared latent processor. The finetuned 4-layer draft, vocab head ripped off. It runs once per intercept, producing a 1024-d latent. Because there’s no 262k projection, it’s ~ms and CPU/Hexagon-pinnable (your <2 ms point holds — the body is tiny; the vocab matrix was the whole cost).
A registry of specialized heads tapping that latent, each finetuned for a task, each staying in latent space:

  • Action head (HID->A): the KAIROS NO_OP/KEEP/FORGET/E2B/ACTION gate.
  • Memory head (HID->63-byte C2 Spinor): writes MEM-OKF directly from the latent — the curator’s ADMIT path, no tokenization.
  • Tool head (HID->32-tool MCP logits): fires the harness decorator (E2B python for the strawberry-class problems) from a latent trigger.

Latent injection (return path): tool result → cyclotomic-ring residue → gemma4_kv_inject into the target KV ring. The model feels the result, never reads it.

So the heads are the routers; the body is the shared manifold they all read. One body pass, many latent destinations — that’s the framework, and it’s extensible to anything (the possibilities are, as they say, endless).

Pivot → Latent Interceptor → the draft repurposed as a latent-routing framework. Scaffold done: contract (shared body + action/memory/tool head registry), 5-action space grounded in the curator’s real ops, SP_LI_CAPTURE, the probe trainer, and a baseline that classifies the latent at 1.000 (mechanism proven — routing without tokenization is real).

The full closed loop works end to end:

Telepathy (latent→latent between models) — excellent endgame, one hard constraint. The right abstraction(LatentBridge{src,dst,adapter,dims,scale,basis,flags} + gemma4_kv_inject), and the framework just built is its substrate.

event: "count letters in strawberry"
TOOL HEAD fired (latent->tool id): PYTHON          <- the latent routed to the right tool
tool ran -> result = "3"                            <- real python subprocess executed it
result injected into KV ring (6 tokens)             <- return path, no re-prompt
model continues: "[tool name] count_letters [tool input] strawberry [tool output]"

Event → latent → Tool Head (PYTHON) → fire real tool → result “3” → return-path inject → model continues — no tokenizer in the decision or the return, only the final continuation is text. That’s the capstone: the complete latent-native agent loop in one call.

100% agree with you on the Gemma models. Google seems to have dropped some impressive stuff lately and hardly anyone takes them seriously. Google just drops them and basically leaves them for the community to figure out exactly what they are. They are hugely underrated.