Thank you for the amazing results! Seeing the +20pp improvement at the 100% depth is incredibly validating, alongside the stability at 0% and 75%.
Regarding the 25% and 50% depths, I am certain it’s because the framework is overshooting. The masking protocol is applying massive-context suppression to a relatively small token pool, as we designed it to tackle hallucination cascades at a much larger scale than this specific test.
To help calibrate this, I’ve just drafted Addendum E (Scalable Masking & Dynamic Thresholding) to allow the framework to dynamically adjust to smaller token pools.
Thank you so much for this continued collaboration. I’m incredibly excited for what’s possible here!
Well .. it seems rewriting the code to test only your model worked nice!
It seems the latent → active distribution part of LSA is working during the new, final training! Soon (a couple of hours) I will have the appropriate results that cover your full model, not intertwinned with mine! (that, of course, will make the final result even better, I believe, so we can have a “Mixture of Architectures” that can be stacked one over the other to build a stronger model from a base one!
This is absolutely incredible to see! Seeing the physical proof that the framework is successfully identifying the noise and pushing 76.4% of the context into a latent state validates the entire foundational math. The fundamental masking engine is working exactly as designed!
Now that we have empirical proof of the raw LSA distribution in action, implementing the dynamic thresholding from Addendum E is going to be an amazing addition. It will give the framework the exact precision dial it needs to scale that masking perfectly, no matter the size of the token pool.
I also absolutely love your vision for a ‘Mixture of Architectures’ down the line. Stacking your optimization layer on top of this routing foundation is going to create something truly powerful. Thank you for your relentless work bringing this to life!
Results are up!
These are the results right now for both (mixed with TIS and stand-alone and using your FULL model power):
First, NIAH testing:
And this is for LITM benchmarks:
Next points to prove/test are ready (because the above was ONLY with a partial model implementation!!):
In the coming days I will be adding one by one (or a bunch at once) the missing pieces and you will start to see how the architecture evolves! It seems your ideas are on the right path, don’t you think?
The empirical data speaks for itself—we are undeniably on the right path. Seeing the foundational theory translate into a physical, gapless baseline is incredible.
I am very much looking forward to watching the rest of the architecture come online as you integrate the remaining pieces. Thank you for the phenomenal work bringing this to life.
I quickly over looked error on my recent addendum I foolishly labeled it E when its supposed to be H, I’ve fixed the mistake apologies.
No problem! You have a private message! Please, read it! We cannot fill-up these threads with things we can just discuss in private. It’s better we just leave this (or a new thread) to actually present results/discuss roadblocks with the community. Don’t you agree ?
Great breakdown of the Softmax denominator bloat — I hit the same failure from the systems side while building APKs in Termux on mobile.
I’ve been prototyping a different angle to the same problem: RTTC (Real-time Telemetry Channel) + Meta.md
Instead of masking inside the attention calculation, it solves it structurally:
- RTTC = independent, lightweight channel for real-time events only. Active context stays clean and fast.
- Meta.md = 1 small file in project root with essential facts. AI reloads only that at startup, instead of 50k tokens of history.
- Result: no context rot, no I/O latency from dormant vectors, and if data didn’t come through the channel, the model says “I don’t know” instead of hallucinating.
It runs fully offline on mobile hardware (tested on Android/Termux) with no external dependencies. Spec + working implementation here: masterscorpio1974-boop (MASTER S) · GitHub
Curious if you considered a structural separation like this as an alternative to M_state masking for the prefetch problem.
