Removing the embedding from my embedding: a byte transformer with a 0-parameter input layer (25M, single RTX 4070)

[ggunio]

Hi everyone, a follow-up — and a slightly absurd experiment that worked.

Since the last post, the substrate ablation toolkit shipped inside the encoder (hsl_embedding.ablation — capacity-matched hsl / learned / random / permuted arms, as discussed in this thread). While running the full A/B I got curious about a stranger question:

what happens if I remove the embedding from my embedding?

I.e. feed the frozen 27-D signal features straight into the transformer through a fixed zero-pad — no tokenizer, no embedding table, no learned input projection. Zero learned parameters at the door.

It runs!!

input front door	text bpb	caption bpb	learned input params
zero (frozen features, zero-pad)	2.456 ±0.027	1.526	0
learned projection on same features	2.443 ±0.014	1.402	~125k
plain learned byte embedding	2.773 ±0.076	2.556	~132k

(2 seeds, same lean ~25M body, same 3-modality byte mix, fixed 3000-step budget. Doubling bytes-per-slot (K=16, half the prefix positions) holds text bpb at 2.455.)

Reading this honestly: not “embeddings are beaten.” At this small budget the frozen substrate already carries what a learned front door would have to learn, and a plain learned byte embedding doesn’t get there in 3k steps — it may well close the gap with a longer schedule. One consumer GPU, small body, the table is the claim.

So I shipped it as a tiny package plus a live proof model:

• pip install hsl-embedding-zero — the zero door as a drop-in module (GitHub, MIT, DOI 10.5281/zenodo.20643551)
• HoLo_ZeRo — a 25M model trained entirely behind the zero door (the casing is the signal: HoLoZeRo = 10101010): weights · live demo (byte generation + the 27-D cosmos it literally reads)

If you’re curious, poke it and tell me where it breaks.

(Also since last time: HoLo 6.5.1 finished its 3-stage curriculum — weights public, knowledge-grounding gap grew 0.001 → 1.835 across training, full numbers in the repo.)

[John6666]

Hmm… maybe something like this?:

---

I think this makes the earlier HoLo/HSL line much sharper.

I would read this less as “removing embeddings” in the broad, final sense, and more as a front-door experiment:

how much input geometry has to be learned before attention sees the bytes?

That framing makes the result easier to place. The interesting claim does not seem to be simply “no embeddings.” There is already a close embedding-free byte precedent in Shaham & Levy, Neural Machine Translation without Embeddings, where UTF-8 bytes can be represented with one-hot vectors instead of a learned embedding layer.

The sharper claim here seems to be the combination:

zero learned input door + K-byte packing + deterministic HSL geometry

That puts HoLo_ZeRo between a few existing lines of work:

Existing thread	What it already covers	What HoLo_ZeRo adds
Shaham & Levy	Embedding-free byte input via one-hot bytes	A packed, shorter-than-raw-byte-position input using deterministic HSL geometry
CANINE / Charformer	Token-free character/byte modeling with downsampling or learned subword formation	A zero learned input door rather than a learned front-end
MEGABYTE	Byte patching with local/global multiscale modeling	Fixed HSL feature packing instead of learned/local patch modeling
Byte Latent Transformer	Dynamic byte patches and FLOP-controlled byte scaling	Deterministic K-packing as a simpler fixed front door
fixed / random / compressed feature maps	Learned embeddings are not the only way to create vector input	Structured byte-signal geometry as the fixed map

So the way I would summarize the research position is:

Shaham & Levy asks: can bytes avoid learned embeddings?
Charformer / MEGABYTE / BLT ask: how do we shorten byte streams?
HoLo_ZeRo asks: can we do both with a deterministic byte-signal front door?

That is a pretty clean question.

The next useful artifact: a failure-boundary map

Since you explicitly asked where it breaks, I think the most useful next artifact may not be another single-winner table, but a failure-boundary map.

Something like:

Boundary	Question	Why it matters
Schedule boundary	Does the plain learned byte embedding catch up after longer training?	Separates “zero door has better cold start” from “zero door has better final behavior.”
Scale boundary	Does the gap persist at 50M / 100M / 300M bodies?	Larger bodies may learn input geometry that a small model cannot.
K boundary	What happens across K=4 / 8 / 16 / 18?	K controls sequence density versus fine-grained binding.
Modality boundary	Does it work equally for text, decoded audio, decoded image bytes, and compressed-like bytes?	HSL geometry may be more natural for signal-like bytes than semantically arbitrary byte streams.
Geometry boundary	HSL vs learned / random / permuted / raw-bit controls	Tests whether the structured geometry matters, not just capacity or invertibility.
Binding boundary	Does the binding gap survive harder negatives?	Separates real cross-modal association from shortcut learning.
Implementation boundary	Does tail padding / streaming / byte recovery behave safely under odd lengths?	K-packed byte models can get misleading results if bytes are silently dropped or shifted.

That table would answer the exact question I think many readers will have:

where does the zero door stop behaving like a learned input door?

K looks like the main design knob

The K-sweep seems especially important to me.

K does several things at once:

Increasing K does this	Benefit	Possible cost
Reduces the number of attention positions	Lower input-side attention cost
Packs more local bytes into one slot	Higher local byte density
Makes each slot cover a wider byte span	Coarser temporal / modal alignment
Preserves zero learned input params	Keeps the front door fixed
Makes slot attribution more complex	Harder to tell which byte/channel drove a decision

So I would treat K less like a normal hyperparameter and more like a design knob:

sequence density versus fine-grained binding

The K=16 result is interesting for exactly that reason. If text/caption bpb holds up while binding softens, that suggests a useful compression/alignment frontier rather than just “K bigger is better” or “K smaller is better.”

A nice future plot might be:

K	text bpb	caption bpb	audio→caption binding gap	image/audio/text retrieval	tokens/sec or bytes/sec	VRAM
4
8
16
18

That would make the trade-off visible.

Short-term experiments that seem most useful

If I were trying to poke this in the most useful way, I would prioritize these:

Test	What it would clarify
Longer schedule for learned byte embedding	Whether the learned embedding is just slower to warm up.
Multi-seed runs	Whether the 25M / 3000-step gap is stable or seed-sensitive.
Same-FLOP / same-wallclock reporting	Whether K-packing improves the compute-quality frontier, not just the loss table.
HSL vs learned / random / permuted controls	Whether HSL geometry matters beyond capacity or invertibility.
K-sweep	Whether compression and binding pull in different directions.
Hard-negative binding	Whether the audio/image/text relation survives more difficult mismatches.
Tail and odd-length tests	Whether K-packing is safe under non-divisible byte lengths.
Streaming path tests	Whether the AR path behaves consistently with the packed prefix path.

The cleanest short-term question, to me, is:

At what training budget does a learned input door catch up, if it catches up at all?

The second cleanest is:

At what K does sequence compression start to damage binding?

Those two together would already say a lot.

Geometry controls

The geometry controls seem central.

I would want to see the same body and data under something like:

Input door	What it tests
zero HSL	The proposed structured deterministic geometry
learned projection over HSL	Whether early learned mixing helps
plain learned byte embedding	Standard learned byte identity baseline
raw bit features	Whether simple bit identity is enough
random fixed features	Whether fixed capacity is enough
permuted HSL LUT	Whether the HSL value geometry matters
learned byte projection with same dimension	Whether the model can learn equivalent geometry itself

The permuted HSL control seems especially useful. If it keeps marginal feature statistics but breaks byte-value adjacency geometry, then the comparison is more informative than just random features.

A rough interpretation table could look like this:

Result pattern	Possible interpretation
HSL > permuted ≈ random	HSL geometry likely matters.
HSL ≈ permuted > random	feature distribution/capacity may matter more than value geometry.
HSL ≈ learned projection ≫ learned byte embedding at short schedule	fixed geometry gives a cold-start advantage.
learned byte embedding catches up at long schedule	zero door may be a sample-efficiency / early-training advantage rather than a final-performance advantage.
learned projection over HSL wins consistently	HSL features help, but early learned mixing still matters.
all fixed variants collapse on harder data	the zero-door effect may depend on toy-scale or signal-like structure.

That kind of result table would make the claim much easier to read.

Modality boundary

One place I would expect different behavior is modality.

HSL geometry should be more natural when byte values have local numeric meaning. That is true for decoded signal-like data, but less obviously true for text or compressed media bytes.

Input type	Expected HSL fit	Reason
decoded grayscale pixels	high	nearby byte values often mean nearby brightness
PCM / μ-law audio	high to medium	byte values often relate to amplitude-like quantities
rasterized numeric signals	high	byte values often preserve numeric locality
UTF-8 text	medium to low	numeric byte adjacency is not the same as semantic adjacency
compressed image/audio/video bytes	low	codec structure and entropy coding dominate
encrypted/random bytes	near zero	byte adjacency has no semantic meaning

So I would not expect one global answer to “does zero door work?”
I would expect a boundary map by data type.

That could be a useful long-term table:

Modality / encoding	zero HSL	learned projection	learned byte embedding	random/permuted	Notes
UTF-8 text					byte identity may matter more than numeric adjacency
grayscale raster					value geometry should help
μ-law audio					amplitude-like structure may help
decoded video frames					likely K/alignment-sensitive
compressed bytes					useful negative control
shuffled/random bytes					sanity check

This would also help avoid overgeneralizing from the current mixture.

Binding probes

The binding result is probably where I would be most careful.

For text bpb, K-packing can be judged fairly directly. For cross-modal binding, I would want harder negatives.

Possible hard negatives:

Probe	Why it helps
same-class wrong-instance	Reduces class-label shortcuts
same-length caption mismatch	Reduces length cues
entropy-matched caption mismatch	Reduces byte-distribution cues
shifted audio/video windows	Tests temporal leakage
image-only / audio-only / image+audio ablation	Shows which modality contributes
top-k retrieval over captions	Easier to interpret than bpb alone
cross-dataset transfer	Tests whether binding survives outside the original toy distribution

If K=16 preserves caption bpb but weakens binding, the hard-negative setting may be the best way to see whether K is losing fine-grained alignment or just losing an easy shortcut.

Fixed channel addresses are also an interpretability opportunity

One thing I like about the zero-door design is that the channels enter unmixed.

That is not only a parameter-saving trick. It may make the first learned layer unusually inspectable.

For example, one could ask:

Analysis	What it might show
first-layer attention by feature group	whether Δ / Δ² / Fourier / phase-like channels are used differently
modality-specific channel usage	whether text, audio, and image bytes rely on different feature groups
K-slot byte-position attribution	which byte positions inside the packed slot matter most
learned projection comparison	whether the learned projection rediscovers similar channel mixing
feature ablation during eval	which channels matter after training

This might become one of the cleaner advantages of the zero-door setup: the model has fewer learned parameters before the first attention operation, so attribution at the input boundary is less hidden.

Longer-term: separate input door from output head

Longer-term, I would also separate the zero input door question from the output-head question.

Shaham & Levy is relevant here because that work discusses replacing embedding layers with one-hot byte representations in the first and last layers. HoLo_ZeRo, as I understand it, is mainly about the input front door.

Those are related but not identical questions:

Question	Why separate it
zero input door	asks how bytes should enter the model
output byte head	asks how predictions should be parameterized
weight tying	changes meaning when there is no learned input embedding table
HSL-aware output geometry	possible future direction, but separate from the present claim
AR streaming path	may interact with output-side design

I would keep the current claim focused on the input door, but maybe mark output-head structure as a later research branch.

Small implementation note

I also like the explicit tail handling.

In K-packed byte models, silent byte loss would be an easy source of false confidence. Making pad/drop behavior explicit is a small but important engineering detail.

Bottom line

My rough read:

This is not just an “embedding removed” experiment. It is more specifically a test of whether a deterministic HSL byte-signal geometry can replace the learned input door while also reducing raw byte-position length through K-packing.

The next most useful thing may be a failure-boundary map:

1. when learned embeddings catch up;
2. when larger models erase the advantage;
3. when K starts hurting binding;
4. which modalities benefit from HSL geometry;
5. whether HSL beats random/permuted/learned controls;
6. whether hard-negative binding still holds.

If that map looks good, then the result becomes much more than a neat parameter-saving trick. It becomes a concrete design rule for where a fixed byte-signal front door is useful.

[ggunio]

This is exactly the framing I was missing — thank you. “How much input geometry has to be learned before attention sees the bytes” is a sharper question than “no embeddings,” and the prior-art table places it correctly. I’m adopting the failure-boundary map as the actual roadmap, not another single-winner table. Here’s where the boundaries already stand, and where they’re still blank:

Already measured:

• Geometry boundary (same 25M body, 3-modality mix, 2 seeds, via hsl_embedding.ablation):

input door	text bpb	learned input params
zero HSL	2.456	0
learned projection over HSL	2.443	~125k
plain learned byte embedding	2.773	~132k
random / permuted HSL	2.44–2.45	0

The interesting wrinkle: random/permuted land near zero-HSL on text bpb but lose the cross-modal binding gap — which points exactly at your “HSL ≈ permuted > random ⇒ distribution matters more than value-geometry” interpretation for text, while value-geometry shows up in binding. So the answer is already modality-dependent, as you predicted.

• K boundary — your prediction was correct. K=16 (half the raw-byte-position length) holds text bpb (2.48→2.48) while the audio→caption binding gap softens (0.063 → 0.042). Compression and fine-grained binding pull in different directions, exactly as you framed it. K up to 18 stays at 0 learned params on dim-512.
• Implementation boundary — tail handling is explicit (pad/drop), no silent byte loss; an adversarial pass caught a real bug where a zero-door checkpoint silently loaded random Linear doors, now gated by a round-trip test.

Still blank — and the order I’ll fill them, taking your “cleanest question” first:

1. Schedule boundary (does plain learned embedding catch up with a longer budget?) — this separates “better cold-start” from “better final,” and I agree it’s the single most informative one.
2. Binding boundary with hard negatives (same-class wrong-instance, length-/entropy-matched mismatch) — to tell real association from a shortcut.
3. Scale (25M → larger), then the modality boundary by data type (decoded pixels / μ-law vs UTF-8 vs compressed/encrypted as a negative control), which your numeric-locality argument predicts should split.

Two of your points I want to flag as already on my path: the unmixed fixed channels as an interpretability handle (Δ/Δ²/Fourier/phase enter at fixed addresses, so a first-layer feature-group attribution + eval-time channel knockout is cheap), and keeping the input door separate from the output head — the current claim is input-only; HSL-aware output geometry is a later branch.

I’ll report these as boundaries, including the ones where the zero door stops behaving like a learned door — that’s the useful result either way. This is “it works, and here’s where it breaks,” not a superiority claim. Genuinely grateful for the map.

[ggunio]

A bigger update — data for several more of the boundaries you mapped, plus the places it breaks. (Geometry/K/implementation were in my last reply; this adds modality, schedule, grounding, and a first sensor pass.)

One framing note up front so the numbers are comparable: these come from three different runs, not one, and I’ll label which is which — a per-byte text/AV checkpoint (knockout + schedule), a new 61M I/O-symmetric multimodal model (grounding + the larger curriculum), and a tiny single-seed sensor probe. The input door is the published hsl-embedding-zero package in all of them (0 learned input params, verified bit-identical to the frozen substrate).

1. Modality boundary — channel-knockout matrix (per-byte checkpoint). Because the substrate enters at fixed addresses (Δ at dims 0–7, Δ² at 8–15, boundary at 16, Fourier at 17–24, phase at 25–26), I can zero a channel group at eval time on the trained checkpoint and read each modality’s reliance — no retraining. Δbpb when a group is knocked out (higher = more load-bearing):

channel group	text	image	audio	caption
Δ (dxor)	1.30	0.61	0.71	0.33
Δ² (d2xor)	2.19	0.11	0.21	0.27
boundary	0.01	−0.01	−0.05	0.01
Fourier	6.99	0.29	0.62	1.45
phase	0.18	0.47	0.38	−0.05

It splits the way your numeric-locality argument predicted: text/caption lean hard on Fourier, image leans on Δ+phase, audio spreads over Δ+Fourier+phase. The boundary channel is ~0 everywhere — the honest negative: at a fixed patch size the model has no reason to consult “where a unit ends,” so it idles. I’d expect it to wake under adaptive (content-determined) slotting — now a falsifiable prediction rather than a hope.

2. Modality boundary — extended to real sensor bytes (single-seed probe). You predicted HSL geometry should suit signal-like data more than symbolic text. I serialized real radar / lidar / depth (KITTI, RadarScenes, RaDICaL ADC) through the same frozen substrate with zero per-modality engineering. Next-byte bpb (dim512/d4/K8, single seed, 3k steps, not converged — text/radar still descending):

	HSL (0 params)	learned (78k)	raw scalar
text	2.82	2.86	5.98
radar	4.54	4.65	6.62
lidar	0.358	0.358	1.31

HSL ≈ learned across all three — every |Δ| is within single-seed noise, so I read it as “on par,” not ahead (on lidar the learned door actually edges HSL by 0.0003). raw scalar is far worse everywhere. Cross-sensor binding also works: real radar ADC → camera depth, 4-way distance quartile 0.84 (chance 0.25, non-circular). Where it breaks: on that absolute-distance task the substrate (0.84) loses to a raw range profile (0.89) — HSL is a change-rate substrate with no absolute-magnitude channel, so absolute-size tasks aren’t its strength. That points at a concrete v-next encoder card: add one coarse absolute channel without breaking losslessness. All of this is single-seed/toy/unconverged, i.e. exactly the multi-seed + scale you asked for — still owed.

3. Schedule boundary — partial, two runs. I ran the zero door through full from-scratch curricula (not the 3k probe). Per-byte model: text bpb 1.674 → 1.531 → 1.578 across stages (the last rise is the chat/knowledge mix pulling on the text budget). A new 61M run with an I/O-symmetric output door (below): 2.25 → 1.965 → 1.999. Either way the zero door holds up over a long schedule on its own. The head-to-head you actually asked for — does a plain learned embedding catch up given the same long budget — I still haven’t run; that’s the clean next experiment and I’ll report the curve, not a point.

4. Grounding — the break, the fix, and the recovery. In my last reply I flagged that the full run broke disk-grounding: the gap (extra bits/byte when retrieved facts are swapped for wrong ones) collapsed from 1.835 on a learned-door model to ~0.007 under the zero door. Diagnosis: the zero door also zero-pads the retrieved memory features, so they dilute against the slot positional encoding and the model stops reading their content. Fix: keep the input door at 0 params, but give the memory path (retrieved knowledge ≠ input bytes) its own small learned projection. The recovery result, on the 61M multimodal-first curriculum:

step	1	1k	2k	3k	4k	5k	6k	7k	8k
gap	0.000	0.001	0.016	0.025	0.052	0.067	0.087	0.115	0.131

Monotonic, no collapse — the memory-path projection does restore disk reading. Three honest caveats: (1) 0.131 ≪ the old learned-door 1.835 — it reads the disk again, but modestly. (2) In an isolated knowledge-only probe the same fix overshot to ~1.7 and then overfit — it memorized the store and text bpb invaded; in the mixed run (knowledge = 20% of the batch) it stays at a stable 0.131, so the mix regularizes it and magnitude is regime-dependent. (3) clean isolated-vs-mixed and multi-seed comparisons are still pending. Net: the break is fixed in direction; the ceiling under a zero input door is an open number.

5. A note on the output head (not a scale claim). The model in the original post was 25M with a per-byte output. The 61M run above also packs the output the same way as the input — K-byte within-slot autoregressive, so input and output are both K-packed (I/O symmetric). It costs bpb (the per-byte numbers were lower), so “bigger” and “bundled output” are confounded here — I’m not reading 61M as a scale result. I flag it only because you separated input-door from output-head: this is a first look at giving the output the same zero-shaped K-packing, and the honest read is that the symmetry is a structural choice that costs bits rather than a win.

Still blank, in your order: schedule head-to-head (does a plain learned embedding catch up given the same long budget — the clean one), then hard-negative binding, then a real multi-seed scale sweep. Thanks again — the map keeps paying off, including where it says “this part doesn’t work yet.”

[John6666]

Oh. Looks like there’s a useful next framing here:

---

Short version

This update makes the result more interesting, not less, because it shows real boundaries.

My updated read would be:

zero HSL door looks promising for primary byte streams; fixed channel addresses make it inspectable; K is a compression/alignment knob; retrieved memory probably needs a separate learned interface; output packing is a separate question; and absolute sensor tasks may need an explicit magnitude channel.

So the next design rule may not be:

use zero door everywhere.

It may be:

use zero HSL for primary observations, add learned interfaces where the path is semantically different, and add explicit channels only when a measured boundary shows they are needed.

That feels like a much more useful research direction.

This is now a path-specific door question

This update is useful because the map is no longer just:

does the zero door work?

It is becoming:

which path can use a zero door, which path needs a learned interface, and which task needs extra absolute information?

That distinction feels important.

A useful reference point here is the earlier embedding-free byte line, especially Shaham & Levy, Neural Machine Translation without Embeddings, which showed that byte models can avoid learned embeddings by using fixed byte representations. But HoLo_ZeRo is now moving into a more specific question: not merely “can the input embedding be removed?”, but “which interfaces can remain fixed, and which interfaces need learned adapters?”

That is a sharper question.

The channel-knockout matrix is probably the most important new result

The channel-knockout table seems especially valuable because it makes the fixed-address design pay off.

The zero door is not only saving parameters. It is making the first interface inspectable.

Observation	My read
text/caption lean heavily on Fourier	local byte prediction may rely more on bit-pattern structure/distribution than on raw numeric locality
image leans more on Δ + phase	decoded image-like bytes seem to use numeric-local / phase-like structure
audio spreads over Δ + Fourier + phase	audio seems to use a mixed signal-like representation rather than one dominant channel family
boundary is near zero everywhere	useful negative control under fixed K-packing

The boundary-channel result is actually nice. If every channel looked important, the knockout would be harder to trust. The fact that boundary stays idle under fixed patching makes sense: with fixed K, the model does not need to ask “where does the unit end?”

That gives a clean future prediction:

boundary should matter more under adaptive or content-determined slotting.

So the channel matrix is not only diagnostic; it creates a falsifiable next experiment.

This also connects nicely to the broader byte-modeling literature. CANINE, Charformer, MEGABYTE, and Byte Latent Transformer all deal, in different ways, with the fact that byte/character streams are long and need some kind of downsampling, grouping, or patching. What is different here is that the fixed HSL channels let you inspect which parts of the byte-signal representation are load-bearing after training.

That inspectability may be one of the strongest reasons to keep the zero-door setup around, even apart from parameter count.

The grounding failure is the most informative break

The disk-grounding result looks like the most important failure case so far.

My updated read is:

Path	Current read
primary byte input	zero HSL door looks plausible
decoded sensor-like input	zero HSL door can be competitive with a learned door, at least in these small probes
retrieved memory / disk facts	zero-padding alone is not enough; this path likely needs its own learned interface
output head	K-packed symmetry is a separate structural choice and currently seems to cost bits
absolute-distance tasks	change-rate geometry may need an explicit absolute-magnitude channel

That makes me think the next map should be a path-specific door map, not only a failure-boundary map.

Something like:

Path	Candidate door	Why
primary decoded byte stream	zero HSL door	fixed geometry seems enough for ordinary observation bytes
retrieved memory / disk facts	small learned projection	retrieved facts are not just another byte stream; the model must be forced to read their content
output head	separate design, not automatically symmetric	input K-packing and output K-packing do not have to share the same answer
absolute sensor tasks	HSL + coarse absolute channel	change-rate features may miss magnitude-critical information
adaptive slotting	HSL + active boundary channel	boundary may wake up only when slot boundaries become content-dependent

The grounding break is useful because it suggests that “zero learned door” may be path-specific, not universal.

That is a good result, not a bad one. It narrows the design rule.

Memory path may be a different object than input bytes

The retrieved-memory failure suggests a conceptual distinction:

primary input bytes are observations; retrieved memory is evidence.

Those may need different interfaces.

For primary input, zero HSL features can work because the model is learning from local byte structure.
For retrieved facts, the model has to treat the memory content as something to consult. If the memory path is too thin or too diluted by positional structure, the model can learn to ignore it.

So I would phrase the result this way:

zero door is promising for primary observation streams, but retrieved memory probably needs an explicit learned reading interface.

That is a more specific and more useful claim than either:

• “zero door works everywhere,” or
• “zero door fails grounding.”

The monotonic recovery after adding a small learned memory projection is encouraging, even if the current gap is still far below the old learned-door number. It shows directionality: the model can be made to read the disk again when the memory path has its own interface.

The next useful split might be:

Condition	What it tells us
zero input door + zero memory door	whether the pure zero-door version can read retrieved content
zero input door + learned memory projection	whether a separate memory interface restores grounding
learned input door + zero memory door	whether the failure is specifically memory-path dilution
learned input door + learned memory projection	upper/reference condition
zero input door + low-rank memory adapter	whether a very small adapter is enough
zero input door + per-channel scale only	whether memory needs full projection or just rescaling
isolated knowledge-only vs mixed training	whether the memory path overfits when the batch is too knowledge-heavy

The isolated-vs-mixed distinction seems important. If the isolated knowledge probe overshoots and overfits, but the mixed run stays modest and stable, then the memory path may need curriculum/mixing control as much as architecture control.

This is close in spirit to the broader lesson from retrieval-augmented modeling: retrieval content is not just “more input.” It is a different information path. Work like RAG makes that separation explicit by giving retrieved evidence its own retrieval/conditioning mechanism. I do not mean HoLo_ZeRo should copy that architecture, only that the failure mode here fits a known pattern: retrieved evidence often needs a distinct interface.

The absolute-distance failure is also a useful encoder hint

The sensor probe is interesting because it gives a more concrete version of the modality-boundary story.

The result is not simply:

HSL works on sensor bytes.

It is more like:

HSL is competitive on several signal-like byte streams, but absolute-magnitude tasks may expose a missing channel.

That is a much more useful finding.

If HSL is mainly change-rate / relational geometry, then losing to a raw range profile on an absolute-distance task makes sense. The next encoder hypothesis becomes very concrete:

add a small coarse absolute-magnitude channel and test whether it improves absolute tasks without damaging change/binding tasks.

I would probably test it as an explicit card:

Encoder variant	Test
current HSL	baseline
HSL + coarse absolute bucket	does absolute-distance improve?
HSL + normalized absolute channel	does magnitude help without scale instability?
HSL + per-modality absolute channel	does this help only sensors, or also image/audio?
HSL + absolute channel ablated at eval	does the model actually use it?

The important part is to avoid making the encoder more complex without a direct failure case. Here there is a direct failure case, so the extra channel has a clear reason to exist.

This also fits the broader pattern from byte/signal models: the usefulness of a byte representation depends on whether byte-value geometry corresponds to something meaningful in the underlying data. Decoded grayscale, μ-law audio, lidar/radar ranges, UTF-8 text, and compressed bytes should not necessarily behave the same way.

Output head should stay separate from input door

The I/O-symmetric run is useful precisely because it did not simply validate symmetry.

My read would be:

input-side zero K-packing and output-side K-packing are related ideas, but not the same claim.

Input packing gives the model a compressed observation interface.
Output packing changes the prediction factorization.

Those can have very different costs.

Component	Question
zero input door	can fixed geometry replace a learned input embedding/projection?
K-packed input	can several bytes share one attention slot efficiently?
K-packed output	can several output bytes be bundled without paying too much bpb?
I/O symmetry	is structural neatness worth the output prediction cost?

So I would keep the output-head result as its own branch:

structurally interesting, but currently not a win.

That is still useful. It prevents the input-door result from being overextended.

This distinction is also useful relative to prior embedding-free work. Shaham & Levy discuss replacing learned embeddings in byte models, including the first/last layer framing. But HoLo_ZeRo’s strongest current evidence seems input-side. Output-side bundling should probably remain a separate experiment rather than being assumed from the input-side result.

Updated map

My updated map would be:

Area	Current state	Next useful test
primary input door	promising	long-schedule learned-embedding head-to-head
channel geometry	now inspectable	repeat knockout across seeds/checkpoints
modality boundary	starting to appear	decoded signal-like vs UTF-8 vs compressed/encrypted controls
sensor boundary	HSL ≈ learned in small probes, but absolute distance exposes weakness	HSL + coarse absolute channel
K boundary	K=16 preserves some bpb but softens binding	K sweep with binding/hard negatives
grounding	zero memory path breaks; learned memory projection recovers directionally	path-specific memory-door ablation
output head	I/O symmetry costs bits	keep separate from input-door claim
boundary channel	idle under fixed K	adaptive/content-determined slotting

That is a much sharper story than a single “zero door wins/loses” table.

What I would prioritize next

Given this update, I would prioritize three things.

1. Long-schedule learned-embedding head-to-head

This is still the cleanest missing test.

It separates:

Possibility	Meaning
learned embedding catches up	zero door is mostly a cold-start / sample-efficiency advantage
learned embedding does not catch up	fixed HSL geometry remains useful even after enough schedule
learned embedding catches up on bpb but not binding	HSL geometry matters more for association than local prediction
learned embedding wins text but loses sensors	modality boundary becomes the main story

This curve matters more than one endpoint.

2. Path-specific door ablation

The grounding failure makes this almost as important as the schedule curve.

I would test:

Input door	Memory door	Expected use
zero	zero	pure zero-door stress test
zero	learned projection	current recovery direction
learned	zero	tests whether memory failure is independent of input door
learned	learned projection	reference
zero	low-rank / gated memory adapter	minimal learned memory interface
zero	per-channel scale only	tests whether memory needs full projection or just rescaling

This would answer:

how much learned interface is needed for retrieved memory?

3. Hard-negative binding

Because the channel/geometry story now looks different for text bpb and binding, binding deserves its own stronger test.

Useful negatives:

Negative	Why
same-class wrong instance	removes simple class shortcut
same-length caption mismatch	removes length cue
entropy-matched caption mismatch	removes byte-distribution cue
shifted audio/video window	tests temporal leakage
image-only / audio-only ablation	shows modality contribution
top-k retrieval	easier to interpret than bpb alone

If HSL geometry still helps under hard negatives, that becomes a much stronger result than “zero door gives better text bpb.”

Links / orientation points

I would keep these as orientation points, not as direct baselines:

Link	Why it is useful here
Shaham & Levy — Neural Machine Translation without Embeddings	closest reference for embedding-free byte input
CANINE	tokenization-free character encoder with downsampling
Charformer	byte/character sequence shortening via learned GBST
MEGABYTE	byte patching and long byte-sequence modeling
Byte Latent Transformer	dynamic byte patches and FLOP-controlled byte scaling
RAG	useful reminder that retrieved evidence is often a separate interface, not just more input

Bottom line

This update makes the result more interesting because it shows real boundaries.

My updated summary would be:

zero HSL door looks promising for primary byte streams; fixed channel addresses make it inspectable; K is a compression/alignment knob; retrieved memory probably needs a separate learned interface; output packing is a separate question; and absolute sensor tasks may need an explicit magnitude channel.

So the next design rule may not be:

use zero door everywhere.

It may be:

use zero HSL for primary observations, add learned interfaces where the path is semantically different, and add explicit channels only when a measured boundary shows they are needed.

That is a more useful research direction than a universal zero-door claim.

[ggunio]

Update: I built two speech models on top of this 0-param substrate — tokenizer-free

STT + TTS (HoLo-ToLk). Short version: substrate + a spectral lens beats a mel baseline

on STT (CER 0.194 vs 0.213, multi-seed); TTS is a tokenizer-free single-speaker

feasibility demo. Full writeup + combined demo here: [New Post]