HoLo/HSL: a 100M change-rate-based multimodal toy model on a single RTX 4070

Models
1

Hi everyone,

I’d like to share a small personal research POC I built: HoLo/HSL.

HoLo is a ~100M parameter toy-scale multimodal model trained on a single consumer GPU (RTX 4070).
It is not meant to compete with existing models, and the generation quality is still very rough. At this stage, the goal is simply to test whether the core architecture can work.

The main idea is to explore whether change-rate, rather than tokens, can serve as a shared signal substrate across modalities such as text, image, audio, and video.

What currently works:

  • byte-native input/output pipeline
  • text, image-frame, audio, and early video generation plumbing
  • closure signal for distinguishing end-of-content vs end-of-window
  • multimodal signal-space visualization
  • phase/affect channel experiments
  • external memory/offload toy experiments
  • synthetic sensor-fusion toy experiments
  • a small public demo showing the generated signal trajectory

Important caveats:

  • This is a feasibility study, not a benchmark claim.
  • The model is very small.
  • Generation quality is currently poor.
  • Some experiments are toy-scale.
  • I am not claiming superiority over existing systems.

Still, I wanted to share it because the full pipeline now runs end-to-end, and I think the change-rate / signal-substrate direction may be interesting for multimodal research.

SPACE : ggunio/holo-demo-space

Paper:

[Zenodo link](https://doi.org/10.5281/zenodo.20581805)

Demo: https://holo-demo-p5txmh4dda-as.a.run.app

GitHub:

[GitHub link](https://github.com/Woojiggun/holo-hsl)

I’d be happy to hear feedback, especially on the architecture, evaluation design, and possible next experiments.

2

Hmm, I’m not deeply familiar with this area myself, but after digging around a bit, I think the picture is roughly this:

I think the most useful way to sharpen this is to separate a few claims that can otherwise get blended together.

The part that seems most distinctive to me is not simply “byte-level modeling.” Byte-level / tokenizer-free modeling already has a real research context — for example ByT5, MEGABYTE, and Byte Latent Transformer / BLT.

What seems more specific to HoLo/HSL is the attempt to use a byte-to-signal substrate — change-rate / spectral / phase-like features — as a shared low-level representation across modalities.

So the question I would try to make maximally clear is:

under controlled conditions, what part of the result comes from raw bytes, what part comes from dense packing / effective context, and what part comes from the HSL substrate itself?

That framing might make the project easier for outside readers to discuss, because it separates several different ideas:

Layer Question
Byte-native modeling Can bytes work as the common input/output unit?
Sequence-length handling Does dense packing / larger effective context help?
HSL substrate Does the change-rate / spectral / phase feature map give a useful inductive bias?
Architecture Does the dense encoder + byte-autoregressive decoder help beyond simpler byte baselines?
Multimodal binding Is the model learning cross-modal association, or mostly exploiting simple shortcuts in the toy setup?
Reproducibility Which parts can outside readers reproduce directly, and which require the private encoder/API?

That is not a criticism. I think it is a useful way to make the interesting part stand out.

What I would compare it to conceptually

I would not treat these as direct competitors or required baselines. I would treat them as orientation points: examples of adjacent projects that help clarify what kind of claim is being made.

Thread Useful examples What they help clarify
Byte-native / tokenizer-free modeling ByT5, MEGABYTE, BLT Bytes remove tokenizer assumptions, but sequence length and compute allocation become central.
Modality-agnostic perception Perceiver, data2vec How much modality-specific machinery can be removed before learning becomes unstable or inefficient?
Cross-modal binding CLIP, ImageBind Matched/mismatched and retrieval-style probes can make binding evidence easier to interpret.
Unified multimodal systems Chameleon, Unified-IO 2, AnyGPT, 4M, Show-o Many unified systems still rely on tokenized or discrete modality representations; HoLo/HSL seems to be asking a lower-level substrate question.
Reproducible reporting Model Cards, HF model cards, Datasheets for Datasets Useful patterns for making “what is claimed / what is not claimed / what is reproducible” easy to read.

The main distinction I would emphasize is:

HoLo/HSL is probably not best framed as a small Chameleon or a small Unified-IO. Those systems mostly unify modalities after tokenization or discrete representation design. HoLo/HSL seems to be asking whether a byte-to-signal substrate can sit below modality-specific tokenization.

That makes the idea easier to place.

What I would try to isolate next

If the next goal is to make the mechanism easier for others to evaluate, I think small controlled experiments would be more useful than a larger demo.

The most useful next table might be something like this:

Experiment What it isolates
Same-parameter raw-byte baseline Whether the gain is coming from model size.
Same-context raw-byte baseline Whether the gain is coming from larger effective context.
Same-FLOP baseline Whether the method is actually more compute-efficient.
Same architecture + learned byte embedding Whether the handcrafted substrate helps beyond a learned byte representation.
Same architecture + simple public byte features Whether the open architecture works without the private substrate.
Same architecture + random fixed invertible map Whether invertibility alone explains anything.
HSL feature-family ablations Which parts of the substrate matter: bit-level, first-order change, second-order change, spectral, phase-like, etc.
Multi-seed small runs Whether the reported deltas are stable at toy scale.

The key thing I would want to know is not just:

does HSL beat raw bytes?

but rather:

under matched parameter count, context length, FLOPs, data, and seed budget, which part of the system actually moves the metric?

That would make the central claim much easier to evaluate.

Substrate ablations that might be especially informative

Since the substrate is the unusual part, I think ablations around it would be very valuable.

For example:

Ablation / control What it tests
Bit-level only Are raw byte/bit features already enough?
Bit + first-order change Does change-rate help?
Remove second-order change Does acceleration-like structure matter?
Remove spectral channels Are frequency-domain features useful?
Remove phase-like channels Are phase-like features doing work?
Shuffle feature channels Does feature geometry matter, or only feature capacity?
Random fixed invertible map Is “lossless/invertible” enough, or does this specific geometry matter?
Learned byte projection Can the model learn an equivalent representation by itself?
Public surrogate substrate Can outside readers reproduce the architecture-side result without the private encoder?

The “invertible” part is important, but I would not stop there. A random invertible map is also information-preserving. The more interesting question is whether this particular feature geometry gives the model a useful inductive bias.

Binding probes

The matched-vs-mismatched bpb results are interesting as binding probes. I would probably strengthen that direction with harder negatives and more intuitive retrieval-style metrics.

Possible additions:

Probe Why it helps
Same-class wrong-instance negatives Reduces the chance that the model only learned the class label.
Same-length caption negatives Reduces trivial caption-length effects.
Entropy-matched caption negatives Reduces byte-distribution shortcuts.
Image-only / audio-only / image+audio ablations Shows which modality is actually contributing.
Shifted video windows Tests whether nearby temporal context explains the gap.
Top-k retrieval Easier for readers to interpret than bpb alone.
Cross-dataset transfer Tests whether the binding survives beyond the original toy distribution.

For example, in a digit setup, a stronger negative is not just “wrong digit word.” It could be:

  • same digit class, wrong image instance;
  • same digit class, wrong speaker/audio instance;
  • same caption length;
  • similar byte entropy;
  • image-only vs audio-only vs image+audio.

That would help separate “the model found a shortcut” from “the model is actually binding the modalities.”

A small public reproducibility packet

Given that the original byte-to-signal encoder/codec is withheld, I think a small public reproducibility packet would help a lot.

I do not think the withheld substrate is a reason to dismiss the project. But it does change what outside readers can verify. A compact evaluation packet could make the open parts much easier to discuss.

Something like:

Public artifact Why it helps
Fixed tiny dataset split Everyone tests the same examples.
Exported feature tensors People can train/evaluate the open architecture without the private encoder.
Public surrogate feature extractor People can test whether the architecture-side idea survives without the original substrate.
Expected metric outputs Easy sanity check for reproduction.
Exact train/eval commands Lowers reproduction friction.
Configs for matched baselines Makes comparisons less ambiguous.
Seeds and logs Helps separate real deltas from toy-scale variance.
Small checkpoint Lets people verify the inference path.
One-page claim/evidence/caveat table Helps forum readers understand the project quickly.

That might be more useful at this stage than trying to scale the demo immediately.

A possible “claim table”

One thing that might help readers is a short table like this in the repo or paper:

Claim Current evidence Caveat Next test
Byte-native pipeline runs end-to-end Text/image/audio/video-like streams can pass through one pipeline Does not yet imply strong generation quality Fixed reproducible demo + small checkpoint
Dense input helps AsymHSL direction looks promising Params/context are confounded Same-param, same-context, same-FLOP baselines
HSL substrate is useful Feature-based runs are plausible Exact substrate is private; effect not fully isolated Public surrogate + ablations
Cross-modal binding appears Matched/mismatched bpb gaps Possible shortcuts in toy setup Hard negatives + retrieval
One architecture can process multiple modalities Same model family runs on several byte streams Decoded/normalized streams are not arbitrary raw media formats Clear preprocessing description

That kind of table would prevent readers from arguing about the wrong claim.

Minor documentation clarifications that might help

A few small clarifications could make the project easier to read from the outside:

Clarification Why it helps
What exactly counts as “video” in each experiment Raw MP4 bytes, decoded frames, frame/audio/caption windows, and generated raster frames are different things.
Which results require the private encoder Helps readers know what can be reproduced directly.
Which results can run with a substitute feature map Makes the open architecture easier to test.
Exact preprocessing for image/audio/video streams Reduces confusion around raw media vs decoded/normalized byte streams.
Which datasets are toy proxies vs real-world sources Helps calibrate the strength of the evidence.
Which comparisons are matched vs intentionally not matched Prevents over-reading of early numbers.
A short “what this is / what this is not” section Useful for readers who only skim the repo.

Again, I do not mean this as criticism. I think the unusual part of the project is interesting enough that making the boundaries extra explicit would help people give better feedback.

Bottom line

My bottom-line read is:

HoLo/HSL seems most interesting as a mechanism-level PoC around a lower-level multimodal representation idea.

The strongest next step, in my opinion, would be to make the core claim easier to isolate:

  1. separate byte-native modeling from the HSL-substrate claim;
  2. compare against matched raw-byte / learned-byte / public-feature baselines;
  3. ablate the substrate feature families;
  4. strengthen matched/mismatched binding tests with hard negatives and retrieval metrics;
  5. publish a tiny reproducibility packet for the open architecture path.

That would make it easier to discuss the idea on its own terms, without forcing it into either “new SOTA model” or “not useful yet.”

3

Thank you for this review — it is exactly the kind of feedback this project needs, and we are adopting your framing more or less wholesale. The layer separation you propose (byte-native / dense packing / HSL substrate / architecture / binding / reproducibility) is now literally our experiment plan. A few things can be answered today; the rest we are committing to with a concrete roadmap.

One factual correction first

“the original byte-to-signal encoder/codec is withheld”

This premise is no longer true (and mostly never was): the encoder is fully public — pip install hsl-embedding, MIT license, formulas + tests + codec included. The only private artifacts are the trained HoLo model weights, which are work-in-progress. So every encoder-side claim is directly reproducible from PyPI today; no surrogate substrate is needed. (Your “which results require the private encoder” table item resolves to: none of the substrate-side ones.)

Shipped today, in direct response to this review (v0.5.0)

The package now contains hsl_embedding/ablation.py so that anyone — including you — can run the controlled comparisons you asked for with a one-line swap:

  • ControlEmbedding(kind, seed) with four variants sharing the identical 27-D layout and the same 9 context dims: hsl (bit-identical to the real encoder, test-enforced) / learned (trainable byte projection, +4,608 params — your “learned byte projection” arm) / random (seeded random injective LUT, moment-matched per channel — your “random fixed invertible map” arm) / permuted (HSL’s own 256 LUT rows shuffled — per-channel distributions exactly identical, only the value-adjacency geometry is destroyed; we think this is a sharper control than channel shuffling for the capacity-vs-geometry question).
  • feature_groups() / select_channels() for the feature-family ablations (drop Δ², drop FFT, …), plus a value(18)/context(9) split.
  • value_lut() — the frozen 256×18 table exported as a tensor (your “exported feature tensors” item).

What we can already state (with honest caveats)

Structural facts (verifiable from the public package):

  • The per-byte Δ is exactly the binary-reflected Gray code v ^ (v >> 1) — adjacent byte values differ in exactly one Δ coordinate (raw bits: up to 8). This is the mathematical content behind the “change-rate” framing, and it connects the substrate to the minimal-change-encoding literature.
  • The 27-D base decomposes as a frozen 256×18 value LUT + 9 context dims (Δ², boundary). So the substrate question reduces cleanly to: does this particular frozen embedding geometry beat a learned/random/permuted one at matched everything? (Linear rank of the value dims is 17/18 — one dependency, dxor0 lies in the FFT span — so we will not claim “18 independent channels”.)
  • One channel-scale caveat we will control for: fft_re0 (the DC term) spans 0–8 while other channels are ±1–2, so input-normalization placement will be held identical across all ablation arms.
  • The FFT dims are the spectrum of the bit pattern of each byte, not a temporal spectrum of the waveform — your documentation-clarity point is correct and the docs now say so.

Preliminary measurements (small scale, single-seed, prior encoder revision — to be re-run multi-seed with the ablation kit before we treat them as findings):

  • Same decoder-only architecture, same data/steps: 27-D HSL input 2.058 bpb vs learned byte embedding 2.118 bpb on a byte-level LM task. This is the single number that most needs the multi-seed matched-baseline treatment you describe, and it is first in the queue.
  • Architecture axis at matched data/budget: decoder-only prefix-LM (~11M params) outperformed an encoder–decoder twice its size (2.227 vs 2.275 bpb) across all depths tested.
  • Mechanism (not quality) results for the disk-offload tier: with the answer present only in a disk-resident value, retrieval-ON reaches 1.000 task accuracy while ablated retrieval and no-memory controls sit at chance — the read mechanism is load-bearing, not decorative.

Claim table (current, your format)

Claim Evidence today Caveat Next test
Byte-native pipeline runs end-to-end text/chat/knowledge/video(539B windows) through one trainer works ≠ quality; generation demo pending trained ckpt fixed demo + small checkpoint
HSL substrate is useful beyond learned bytes 2.058 vs 2.118 bpb (matched arch/data) single seed, toy scale, prior encoder rev multi-seed ControlEmbedding A/B (hsl/learned/random/permuted)
Substrate geometry (not just invertibility) matters Gray-code structure; permuted control exists unmeasured the raw-bits(8) vs Δ(8) minimal pair — identical information/dims/scale, geometry only
Dense prefix + byte-AR decoder helps dec-only beat 2× enc-dec at matched budget params/context confounds partially addressed, not fully same-param/-context/-FLOP grid
Cross-modal binding matched/mismatched gaps in earlier prototypes shortcuts not excluded hard negatives (same-class wrong-instance, entropy-matched) + top-k retrieval
Knowledge lives on disk, not FFN ON 1.000 / ablated chance mechanism proof, synthetic facts knowledge-mode training on a real 73k-fact store (wired, training next)

What is running / next

A depth sweep on the final wired architecture is finishing now; the first full training run on the pinned public encoder follows, then: (1) the multi-seed substrate ablations above, (2) binding probes with your hard-negative list, (3) a small reproducibility packet — fixed tiny split, exact commands, seeds/logs, a small checkpoint, and the claim/evidence/caveat table maintained in the repo.

If you want to poke at the substrate before any of that lands: pip install hsl-embedding (>= 0.5.0), then from hsl_embedding.ablation import ControlEmbedding — the four variants are a one-line swap in any byte-LM training loop. A complete runnable comparison (examples/substrate_ablation.py) ships in the source distribution on PyPI. Thanks again — the review measurably improved the package within a day of being posted.