HoLo-ToLk: tokenizer-free speech (STT + TTS) on the 0-parameter HSL byte substrate

Models
1

Follow-up to my earlier post on the 0-parameter input layer.

I took the HSL byte substrate (no tokenizer, no learned input embedding) and built
two small speech models on top, to see whether “bytes as signal” carries through to
audio. I’m calling the line HoLo-ToLk.

STT (speech → text) — the result I’m most confident about.
Feeding the raw HSL substrate to a char-CTC baseline is weak on its own (CER ~0.67).
Adding a small model-side spectral lens (log-mel + a learnable gated fusion over the
frozen substrate) flips it: CER 0.194, beating a mel-spectrogram baseline (0.213) in
the same setup, confirmed across 4 seeds. So the honest takeaway is a controlled
comparison — substrate + lens > mel, same setup — not a SOTA number (8 kHz, char-CTC,
no LM; readable but rough).

TTS (text → speech) — here the byte substrate is even more natural: UTF-8 text bytes
go straight in as HSL features, no tokenizer/vocab. A small AR transformer + guided
attention + HiFi-GAN gives a single-speaker voice. Held-out teacher-forced mel-L1 is
0.296 (multi-seed) and some samples sound genuinely natural — but free-run synthesis
on arbitrary sentences is still rough/unstable. So I’m framing TTS as a feasibility
demo, not a usable TTS.

Both are research/devlog results, not production or SOTA. The two models are separate
today; the goal is to unify them into one over time.

Try it (combined demo, both tabs):

Substrate: pip install hsl-embedding-zero

Happy to answer questions on the lens design or the byte→signal encoding — and very
open to critique, especially on the TTS free-run instability.

2

Hmm… I looked into it a bit on Colab:

Short read

My read is that the STT side is already a fairly clean path-specific lens result, while the TTS side should probably be treated as a free-run acoustic-path debugging problem, not first as an HSL text-input failure.

For STT, the important result is not “HSL alone solves speech.” It is more specific:

fixed HSL substrate + a speech-specific spectral lens + gated fusion can beat the mel baseline in the same small setup.

That is useful because the weak hsl / stronger hslspec pattern says where the missing structure is: speech recognition needs time-frequency structure, not only raw byte/signal locality. Your own framing above is careful here: 8 kHz, char-CTC, no LM, not SOTA, controlled comparison, with CER 0.194 for substrate+lens vs 0.213 for mel in that setup. The STT repo is the better external anchor for that part.

For TTS, I would separate three questions:

  1. Can the HSL text front end condition the decoder at all?
  2. Can the autoregressive mel decoder cover the text and stop at the right time?
  3. After it covers/stops, is the generated mel/audio phonetically clear enough?

Those are different failure surfaces. A rough waveform by itself does not identify which one failed.

So the next clean artifact I would want is not another single teacher-forced mel-L1 number. It is a small free-run instability map: coverage, stop/duration, acoustic clarity, and vocoder/mel controls.

1. The useful framing: fixed substrate + path-specific lenses

The line seems to be moving from:

“Can the learned input door be removed?”

to something more practical:

“Which paths can use a fixed HSL substrate directly, and which paths need a small lens/interface for missing structure?”

A compact map might be:

Path What HSL gives Missing structure / risk Minimal lens or interface
Primary decoded byte stream fixed byte/signal geometry maybe none at the door zero HSL door
STT audio byte/signal substrate time-frequency structure spectral lens + gated fusion
TTS text input tokenizer-free UTF-8 byte conditioning probably not the first suspect HSL text-byte front end
TTS acoustic output generated mel/audio path duration, stop, alignment, phonetic clarity AR mel decoder + diagnostics
Retrieved memory / facts bytes, but semantically different path evidence-reading interface learned projection / adapter
Output head prediction factorization not symmetric with input separate output design

That makes the STT result cleaner, not weaker. It says the fixed substrate is useful, but the path still needs the right measurement lens. Speech recognition needs a spectral lens. TTS text input may not.

More detail on the STT/TTS split

The STT result is useful because it does not only report a score. It gives a boundary:

Front end How I would read it
hsl raw/fixed HSL substrate is not enough for ASR
hslwav adding more time-domain signal does not by itself solve the missing structure
mel conventional time-frequency baseline
hslspec + gate fixed substrate plus spectral lens gives the best result in this setup

That avoids overclaiming “HSL replaces speech features,” and instead says:

HSL can be a common substrate, but ASR still needs a speech-shaped lens.

The gate is also a useful diagnostic. If the gate stays near zero, the model is ignoring the HSL side. If it moves upward consistently, then the HSL path is contributing something beyond the mel-like path. I would keep that gate curve/table in future reports.

The TTS side is different. The TTS repo describes the path roughly as:

UTF-8 text bytes → frozen HSL features → Transformer text encoder → AR mel decoder + guided attention → postnet → HiFi-GAN

That pipeline has several places where free-run synthesis can fail even if the text front end is reasonable:

Component Possible failure
Text/HSL input punctuation, numbers, rare bytes, normalization-sensitive prompts
Encoder-decoder attention skipped words, repeated words, partial coverage
Stop head early stop, late stop, frame-cap hits
AR mel feedback drift, loop, acoustic blur after several seconds
Postnet / mel target over-smoothing, weak consonants, unstable high-frequency structure
Vocoder waveform artifacts or mel/vocoder mismatch

That is why I would not jump from “free-run is rough” to “the HSL text input failed.” A stable teacher-forced mel-L1 does not prove free-run stability, but it does suggest the text conditioning path is at least worth separating from the acoustic generation path.

Autoregressive TTS has known failure surfaces around attention, duration, and free-run mismatch. For example, FastSpeech explicitly targets skipped/repeated words from autoregressive alignment failures, and One TTS Alignment To Rule Them All discusses alignment robustness as a central TTS problem. That does not mean HoLo-ToLk has the same root cause. It only means this is a useful diagnostic vocabulary.

2. TTS: I would make free-run failures observable first

The next useful TTS artifact might be very small and very diagnostic.

Not:

“Here is one more aggregate mel-L1.”

Instead:

“Here is where free-run breaks: coverage, stop, duration, phonetic clarity, or vocoder/mel path.”

A minimal table could be:

Bucket N Why it matters
short plain sentence 5 sanity check
medium LJSpeech-like sentence 5 normal operating region
long sentence 5 duration / coverage / cap stress
punctuation-heavy 5 pauses, punctuation, byte patterns
numbers / abbreviations 5 text normalization and OOD phonetic stress
technical terms 5 phonetic clarity under unusual words

For each output, I would log:

Field Why
input chars / bytes input length
generated frames duration behavior
hit max_frames stop failure or cap too low
max stop probability whether stop head ever becomes confident
stop frame early/late termination
final attended text position coverage
attention coverage ratio skipped/unfinished text
rough manual label or ASR proxy content preservation
mel image path acoustic inspection
wav path listening / external checks

Even 25–30 prompts can tell a lot. It is not a benchmark. It is a failure map.

How I would interpret that free-run map

A useful decision table might be:

Observation First suspect Next check
Short prompts fail immediately model/checkpoint/basic acoustic path inspect mel image + simple sanity prompts
Short prompts align and stop input path is not globally broken move to longer/stress prompts
Long prompts hit frame cap before covering text max_frames / duration / long-form coverage rerun with larger cap
Attention reaches the end but generation still hits cap stop-token calibration stop-threshold sweep
Attention jumps backward repetition / AR loop attention trace + repeated phrase prompts
Attention skips forward coverage / deletion attention trace + ASR round-trip
Coverage is good but ASR gives near-word substitutions acoustic/phonetic clarity mel/vocoder/teacher-forced controls
Waveform sounds bad but mel looks plausible vocoder or mel scaling oracle-vocoder check
Numbers/abbreviations fail but plain text works normalization / data sparsity / OOD text compare “8 kHz” vs “eight kilohertz” vs “eight thousand hertz”

This is why I would log coverage and stop before changing the text front end. Otherwise several different failures get collapsed into “TTS is rough.”

One easy trap is to treat a bad free-run sample as a single failure. For this kind of AR mel decoder, I would first separate:

  1. Did it cover the text?
  2. Did it stop?
  3. Did it generate enough frames?
  4. Did the audio preserve the words?

Those can disagree.

For example, if a long prompt only reaches half the text before max_frames, the text front end may be fine; the cap may simply be too low or duration too long. If extending max_frames lets attention continue toward the end, that is more a duration/cap/termination issue than an encoding issue.

If attention coverage is already high but the audio is still hard to transcribe, then the next boundary is different: phonetic clarity, mel quality, postnet behavior, vocoder compatibility, or free-run acoustic drift.

This is also why stop_thresh is not just a UI knob. It is a diagnostic knob. The TTS README already hints at this by distinguishing teacher-forced evaluation from true free-run synthesis and suggesting a lower stop threshold when synthesis hits the frame cap.

3. ASR round-trip is useful, but only as a limited proxy

Since this project already has an STT half, a round-trip check is tempting:

text → TTS → audio → ASR → CER/WER

I think that is useful, but only with a narrow interpretation.

It can help detect:

ASR round-trip signal Possible meaning
many deletions coverage failure / skipped text
repeated words repetition loop
near-word substitutions weak phonetic clarity
numbers/technical terms fail OOD text or normalization stress
all ASR models disagree wildly ASR proxy may be unreliable
stronger ASR still makes similar substitutions generated audio may be acoustically ambiguous

But I would not present ASR CER/WER as TTS quality. It is not MOS. It is not naturalness. It mixes TTS errors with ASR errors.

A safer wording is:

ASR round-trip is a content-preservation proxy, mainly useful for deletion/repetition/near-word substitution diagnostics.

For background, ESPnet-TTS uses ASR-based objective evaluation in an ASR/TTS framework, and tools like JiWER make CER/WER easy to compute. If using Whisper, I would also keep the Whisper docs in mind: it is an ASR model, not a TTS evaluator.

4. I would add one vocoder/mel control before blaming the front end

A rough waveform does not locate the failing component. In a Tacotron-like cascade, the badness can come from the acoustic model or the vocoder path.

The classic split is:

text → mel → waveform

Tacotron 2 is the standard reference point for this decomposition: text is mapped to mel spectrograms, then a vocoder produces waveform audio.

For this project, since the TTS side uses a HiFi-GAN vocoder, I would add a small oracle-vocoder check:

Input to vocoder What it tells you
ground-truth/reference mel → vocoder whether the vocoder/config can reconstruct clean speech
teacher-forced predicted mel → vocoder whether the acoustic model works when given the correct history
free-run generated mel → vocoder whether self-fed inference drifts

The SpeechBrain HiFi-GAN LJSpeech model card is relevant here because it describes a vocoder that takes a spectrogram and produces waveform audio. Its notes also make the same practical point: vocoder use depends on compatible spectrogram settings, such as hop length and mel layout. So I would avoid saying “the vocoder is bad” or “the HSL front end is bad” until this control is done.

A compact diagnostic card

One possible compact report:

Prompt bucket N cap hit % coverage mean stop fired % ASR CER proxy Main failure
short 5
medium 5
long 5
punctuation 5
numbers / abbreviations 5
technical terms 5

Then a second table for interpretation:

Dominant failure Design implication
cap hit before coverage increase/learn duration budget, inspect stop head
coverage good but no stop stop calibration / termination loss
coverage bad with jumps attention / monotonicity / alignment
coverage good but ASR substitutions high acoustic clarity / mel target / vocoder path
only numbers/technical terms fail normalization / text distribution stress
repeated phrase loops AR feedback / exposure bias

This would make the TTS caveat more actionable than “free-run is rough.”

5. Delay the big redesign until the failure map says which branch is needed

There are several possible next architectures, but I would not jump to them before the small map above.

If the failure map says alignment/duration is dominant, then duration or monotonic-alignment ideas become natural. FastSpeech is the obvious reference for a duration/length-regulator branch. Glow-TTS is another useful reference for monotonic alignment / non-AR TTS.

If the map says teacher-forced is fine but free-run drifts, then the issue looks more like exposure bias / self-fed acoustic drift. A relevant reference point is Teacher-Student Training for Robust Tacotron-based TTS, which discusses training/inference mismatch in Tacotron-style systems.

If the map says coverage and stop are fine, but words are phonetically unclear, then I would inspect mel targets, postnet behavior, vocoder compatibility, and maybe whether mel regression is too blurry for the small model.

So the branch order I would use is:

  1. Make free-run observable.
  2. Split coverage vs stop vs acoustic clarity.
  3. Only then choose duration/monotonic/vocoder/acoustic-model changes.
Reference anchors

Project links:

TTS / alignment / duration references:

Evaluation / vocoder references:

Bottom line

The next clean framing might be:

STT: fixed substrate + spectral lens is the useful result.
TTS: do not collapse free-run roughness into “HSL input failed.” First split coverage, stop/duration, acoustic clarity, and vocoder/mel path.

If those diagnostics show that text coverage is easy but phonetic clarity remains poor, then the next problem is probably not the zero/HSL text door. It is the acoustic generation path.

That would actually make the story stronger: the project would no longer be trying to say “zero learned interface everywhere.” It would be showing where a fixed substrate is enough, where a path-specific lens is needed, and where generation-side interfaces still need work.

3

Thanks again — this is genuinely one of the most useful replies I’ve had. You actually
went through the repos and the implementation, gave me a clean decomposition (text
conditioning vs decoder coverage/stopping vs acoustic clarity) plus a concrete protocol,
and the reference list is exactly the right reading. I really appreciate the time.

On the free-run point: you’re right that “free-run is rough” shouldn’t stand as monolithic
evidence about the front-end. The eval already separates held-out teacher-forced mel-L1
from free-run stop-firing + length-sanity, but I’ve been reading them too coarsely. I’m
adding the two things you suggested — the oracle-vocoder check (HiFi-GAN on ground-truth
mels, to split vocoder from mel-generation) and a fixed free-run instability map over
~25–30 prompts — and I’ll defer any redesign until those isolate the failure.

I also want to be honest about where my thinking has moved more broadly, because it lines
up with your critique:

  • I no longer insist on — or claim — “zero learned parameters” as a hard constraint. A
    string of failures taught me that losslessness doesn’t unlock everything; it’s a useful
    substrate, not a silver bullet.
  • So I’ve paused the unified model and gone per-modality first. STT/TTS was the first pair,
    and that’s exactly where I learned that modality-specific lenses are necessary — the same
    lesson your “don’t over-attribute to the front-end” framing points at.
  • For text generation, the byte-level nature of HSL runs into word salad without a
    large-scale dataset to tame it, and I don’t have that scale.
  • To sidestep the scale requirement, I’ve walked back my original no-vocab goal and brought
    in BPE-style segmentation — but deliberately not as learned embeddings. BPE cuts the input
    into meaning-unit chunks, and each chunk is then streamed through the same frozen HSL 27-D
    encoding. So, ironically, the attempt to escape tokenizers/vocab circled back toward vocab —
    but since there’s still no embedding table, the parameter count doesn’t grow, while the
    practical benefits of a vocab (semantic chunking, far less word-salad) carry over.

The honest caveat is compute and budget — I can’t show these as fast as I’d like. But the
direction is settled: keep the parameter-free input property, drop the dogma, and let each
modality have the lens (or segmentation) it actually needs.

Thanks again for pushing on this — it’s the right kind of pressure.

4

I think that’s probably the safer direction. If I were organizing that route, I’d frame it like this:

Short read

This route becomes easier to evaluate if each branch gets its own narrow label.

For the speech branch:

tokenizer-free speech on the fixed HSL substrate

For the BPE text-generation branch:

embedding-table-free text generation over BPE-segmented HSL streams

That keeps the useful constraint visible without forcing every branch into the same label.

The pattern is becoming:

fixed HSL substrate where possible, explicit path-specific lenses where the path needs extra structure.

For STT, the lens is spectral.
For TTS, the next lens is diagnostic: coverage / stop / acoustic clarity / vocoder controls.
For text generation, BPE-style segmentation becomes a segmentation lens.

1. The route becomes clearer if BPE is treated as a segmentation lens

The route becomes easier to evaluate if the global constraint is split into path-level claims.

In that map, BPE is not just an implementation detail. It is the text-generation lens.

Path What changed Clean framing
HoLo_ZeRo input remove learned byte embedding door zero-param observation door
STT raw HSL alone is weak, spectral lens helps speech needs a time-frequency lens
TTS text input can use HSL, free-run still rough acoustic generation is a separate path
text generation raw byte HSL gives word salad at small scale text generation needs segmentation structure
memory / grounding zero memory path was weak retrieved facts need a learned/explicit interface

That gives a consistent story:

fixed substrate where possible, explicit lens where necessary.

2. The BPE branch needs its own label

Once BPE is in the loop, the branch has segmentation and a vocabulary. So I would label the branch around the remaining constraint:

no learned embedding table for the segmented units; the chunks are still encoded through the fixed HSL substrate.

A useful naming table:

Possible phrase Why it works / risk
segmentation lens fits the path-specific lens story
embedding-table-free BPE-HSL precise and compact
BPE-segmented HSL stream descriptive and low-risk
tokenizer-free with BPE mixes two different claims
semantic chunks useful intuition, but too strong if literal

A clean version might be:

BPE-HSL: embedding-table-free generation over HSL-coded subword chunks

That separates three things:

Layer Question BPE-HSL answer
Segmentation / vocabulary Are there explicit chunks? yes, BPE-style chunks
Embedding table Does each chunk get a learned vector? no, if each chunk is streamed through HSL
Model body Does the Transformer learn mixing/generation? yes

So the interesting text-generation question becomes:

how much segmentation structure is needed before the fixed HSL substrate becomes useful for generation?

Background note on BPE wording

BPE chunks are often useful subword units, and they can line up with meaningful pieces of words. But mechanically, BPE is based on frequent merges, not semantic parsing.

The Hugging Face tokenizer summary describes BPE as a subword tokenization algorithm that iteratively merges frequent adjacent pairs. The HF BPE course page gives the same merge-based framing. Sennrich et al. used subword units for rare/open-vocabulary handling in neural machine translation.

So the wording I would use is:

BPE gives useful subword segmentation and shorter/higher-level text units, but not a guaranteed semantic parser.

If the tokenizer training part needs to be more explicit and reproducible, SentencePiece is a useful reference because it provides unsupervised subword tokenization and supports BPE/unigram-style workflows.

A nearby comparison point is byte-level subwords: keep byte-level robustness while using subword structure to reduce sequence burden. That is not the same as HSL, but it is a useful reference point for the raw-bytes vs byte/subword hybrid design space.

3. The key control is segmentation vs HSL geometry

BPE may help for two separate reasons:

  1. It reduces sequence burden and gives the model larger text units.
  2. The HSL stream over those chunks may still provide useful fixed geometry.

Those are different claims.

A result like:

BPE-HSL improves word salad

would be useful, but it would not yet say whether HSL geometry is the important part. It might simply mean the model needed segmentation.

So I would make the BPE-HSL branch easy to interpret with a small control table.

Condition What it tests
raw byte HSL pure byte-substrate baseline
raw byte HSL + longer schedule whether the issue is only training budget
char-level / byte-level HSL with same model size whether shorter chunks are needed
BPE segmentation + HSL stream segmentation lens effect
BPE tokens + learned embedding baseline what is lost by avoiding embedding tables
BPE segmentation + random/fixed vectors whether segmentation alone explains the gain
BPE segmentation + permuted HSL mapping whether HSL geometry matters after segmentation
vocab-size sweep chunk granularity boundary
rare words / numbers / punctuation bucket whether BPE helps the actual failure cases

The key comparison is not only:

BPE-HSL vs raw byte HSL

but also:

BPE-HSL vs BPE learned embeddings

because that is the cleanest way to test the “embedding-table-free” part.

Possible interpretation table
Observed result Possible reading
BPE-HSL >> raw byte HSL segmentation helps
BPE learned embedding >> BPE-HSL learned token vectors still matter a lot
BPE-HSL close to learned embedding embedding-table-free path is surprisingly competitive
BPE-HSL >> random BPE vectors HSL geometry may be contributing
BPE-HSL ≈ random BPE vectors segmentation may be doing most of the work
smaller vocab worse, medium vocab best, huge vocab worse chunk granularity boundary

That keeps the result from becoming a single winner table. It stays a failure-boundary map.

4. HoLo-ToLk and BPE-HSL can stay related but distinct

The branches can share the same substrate story without sharing the same public label.

Branch Claim
HoLo-ToLk STT HSL substrate needs spectral lens for speech recognition
HoLo-ToLk TTS HSL text front end is feasible, but free-run acoustic generation needs a failure map
BPE-HSL text generation text generation may need a segmentation lens, while still avoiding learned embedding tables

That keeps the speech claim and the text-generation claim from interfering with each other.

The phrase “tokenizer-free speech” can still describe the HoLo-ToLk STT/TTS demo if the text path is UTF-8 bytes and the STT path outputs chars/bytes. Once BPE is added for text generation, I would give that branch its own label.

Something like:

HoLo-BPE-HSL: embedding-table-free text generation over HSL-coded subword chunks

A bit long, but very clear.

5. The updated wording can stay narrow and testable

A scoped version of the claims might be:

Claim Safer status
fixed HSL substrate yes
no learned embedding table plausible branch claim
tokenizer-free only for branches that actually avoid segmentation/vocab
no learned interface anywhere probably too broad
path-specific lens map increasingly useful
BPE for text generation segmentation-lens branch

That makes the route more testable.

Instead of one global claim, each branch gets a measurable boundary:

Branch Main question
STT Does spectral lens + HSL beat mel in the same controlled setup?
TTS Does free-run cover, stop, and preserve phonetic content?
BPE-HSL text Does segmentation help without needing a learned embedding table?
Memory / grounding Does a small learned interface recover evidence use?

Bottom line

I would frame the new route like this:

BPE-HSL is the segmentation-lens branch.
It has a vocabulary/segmentation step, but it can still test the more interesting constraint: no learned embedding table for the vocabulary items.

That fits the larger pattern:

fixed substrate where possible, explicit lens where necessary.

If BPE-HSL reduces word salad while staying reasonably close to a learned-embedding BPE baseline, that would be a strong result. If it only matches random fixed vectors, then segmentation is doing most of the work. Either way, the experiment becomes interpretable.