I built an experimental low bit LLM training method that matched FP32 perplexity at 53M parameters

Hey everyone!

I’ve been working on an experimental LLM training architecture called CECP (Co-Evolving Compositional Pre-Training), and I wanted to share some early results and get feedback from people working on quantization and model efficiency.

The idea came from a question I had a while ago:

What if we trained language models to adapt to extreme compression during training, instead of training them normally and quantizing them afterward?

That’s essentially what CECP explores.

The goal is to minimize the performance degradation associated with low-bit representations, potentially making larger models more accessible on consumer hardware.

The results so far

I’ve been testing CECP against a conventional FP32 baseline using TinyStories.

My latest experiment used:

  • Approximately 52.8M parameters

  • 2,500 optimizer steps

  • Three independent training seeds

  • An NVIDIA Tesla T4

  • Validation perplexity as the primary metric

Results (lower is better):

Model Mean Validation Perplexity
Dense FP32 18.146 ± 0.298
CECP-256 17.986 ± 0.082

CECP achieved essentially the same validation perplexity as the full-precision baseline in this experiment.

It uses an aggressively compressed representation for its projection weights, with a nominal 1-bit-per-weight index payload plus additional supporting data.

One interesting observation is that CECP initially learns more slowly than dense FP32, but the validation curves converge as training progresses.

The limitations

I want to be clear that these are still early results.

CECP currently requires more time to train than a conventional dense model, and these experiments are relatively small compared with modern LLM pretraining.

I also haven’t demonstrated that CECP outperforms established methods like BitNet, AQLM, or optimized low-bit quantization at larger scales.

The results suggest that the approach is worth investigating further, but they don’t establish general superiority.

What’s next?

I’m planning to test larger models, evaluate actual exported model sizes, and compare against stronger low-bit baselines.

I’m also interested in whether compression-aware training becomes more effective as model size increases.

For now, I’ve published a research overview and benchmark screenshots on Hugging Face:

The complete implementation is private while the architecture is still under development, but I’m sharing the preliminary results and general approach.

I’d genuinely appreciate feedback, particula

rly from anyone who’s worked with quantization-aware training, low-bit representations, or model-compression research.

Does this look like a promising direction, and what would you consider the most important next benchmark?

Promising — but before a larger model, settle what “1-bit” actually costs: export the weights and divide every byte needed to reconstruct them (index, codebook, scales, supporting tables) by the parameter count. If that effective bits-per-weight is near 1, the result is striking; if it is several bits, the FP32 comparison is really a multi-bit one. Pair that with the same three seeds run for both methods and report the per-seed differences, since the 0.16 perplexity gap sits inside the FP32 spread you report. Those two numbers are the next benchmark that matters: they say whether CECP beats a strong low-bit baseline at equal size, before scaling spends the budget finding out.

Thanks, this is really useful feedback.

You’re right that the nominal 1-bit index payload isn’t the same as effective bits per weight, and I don’t want to conflate the two.

The 53M model has approximately 49.55M CECP-compressed projection weights. At the current grouping configuration, the indices alone occupy approximately 5.91 MiB, but I still need to finalize the complete exported-storage measurement including codebooks, scales, and metadata.

I also have the per-seed results:

Seed Dense FP32 PPL CECP-256 PPL
0 17.982 17.894
1 17.966 18.010
2 18.490 18.053

I agree that the 0.16 mean difference isn’t enough to establish CECP as better than FP32. I’m treating this as near-parity in validation perplexity rather than a superiority claim.

My next step is to report exact effective BPW from the exported checkpoint and compare CECP with a stronger low-bit baseline at matched storage.

Appreciate the feedback. This is exactly the kind of scrutiny I was hoping to get from sharing the early results.