Breaking the 1.58-Bit Barrier: How BITCOS Squeezes More Out of Ternary LLMs

The ternary LLM story was supposed to end at 1.58 bits. That number - log₂3 ≈ 1.585 - is the information-theoretic cost of encoding one weight drawn from the three-symbol alphabet {-1, 0, +1}, and it has served as a ceiling ever since Microsoft Research's BitNet b1.58 paper made "1-bit LLMs" a mainstream conversation. A new preprint from Intel researchers Evangelos Georganas, Alexander Heinecke, and Pradeep Dubey argues the ceiling is false - and provides a storage format and kernel implementation to prove it.

The Problem With Five-Trit Packing

1 Ternary LLMs store every weight as one of three symbols {-1, 0, +1}, so their cost is conventionally pegged to log₂3 ≈ 1.585 bits per weight. 1 The prevailing deployment format packs five ternary weights into one byte - so-called five-trit packing - and because practical implementations use power-of-two group sizes, this rounds up to 1.625 bits per weight.

That 1.625-bit figure quietly embeds an assumption: that -1, 0, and +1 are equally probable. 1 The assumption treats the three symbols as equiprobable, which feels theoretically tidy but is empirically wrong.

Zero Weights Are Everywhere

1 Measuring the actual weight distributions across 29 real-world ternary LLM models, the authors find that zeros account for up to 51.5% of all weights. That skew is not a rounding artifact - it is a structural feature of how ternary models learn. Zeros naturally dominate because they implement a form of learned sparsity: weights that contribute little to a computation are driven to zero during ternary-aware training.

17 Out of the 29 models surveyed, 26 exhibit zero densities high enough that a distribution-adaptive layout would beat the five-trit baseline. The remaining three are close to the equiprobable case where packing efficiency is similar either way.

Introducing BITCOS

The authors' answer is BITCOS - a layout built on a simple observation: instead of encoding all three symbols uniformly, separate the question "is this weight non-zero?" from the question "what sign does it have?"

18 BITCOS is comprised of a dense presence bitmap plus a compacted sign vector, and costs 2 − z bits per weight element, where z is the measured zero density of that model. The formula is elegant: at 50% zeros the layout lands at 1.5 bits per weight; at the sparsest model in the study it reaches 1.485 bits per weight - comfortably below the 1.585-bit theoretical floor of equiprobable ternary encoding.

18 BITCOS stores weights more compactly than five-trit packing in 26 of the 29 tested models, and the layout is explicitly designed to be amenable to efficient unpacking on modern processors and GPUs. The authors provide optimized unpacking sequences targeting AVX-512, AVX2, and Intel Xe2 GPUs.

Measured Throughput Gains

Reduced storage is only useful if the hardware can actually process the new format efficiently - tighter packing can hurt throughput if unpacking becomes a bottleneck.

18 Measured against production state-of-the-art ternary matrix-vector multiplication kernels at the zero densities real-world models actually exhibit, the realized gain from BITCOS is up to 1.28×. 2 End-to-end in vLLM across seven ternary LLMs, decode throughput improves by 1.10 - 1.18× on one platform, 1.02 - 1.15× on a second, 1.09 - 1.22× on a third, and 1.02 - 1.27× on a fourth. The spread reflects differences in memory bandwidth, cache hierarchy, and the specific zero density of each model, but the direction is consistently positive.

Why the Conventional Framing Was Misleading

This paper is a useful corrective to how the ternary LLM literature frames compression limits. The 1.58-bit figure is a true information-theoretic bound - but only when all three symbols are equally likely. As soon as zeros dominate (which they do, structurally, in trained ternary models), Shannon's source-coding theorem says you can do better with an entropy-adaptive code. BITCOS is essentially Huffman coding applied to weight storage: more frequent symbols (zeros) get shorter codes, rare symbols (±1) tolerate slightly longer ones.

The five-trit packing format became a de facto standard not because it is optimal but because it is simple to implement with byte-aligned operations. BITCOS breaks that tradeoff by providing hardware-efficient unpacking paths - the AVX-512 sequences mean CPUs can decode the new format without scalar fallbacks.

Ecosystem Context

The timing matters. 2 The authors cite a 2025 BitNet b1.58 2B4T technical report and a 2026 Prism ML release of "Ternary Bonsai" models among the ecosystem they evaluated, with newer entries like DeepGrove's Maple (a 20B-A1B ternary reasoning model) also appearing in their benchmark table. Ternary models are no longer a research curiosity - they are being actively deployed for inference on AI PCs and edge hardware, where every fraction of a bit matters for fitting models into memory-constrained environments.

The paper also notes that future work will extend BITCOS and its unpacking sequences to additional CPU and GPU architectures beyond the Intel-focused targets in this release.

Why It Matters

The practical implications are straightforward: if you are serving any ternary LLM - BitNet b1.58, a Bonsai variant, or a post-training quantized model - switching to BITCOS-style storage is a free efficiency win with no accuracy trade-off. The layout is a pure representation change. The throughput numbers are real, measured in vLLM against production kernels, and the gains compound at the memory-bandwidth bottleneck that dominates ternary inference.

More broadly, the paper is a reminder that "information-theoretic limits" in ML deployment are often limits on the assumed distribution, not on reality. Measuring your actual weight distribution before committing to a storage format is, it turns out, worth doing.

The preprint is available at arxiv.org/abs/2609.16338.