The Compression Trick That Could Change Everything About Running AI
Google's TurboQuant Cuts LLM Memory by 6x — With Zero Accuracy Loss
There's a moment in every technology arc where the expensive thing gets cheap and not because the underlying capability changed, but because someone figured out a smarter way to package it. We're at one of those moments in AI inference, and the paper responsible is called TurboQuant.
Google Research published TurboQuant in late March 2026, and it formally lands at ICLR 2026 in Rio de Janeiro on April 25th. The core idea is deceptively elegant: compress the KV cache in large language models down to just 3 bits per value, cutting memory usage by 6x — without any retraining, without any calibration data, and without losing accuracy.
A small primer, because this stuff is genuinely interesting: when you send a message to a large language model, the model generates what's called a Key-Value cache as it processes context. Think of the KV cache as the model's working memory, it holds information about everything in the conversation so far, letting the model be coherent across long exchanges. As context windows have gotten longer (GPT-5.4 now supports one million tokens), these caches have ballooned into one of the biggest bottlenecks in AI infrastructure. Data centers run out of GPU memory not because the models themselves are too big, but because the caches are enormous.
TurboQuant attacks this problem with a two-step approach. First, it applies something called PolarQuant, a random rotation of the data vectors that simplifies their geometric structure, making them much easier to compress efficiently. Then it runs standard quantization on the rotated data. The result: 3.5-bit compression that matches full-precision performance on every standard benchmark, and a 4-bit variant that's actually 8x faster than 32-bit keys on H100 hardware for the attention computation step.
What this means in practice is remarkable. On the same physical hardware, TurboQuant enables models to handle 4 to 8 times longer context windows or, alternatively, to run far larger batch sizes, meaning more simultaneous users at lower cost. And because it requires no retraining and works on any transformer architecture, it can simply be dropped into existing deployments.
The downstream effects here are worth thinking about carefully. If the memory bottleneck in LLM inference shrinks by 6x, the economics of running these models shift significantly. Cloud providers can serve more users per GPU. Self-hosted deployments become more viable on consumer hardware. Startups that couldn't afford the infrastructure to run a large model with a million-token context window suddenly can. Some analysts at TrendForce have noted that TurboQuant could even dampen demand growth for high-bandwidth memory chips, which has already rattled a few corners of the semiconductor market.
This is the kind of research breakthrough that doesn't make front-page news but quietly makes everything downstream cheaper and more accessible. The model capabilities get the headlines. The efficiency gains are what let people actually use them.
Sources:
