Compression and prediction are two views of the same operation. A model that predicts the next symbol well can be used to encode data compactly, and a good compressor implies a good predictive model. This equivalence underpins much of how large language models are understood.
Key takeaways
- Lossless compression and probabilistic prediction are mathematically interchangeable: a model that assigns probabilities to data can be turned into a code, and a code implies a model.
- The bridge between them is the idea that the ideal number of bits used to encode an event is determined by how improbable the model considered that event.
- Large language models are trained to predict the next token in a sequence, which means the training objective is formally the same as minimising the cost of compressing the training data.
- The equivalence is a statement about the objective, not a claim that a language model is a practical file compressor, since running one is far more expensive than running conventional compression tools.
- Whether “compression equals understanding” follows from the mathematics is a matter of interpretation, and informed people disagree about how much the equivalence explains.
What is actually being claimed
The claim is narrow and technical. Suppose a system reads a stream of data one symbol at a time and, before each symbol, outputs a probability distribution over what the next symbol might be. An encoding scheme can use that distribution to write out the actual symbol using a number of bits that depends on the probability assigned to it: high probability, few bits; low probability, many bits. Symbols the model expected cost almost nothing to record. Symbols it did not expect cost a great deal.
Run this over an entire file and the total length of the output is a direct measure of how well the model predicted the file. A better predictor produces a shorter file. The reverse direction also holds: any scheme that reliably compresses data must, implicitly, be assigning shorter codes to some patterns than others, and that assignment is a probability model whether or not its designers described it that way.
So “compression is prediction” is not a metaphor or an analogy. It is a statement that two apparently different engineering problems — making files smaller and guessing what comes next — are the same problem written in different notation.
Why the idea is circulating now
The equivalence is old and well established in information theory. What has changed is the context in which people encounter it. Systems trained purely to predict the next token in text now produce output that reads as fluent, and in some settings as competent. That has prompted a search for a compact explanation of what such a system is doing, and the compression framing offers one: the model is not memorising text so much as finding regularities that let it describe text more briefly than the text itself.
The framing has also gained traction because it reframes a debate. If prediction and compression are the same thing, then arguments about whether “mere prediction” could ever amount to anything substantive have to contend with the fact that compressing a body of data well requires capturing the structure that generated it. Whether that observation settles anything is exactly where the disagreement lies.
There is no single event driving the discussion. It surfaces periodically in technical forums as an explanatory device, and its current visibility reflects interest in how language models work rather than any new result.
The background a newcomer needs
Three pieces of background make the rest legible.
The first is that information can be measured. The amount of information in a message is not about its meaning but about how surprising it is given what the receiver already expects. A message that could have been predicted in advance carries little information; one that could not carries a lot. This is why a long file of repeated characters compresses to almost nothing while a file of random bytes does not compress at all — randomness is precisely the absence of predictable structure.
The second is that compression algorithms all work by exploiting expectations, even the ones that never mention probability. A scheme that replaces repeated strings with references to earlier occurrences is betting that repetition is common. A scheme that assigns short codes to frequent characters is betting on a particular frequency distribution. Each is a model of what the data is likely to look like, expressed as an algorithm rather than as a set of probabilities.
The third is that language model training works by scoring predictions. During training, the model is shown a sequence, asked what comes next, and adjusted according to how much probability it assigned to the symbol that actually followed. Summed over a whole corpus, that penalty is the same quantity as the length of the compressed file the model would produce. Minimising one minimises the other.
Who this matters to and how
For people building models, the equivalence is a practical instrument rather than a philosophical one. It provides a unit — bits per symbol — that makes different systems comparable on the same data, and it connects training loss to a quantity with a concrete meaning rather than an arbitrary scale.
For people trying to understand what these systems do, the framing offers a way of thinking that avoids two common errors. It resists the idea that a model is a lookup table of memorised text, since memorisation is a poor compression strategy for anything larger than the model. It also resists the idea that prediction is a shallow operation, since predicting well over varied material requires representing whatever regularities the material has.
For those working on compression itself, the relationship is not news, and the practical picture is mostly one of costs. Neural models can, in principle, be used as the probability source for a compressor, and the resulting compression ratios can be strong. But the computational expense is enormous compared with standard tools, and both the compressor and the decompressor need the identical model, which must either be shipped alongside the data or already be held by both parties.
Where informed people disagree
The mathematics is not disputed. What is disputed is what it licenses one to conclude.
One position holds that compressing data well requires building an internal account of the process that produced it — that to predict text about physical events, a model must represent something about how physical events unfold. On this reading, the equivalence is evidence that a predictive model must acquire structure that resembles understanding.
The opposing position holds that this conflates the objective with the mechanism. Achieving low bits per symbol says a system has captured statistical regularities; it does not by itself specify what kind of regularities, or whether they generalise beyond the distribution the model was trained on. A system could compress a corpus well by exploiting surface patterns that fail wherever the pattern breaks.
A third line of argument questions whether the framing adds anything. Every supervised learning method has some objective that can be given an information-theoretic reading, so calling the objective “compression” may relabel it rather than explain it.
These are not empirical disagreements that a single experiment resolves. They concern how much interpretive weight a formal equivalence can carry.
What follows in practice
Several concrete things follow from taking the equivalence seriously.
Evaluation gains a common currency. Measuring how many bits a model needs per unit of data gives a comparison that does not depend on a particular benchmark’s design, though it says nothing about whether a model is useful, safe or accurate on any specific task.
Data quality matters in a specific way. If the objective is to describe data as briefly as possible, then data with genuine structure rewards a model that finds it, whereas noise offers nothing to find and simply consumes capacity.
Model size relates to what can be captured. A model has a finite number of parameters, which bounds how much regularity it can encode. This is one framing of why capacity and data volume are discussed together, though the relationship in practice is more complicated than the framing alone suggests.
What to watch next
Three developments are worth following, none of them predictable in advance.
Whether bits-per-symbol style measurements become a standard part of how models are reported, alongside task benchmarks, or whether they remain mainly an internal engineering metric.
Whether neural compression finds settings where its cost is justified — specialised domains, or cases where the model is already present on both ends of a channel for other reasons.
Whether the interpretive debate produces sharper claims. The useful version of the argument would specify what kinds of structure a given level of compression performance does and does not imply, rather than treating the equivalence as settling the question in either direction.
Frequently asked questions
Does compression is prediction mean language models are compression tools?
Not in a practical sense. The equivalence says the training objective of a next-token predictor is formally the same as minimising compressed length. Using such a model to actually compress files is possible but computationally expensive compared with conventional compressors, and it requires both sender and receiver to hold an identical copy of the model. The claim concerns mathematical structure, not a recommendation about which tool to use.
Why does a better predictor produce a smaller file?
Because encoding schemes can allocate bits according to expectation. When the model considers a symbol likely, that symbol can be recorded using very few bits; when it considers a symbol unlikely, recording it costs many bits. Over a whole file, the total is smallest when the model’s expectations most often match what actually appears. Prediction quality and encoded length are therefore two readings of the same measurement.
Is this a new discovery?
No. The relationship between probability models and code lengths is long-established in information theory and is standard material in the field. What is comparatively recent is its use as an explanatory frame for large language models, whose training objective happens to be exactly the quantity the theory describes. The renewed attention reflects interest in explaining these systems, not a new theoretical result.
Does compressing data well prove a model understands it?
That is the disputed part. Compressing data well demonstrates that a system has captured regularities in it. Whether those regularities constitute understanding, or generalise beyond the training distribution, is not determined by the compression figure itself. Some argue that strong compression requires modelling the underlying process; others hold that it only shows statistical structure was found, without specifying what kind.
Why can random data not be compressed?
Compression works by exploiting predictability. If every symbol in a file is equally likely regardless of what came before, there is no expectation to exploit, and no scheme can assign shorter codes to some sequences without assigning longer codes to others. Truly random data therefore has no shorter description than itself. This is why incompressibility is sometimes used as a working definition of randomness.
How does this relate to lossy compression like images and audio?
The equivalence described here concerns lossless compression, where the original data can be reconstructed exactly. Lossy methods deliberately discard information judged unimportant to human perception, which introduces a separate question of what to keep. Prediction still plays a role in lossy schemes, but the clean correspondence between probability and code length applies to the lossless case.
Sources and further reading
- Standard textbooks on information theory, which set out the relationship between probability distributions and optimal code lengths.
- Academic literature on data compression, covering both classical dictionary and entropy-based methods and neural approaches.
- Technical documentation for widely used compression formats, which describes the modelling assumptions each format makes about its input.
- Discussion threads on technical forums, where the framing is debated and where the disagreements about its interpretation are visible.
Surfaced from the hackernews signal “information theory discussion”. AI-assisted draft, editorially reviewed.
