Inference Is Where AI Actually Costs You

When people talk about AI, they almost always mean training — the enormous, expensive, GPU-melting process of building a model. It's the part with the dramatic numbers and the press releases. But if you actually run AI systems, you spend most of your life thinking about the other half: inference. And inference is where the interesting engineering, and most of the money, actually lives.

What inference actually is

Training is how a model learns. Inference is what happens every time you use it. You send an input, the trained model runs a forward pass, and out comes a prediction — a classification, a next token, a recommendation. That's it. Nothing is being learned during inference; the weights are frozen. The model is just applying what it already learned, once per request, forever.

The analogy I like: training is going to medical school. Inference is seeing a patient. One is a huge one-time investment. The other is the actual job, repeated a million times, and it's the part you have to make fast, cheap, and reliable.

Why inference is the part that keeps you up at night

Training happens once, or occasionally. Inference happens on every single request, and that changes everything about how you think about it.

Cost stops being capital and becomes a per-request tax. A model that's expensive to run doesn't cost you once — it costs you on every call, every user, every day, indefinitely. At scale, the inference bill dwarfs the training bill for most companies actually deploying models. The dramatic training number is a headline; inference is the standing order that never stops arriving.

Latency becomes a product feature. Nobody experiences your training run. Everybody experiences the two seconds they wait for a response. Inference latency is the thing users actually feel, and shaving it is a surprising amount of what applied ML work turns out to be.

And it has to keep working. A training run can fail and you retry it tomorrow. An inference endpoint is production — it gets paged, it gets load spikes, it needs the same boring reliability engineering as any other service people depend on.

The tricks are all about making it cheaper

Almost every optimisation you hear about — quantisation, distillation, batching, caching, smaller specialised models, running on the edge — is fundamentally about one thing: making inference cost less without making the output meaningfully worse. That's the whole game. Take a capable model and get the same answer for a tenth of the compute.

This is also why "just use the biggest model" is rarely the right production answer. The biggest model is the best at the task and the worst at the bill. A lot of real engineering is finding the smallest model that's good enough for the specific job, because you pay that difference on every request for as long as the product exists.

The mental model that helps

If you're building anything with AI, the useful shift is to stop thinking of the model as a thing you acquire and start thinking of it as a thing you run. Acquiring it — training it, or picking one — is a decision you make once. Running it is a cost, a latency, and a reliability commitment you make continuously.

Training is the impressive part. Inference is the part that decides whether your product is actually viable. The headline is the model. The business is the forward pass.