A model in production costs, fails and retries

In a notebook, a model call costs nothing, takes as long as it takes and always succeeds, because you rerun it until it works before showing anyone the result. Three false assumptions, and a great many architectures rest on them.

In production, a model is an external dependency, billed per call, with a latency that moves around and a failure rate that is not zero. Exactly like a payment gateway, except that a payment gateway returns the same answer twice for the same question twice.

Cost per call decides the architecture

This is the line that surprises teams looking at their first invoice. A pipeline that chains three model calls to produce one result costs three times, on every attempt, failed attempts included. If the user can retry, it costs as many times as they retry.

The question to ask before writing the first line is not "which model is best", it is "what does one delivered result cost, counting the failures". The answer determines whether you need caching, whether you need to split the work, and whether you need to refuse certain requests outright.

Latency is a product constraint, not a metric

On KLAAS the constraint is written down: a 1280x720 thumbnail, with a consistent face, in under sixty seconds. That is not a performance target, it is the definition of the product. Past that, the user goes back to their editing tool and the product has no reason to exist.

A time constraint stated that way decides the architecture before the choice of model. It says what you can afford to do in sequence, what has to run in parallel, and what has to be prepared in advance because it will never fit inside the window.

Failure handling is not optional

A call that fails needs a written next step: retry, retry differently, or hand back cleanly. All three are product decisions, not implementation details.

Retrying identically on a quota error achieves nothing and costs twice. Retrying with a fallback model changes the result, and somebody has to decide whether that different result is acceptable or whether it is better to say no. Handing back cleanly assumes there is a message that means something to a person who has not read the documentation.

Reproducibility is what sells

The hard part of KLAAS is not generating an image. It is that the creator’s face is the same from one render to the next. A spectacular result one time in three is not a product, it is a slot machine.

What is true of an image is true of everything else. A model that answers well most of the time, inside a system where nobody can say when it answers badly, is not industrialised. It is permanently on demo.

What it changes in practice

Treat the model as an ordinary external dependency. Measure what it costs, measure what it fails at, decide what happens when it fails, and check that the same input yields a usable result. None of this is specific to AI. It is what we have always done with a third party service, and it is precisely what gets forgotten when the novelty takes up all the room.