Per task, not per month
A monthly figure tells you nothing actionable. It moves with usage, so a rise could be growth or could be a regression, and nobody can tell which. Cost per completed task separates the two immediately: if the task count went up and the cost per task held, that is growth. If the cost per task moved, something changed in the system.
Getting there requires a decision at design time, which is why it belongs in the budget document rather than in a later optimisation pass. Every call needs to carry a stable task identifier through retrieval, generation and tool use, so the spend can be summed per task. Retrofitting that identifier across a system that did not plan for it is one of the more annoying pieces of remedial work there is.
Budget the steps, not the total
A single end to end latency number is not a budget, it is a symptom. The budget assigns time to each step, and the assignment is what makes a violation diagnosable.
- Input handling and validation
- Retrieval, including the filter and the vector search, reported separately from generation
- Any re-ranking or context assembly
- Model calls, counted individually, because two sequential calls is a design decision with a price
- Tool and external system calls, which are usually the least predictable line
- Output validation and grounding checks
A worked example, with placeholder numbers
The numbers below are invented for the purpose of showing the shape of the arithmetic. They are not measurements of anything, and the point is the method rather than the values.
Suppose the product requirement is that a support answer appears within four seconds at the ninety fifth percentile. Assign: input and validation two hundred milliseconds, retrieval six hundred, re-ranking three hundred, the model call two seconds, output validation two hundred. That totals three point three seconds and leaves seven hundred milliseconds of headroom for the network and the tail.
Now the design consequences fall out on their own. A second sequential model call does not fit, so if the task needs two, one of them has to be parallel or the requirement has to change. Retrieval has six hundred milliseconds, which decides whether a hybrid search with a re-rank is affordable. And because the budget is written down, the first time the ninety fifth percentile breaches four seconds the question is which line moved, not what happened.
The levers, in the order they usually pay
Caching first, where the workload repeats, which for most support and internal knowledge systems it heavily does. This constrains prompt structure, because a cache only helps if the stable part of the prompt comes first, and that is a decision that has to be made early.
Then routing by difficulty, so the expensive path is reserved for the tasks that need it and the routing decision itself is cheap. Then context discipline, because a retrieval step that returns twenty passages when six would do is paying for fourteen passages on every single call, forever. Then, and usually only then, the provider and model question, which is the one everybody starts with.
What to alert on
Alert on the derivative, not the level. Cost per task rising twenty percent week over week is a signal. Total spend crossing a round number is an accident of the calendar. The same is true of latency: a tail that has moved is worth waking someone for, and an absolute number that has been stable at the edge of the budget for a month is not.
And keep the history in your own warehouse. Provider dashboards are billing tools, they aggregate the way billing wants to aggregate, and they will not answer the question you actually have, which is which change did this.