Observability, cost and drift

Quality, latency and cost per task on a dashboard your team owns, with an alert when any of the three moves.

An AI feature has a running cost that moves with usage, with the corpus, and with a provider decision you did not make. Teams find this out from a bill. The fix is unglamorous: trace every call, attribute cost to a task rather than to a month, budget latency by step, and watch the input distribution as carefully as the output quality. All of it lands on your monitoring stack, because a dashboard we own is a dashboard you lose.

What we actually do

End to end tracing per task with a stable identifier, so a support ticket resolves to an exact run
Latency budgeted per step, with retrieval, generation and tool time separated rather than reported as one number
Cost attributed per task, per feature and per customer where your model needs it, not per invoice line
Caching where the workload repeats, with hit rate reported, and prompt structure arranged so caching is possible at all
Routing by task difficulty, so the expensive path is used where it earns its place
Input drift monitoring on the distribution of what arrives, which usually moves before quality does
Output drift monitoring against the scheduled evaluation run, with alerts on movement rather than on absolute score
Alerts wired to the people who can act, with a runbook attached to each one

What we instrument

Measurements, not results. These are the readouts we put in place so that you end up with numbers about your own system. There are no values on this page because a value here would be somebody else’s.

Cost per completed task, tracked over time
Latency at the median and the tail, per step and end to end
Cache hit rate, and cost avoided by it
Token volume per task, split between input, retrieved context and output
Input distribution shift against the baseline captured at launch
Scheduled evaluation score, with the movement since the previous run

Where this stops working

Attributing cost per customer needs an identifier carried through every call, which is a design decision. Retrofitting it is expensive, so we raise it at the start
Drift detection tells you something changed, not what to do. It buys you the week you need to find out
Observability adds latency and storage cost of its own. We size the sampling deliberately rather than tracing everything and hoping

What we work with

Named as plain text. None of these is a partnership, an endorsement or a default: the right one is chosen per engagement, usually the one your team already runs.

OpenTelemetry, and your existing collector
Datadog, Grafana or whatever your team already reads
Tracing and evaluation tooling including LangSmith
Warehouse tables for cost and quality history, in your warehouse

engineering / the other five

None of these works alone.

C1Retrieval and contextGetting the right passages in front of the model. Most quality problems that look like reasoning problems are retrieval problems.C2Evaluation and quality gatesA labelled set, a threshold agreed in writing, and a gate in CI. Built before the feature, not after the first complaint.C3Agent orchestrationExplicit state, typed tools, bounded loops and a replayable trace. An agent is a distributed system, so we build it like one.C4Guardrails and escalationWhat the system may do alone, what it must hand to a person, and what the person receives when it does.C6Product and platform engineeringThe application around the model: interface, data model, auth, pipelines, infrastructure as code and a deployment your team can run.

Where does your system sit against this?

Send the architecture and the failure you are seeing. You get a written read from an engineer within a working day.

Start a build

What we actually do

End to end tracing per task with a stable identifier, so a support ticket resolves to an exact run

Latency budgeted per step, with retrieval, generation and tool time separated rather than reported as one number

Cost attributed per task, per feature and per customer where your model needs it, not per invoice line

Caching where the workload repeats, with hit rate reported, and prompt structure arranged so caching is possible at all

Routing by task difficulty, so the expensive path is used where it earns its place

Input drift monitoring on the distribution of what arrives, which usually moves before quality does

Output drift monitoring against the scheduled evaluation run, with alerts on movement rather than on absolute score

Alerts wired to the people who can act, with a runbook attached to each one

What we instrument

Cost per completed task, tracked over time

Latency at the median and the tail, per step and end to end

Cache hit rate, and cost avoided by it

Token volume per task, split between input, retrieved context and output

Input distribution shift against the baseline captured at launch

Scheduled evaluation score, with the movement since the previous run

Where this stops working

Attributing cost per customer needs an identifier carried through every call, which is a design decision. Retrofitting it is expensive, so we raise it at the start

Drift detection tells you something changed, not what to do. It buys you the week you need to find out

Observability adds latency and storage cost of its own. We size the sampling deliberately rather than tracing everything and hoping

What we work with

Named as plain text. None of these is a partnership, an endorsement or a default: the right one is chosen per engagement, usually the one your team already runs.

OpenTelemetry, and your existing collector

Datadog, Grafana or whatever your team already reads

Tracing and evaluation tooling including LangSmith

Warehouse tables for cost and quality history, in your warehouse