Production monitoring for an LLM application is more than checking whether an API returned a 200 response. This checklist shows what to log, measure, test, and review so your team can detect rising costs, slow responses, retrieval failures, unsafe outputs, and broken tool calls before they become recurring user problems.
Overview
An AI application has several failure points that a conventional web-service dashboard may not reveal. A request can succeed technically while the model answers the wrong question, cites irrelevant retrieved content, calls a tool with invalid arguments, or produces an output that requires human correction. Effective LLM application monitoring therefore combines standard API observability with evaluation of model behavior.
Organize monitoring into five layers:
- Request and response health: availability, status codes, timeouts, retries, latency, and request volume.
- Usage and cost: input tokens, output tokens, cached tokens where applicable, model routing, and estimated spend.
- Application behavior: structured-output validity, tool-call success, retrieval quality, task completion, and fallback frequency.
- Safety and privacy: sensitive-data handling, policy or guardrail events, prompt-injection signals, and escalation outcomes.
- User and operational feedback: ratings, corrections, abandonment, support tickets, incidents, and human review results.
Track these dimensions together. A latency increase may result from a larger prompt, a retrieval service problem, provider congestion, or repeated tool retries. A rise in user complaints may reflect a model update, a changed system prompt, incomplete context, or a defect in the application layer. Correlated telemetry makes those possibilities easier to separate.
Before deployment, define an owner for each metric, a reasonable baseline, and an action for meaningful deviation. For background on pre-production testing, see How to Evaluate an LLM Before Production.
How to estimate
Monitoring has its own operational cost, so estimate the required volume before choosing retention, sampling, and evaluation settings. Start with simple, repeatable inputs rather than pretending that a single dashboard score represents quality.
Estimate event volume
Use this basic calculation:
Monthly traces = daily requests × operating days × average spans per request
A span might represent the initial model call, a retrieval operation, a tool call, a retry, or a post-processing step. If one request normally creates several spans, storing only top-level request counts will hide where time and failures occur.
Estimate token usage and model cost
For each model or provider, record:
- Average input tokens per request
- Average output tokens per request
- Requests per day or month
- Retry rate
- Traffic share by model
- Any provider-specific cached-input or batch-processing categories
A planning formula is:
Estimated input tokens = requests × average input tokens
Estimated output tokens = requests × average output tokens
Apply the provider's current pricing separately to each token category when making a budget estimate. Do not hard-code prices into application logic or a permanent article; provider pricing, model names, billing units, and included features can change. A cost dashboard should show both raw usage and the pricing version used for the estimate. For related optimization ideas, see How to Reduce LLM Costs: Caching, Routing, and Prompt Design Strategies.
Estimate review and evaluation effort
If a team reviews a sample of production traces, estimate the workload with:
Review hours = traces sampled × minutes per review ÷ 60
Use separate samples for ordinary traffic, failed requests, low-rated responses, safety events, and new model or prompt versions. A small targeted sample can be more useful than a large undifferentiated archive. For automated checks, track false positives and false negatives rather than treating every flagged trace as a confirmed failure.
Inputs and assumptions
Define a minimum event schema before instrumenting every component. The exact field names may vary by framework or provider, but the concepts should remain stable enough for comparison across releases.
Request and response fields
- Timestamp, environment, application version, prompt or workflow version, and model identifier
- Request ID and parent trace ID for connecting model, retrieval, and tool events
- Provider status, error category, retry count, timeout status, and cancellation reason
- Time to first token, total response time, queue time where available, and service time
- Input, output, and cached token counts when supplied by the API
- Finish reason, response format, and whether a fallback model or rule was used
Do not assume that storing complete prompts and responses is always appropriate. Establish a data classification policy, redact sensitive values before logging, restrict access, set retention periods, and record when content has been sampled or masked. Hashing or truncation can help with correlation, but neither automatically makes sensitive data safe.
Retrieval and tool-call fields
For retrieval-augmented generation, capture the knowledge-base version, query representation or safe query identifier, number of retrieved items, filters, similarity or ranking signals when available, and whether the final answer used retrieved context. A retrieval event succeeding technically does not prove that the selected documents were relevant.
For tools, log the tool name, schema version, validation result, execution time, authorization decision, sanitized arguments, result status, and whether the model tried again. Keep secrets, access tokens, and unnecessary personal data out of tool traces.
Quality and feedback fields
Useful signals include explicit ratings, edited or regenerated answers, abandonment, escalation, citation clicks, task completion, human labels, and structured quality checks. Treat these as indicators rather than perfect truth. Users may rate an answer for tone rather than accuracy, and a completed workflow may still contain a subtle factual error.
| Symptom | Likely causes | First corrective action |
|---|---|---|
| Responses become slower | Larger prompts, retrieval latency, provider delay, or tool retries | Break latency into queue, retrieval, model, and tool spans; compare with prompt and traffic changes. |
| Token use rises without more traffic | Prompt growth, duplicated context, longer history, or expanded tool results | Compare token counts by workflow version and inspect the largest prompt components. |
| Valid JSON falls | Prompt or model change, incomplete schema handling, or truncation | Log validation errors and finish reasons; add bounded retries or a deterministic repair path. |
| Answers cite irrelevant material | Weak retrieval query, stale index, poor filters, or context overload | Review retrieved items separately from generation quality and test a fixed evaluation set. |
| Tool failures increase | Schema drift, permission changes, malformed arguments, or upstream outage | Separate validation, authorization, execution, and downstream errors. |
| Users report confident wrong answers | Missing context, unsupported inference, model change, or weak grounding instructions | Sample affected traces, compare against expected answers, and add refusal or escalation criteria. |
Worked examples
Consider a hypothetical support assistant handling 2,000 requests on each operating day. If each request produces one model span, one retrieval span, and an average of 0.3 tool spans, the planning volume is:
2,000 × 30 × 2.3 = 138,000 spans per month.
This is a sizing estimate, not a prediction. Actual volume changes with retries, multi-turn conversations, failed workflows, and background jobs. The team could retain detailed traces for a limited period, keep longer-term aggregates, and sample routine successful requests while retaining all errors and safety events. The retention policy should be reviewed with the application's privacy and security requirements.
For a cost review, suppose the same hypothetical application averages 1,400 input tokens and 350 output tokens per request. Monthly usage would be:
Input: 2,000 × 30 × 1,400 = 84,000,000 tokens
Output: 2,000 × 30 × 350 = 21,000,000 tokens
To estimate spend, apply the current provider rates for the selected model to those two totals, then add expected retries, embeddings or retrieval infrastructure, tool execution, monitoring storage, and any secondary model calls. Recalculate separately for each routed model. If a new prompt version raises the average input from 1,400 to 1,900 tokens, the change is visible even before the invoice arrives.
For an incident review, use a consistent template:
- Incident: what users experienced and when it began
- Scope: affected workflow, model, version, region, and percentage of requests
- Detection: metric, alert, user report, or human review that identified it
- Trace evidence: representative request IDs, latency, token counts, retrieval results, tool events, and provider errors
- Root cause: confirmed cause, with unresolved alternatives listed separately
- Mitigation: rollback, routing change, prompt adjustment, feature disablement, or human escalation
- Follow-up: evaluation cases, alert changes, tests, documentation, and owner
When to recalculate
Revisit your monitoring assumptions whenever a model, provider API, prompt, retrieval index, tool schema, traffic pattern, or pricing input changes. A release can alter tokenization, context behavior, output length, latency, tool-call conventions, or error categories even when the application code appears unchanged. Review provider release notes and deprecation notices before migration; the AI model release notes checklist can help structure that review.
At minimum, run a formal observability review twice a year and during each quarterly reliability or cost review. Recalculate after a major incident, a sustained change in user feedback, a new safety requirement, or a material shift in traffic. Recheck dashboards after changing retention or sampling so that a lower event count is not mistaken for improved reliability.
Use this practical maintenance sequence:
- Export the last review's assumptions, schema, thresholds, and pricing inputs.
- Compare current traffic, latency, token usage, failures, quality labels, and escalation rates with the baseline.
- Inspect a sample of successful, failed, low-rated, and safety-related traces.
- Update model identifiers, provider error mappings, privacy controls, evaluation sets, and alert owners.
- Run a small regression test before deploying changes, then monitor closely through the first release window.
- Record what changed and schedule the next review.
Monitoring is useful when it leads to a decision: investigate, route differently, revise a prompt, refresh retrieval data, limit a tool, escalate to a person, or accept a measured tradeoff. Keep the checklist tied to those decisions, and your AI observability practice will remain useful as the application and its underlying models evolve.
