---
title: "AI Observability: 7 Things Only The LLM Superstars Are Tracking"
description: "If you're monitoring these AI model metrics, your AI strategy is in better shape than most. Seven LLM observability metrics and why they matter."
canonicalUrl: "https://zuplo.com/blog/2026/10/13/ai-observability-llm-metrics"
pageType: "blog"
date: "2026-10-13"
authors: "billDoerrfeld"
tags: "AI Gateway, API Best Practices, ai-agents"
image: "https://zuplo.com/og?text=AI%20Observability%3A%207%20Things%20Only%20The%20LLM%20Superstars%20Are%20Tracking"
---
Most enterprises are adopting AI, using various agents and various large
language models (LLMs) under the hood. But, as many are seeing, AI models can
quickly rack up costs. They're also changing rapidly, making it challenging to
keep track of token costs, as well as model accuracy, reliability, and security
issues.
Observability is the natural response to surface AI gaps, yet it hasn't really
kept pace with this sudden shift to AI. For instance, a
[Cato Networks survey](https://www.catonetworks.com/news/shadow-ai-governance-lags-as-ai-adoption-soars/)
of 600 IT leaders found 61% had discovered unauthorized AI tools, yet only 26%
had solutions in place to monitor AI usage. Another
[Grafana Labs survey](https://grafana.com/observability-survey/2025/) found only
7% of organizations had LLM observability actively in production.
AI observability is in its early days, but this will likely change as
organizations seek to operationalize and optimize their use of AI. LLM-level
observability is essential to track token usage, enforce limits, aid debugging,
and create a feedback loop to
[improve agents](https://opentelemetry.io/blog/2025/ai-agent-observability/).
It's also necessary to inform model switching to match an LLM or agent to the
task at hand.
Below, we'll consider what AI observability should entail, looking at specific
metrics to track and why they're important from a business standpoint. By the
end, you'll have a framework to use to guide how you benchmark models in
production.
## 1. Latency and performance
_How are your requests to LLMs performing?_
In the context of AI models, performance can mean a lot of things, including how
quickly requests are processed. One specific way to
[measure latency](https://nordicapis.com/5-things-that-cause-high-latency-in-your-apis-and-how-to-fix-them/)
is by looking at time to first token (TTFT), a metric that tracks the time from
the initiated prompt to the first token returned.
Benchmarking from [TokenDyno](https://tokendyno.com/blog/llm-latency-benchmark/)
found that TTFT for the Llama 3.3 70B model ranged from a median of 0.71 to 1.22
seconds. TTFT is good to track, since APIs to popular LLMs can vary. They also
tend to have higher rates of incidents than other services, according to a
[2026 report from API Status Check](https://nordicapis.com/api-reliability-report-2026-uptime-patterns-across-215-services/).
Tracking latency is important for stability, as it can pinpoint network
interference, degraded service, and other performance problems. Knowing latency
can also be used to improve the overall business and application performance, as
it can help ensure providers maintain SLAs, and even inform which LLMs to use
for the most highly-performant, quickest results.
## 2. Accuracy
_How accurate are the LLM's responses?_
Although hallucination rates have generally improved, LLMs can still contain
bias and make inaccurate assumptions that produce
[intent errors](https://www.tricentis.com/blog/intent-drift-ai-code-fix-regression-blind-spots)
in agentic situations. This is especially true if prompts are vague or models
lack sufficient training or context.
Many industry benchmarks evaluate frontier models across coding, reasoning,
factual accuracy, hallucination resistance, and more. For instance,
[Artificial Analysis' Intelligence Index](https://artificialanalysis.ai/articles/claude-fable-5-1)
combines nine evaluation frameworks to compare model capabilities. As of
September 2026, the Index ranks Claude Fable and Opus models high for accuracy,
while DeepSeek V4 has a particularly high hallucination rate.
Public evaluations are helpful guidance, but enterprises also need to prove LLMs
work correctly against their own institutional data. Ensuring accuracy is a
challenging discipline that requires custom evaluations, such as conformance
checks and validation against known truths. This might also involve testing
whether AI initiates the intended subtasks or
[agent-to-agent communication](/learning-center/agent-to-agent-a2a-protocol-guide).
Accuracy can make or break the usefulness of an LLM's response. Tracking it can
help avoid inaccuracies, poor end-user experiences, intent errors, misalignment,
and potentially even
[disastrous agentic workflows](https://curity.io/blog/5-agentic-ai-security-incidents-and-vulnerabilities/).
## 3. Malicious attempts
_Are malicious attempts hitting the LLM?_
Even seemingly valid traffic to LLMs can be malicious, such as attackers
masquerading behind hijacked authenticated accounts to perform system
reconnaissance or attempt sensitive data retrieval. Other threats include prompt
injection, tool poisoning that corrupts LLM behavior, and DDoS-like attacks
designed to drain compute resources or rack up token use.
Security company
[Check Point reported](https://research.checkpoint.com/2026/ai-security-report-2026/)
that detections of malicious prompt-injection payloads increased fivefold
between March and May 2026. As such, it's a good idea to have runtime monitoring
in place to spot suspicious activity. For instance, deep behavioral runtime
monitoring can help identify malicious attempts as they happen.
With this sort of AI observability, you can monitor traffic and set policies to
flag unusual behavior, such as odd enumeration patterns or excessive resource
consumption. This can help reduce the risk of LLM-jacking that drains resources,
as well as attacks that could lead to data exfiltration.
## 4. Logs and traces
_Do you have enough context for auditing and debugging?_
Logs are incredibly important for enterprise-grade systems, enabling better
auditing and debugging. For AI traffic, this can include the model used,
timestamps, prompts, token usage, metadata on the requesting party, processing
time, tool calls, and more.
These micro fingerprints can help trace issues to their root causes. They can
inform cybersecurity investigations, for instance, by showing who the caller is,
what permissions they have, and what data or tools were accessed. They can also
help uncover performance degradation, context issues, failed tool calls, and
other problems.
This type of tracing is becoming commonplace among teams building agents.
[LangChain's 2026 State of Agent Engineering survey](https://www.langchain.com/resources/llm-monitoring-observability)
found that 89% of teams had agent observability instrumented.
From a business perspective, detailed, exportable logs and traces help identify
root causes and enable quicker fixes, supporting business continuity. This level
of visibility may also be necessary for security, auditing, and compliance.
## 5. Token usage
_How much is being used, including per department?_
Tokens are the units LLMs use to process inputs and generate outputs, and the
associated costs can quickly spiral without good monitoring and limits. With
nondeterministic agents, these costs become highly unpredictable.
A
[2026 Microsoft Research study](https://www.microsoft.com/en-us/research/publication/how-do-ai-agents-spend-your-money-analyzing-and-predicting-token-consumption-in-agentic-coding-tasks/)
found that agentic coding tasks consumed 1,000x more tokens than code reasoning
and code chat. Token usage could also vary by up to 30x across runs of the same
task.
With observability into token usage, you can understand and compare how much
various LLMs cost, what model providers are charging, and how that equates to
real-world usage. This is critical information if you want to assign token
limits to internal departments, teams, or individual users.
By tracking token usage, you can set usage caps to prevent cost overruns and
reduce unnecessary spending on expensive models. Understanding these cost
metrics can also help guide model routing, such as routing lower-complexity
tasks to cheaper models. For some enterprises, tracking unnecessary AI
consumption could also help inform ESG initiatives aimed at reducing resource
use.
## 6. Actual cost per call
_What is the average cost per call for your LLMs?_
This is a helpful, albeit nuanced metric. It tracks the average token usage per
LLM call and equates that to the average cost per call based on the model's
per-token pricing structure. This gives a more definite window into the costs of
LLM use.
For instance, Anthropic's Opus 5.5 currently costs $4 per million input tokens
and $20 per million output tokens. If your engineering calls average 50,000
input tokens and 10,000 output tokens each, that's about $0.40 per call. This
assumes every request results in a call to the model provider. If you're using
semantic caching, however — more on that below — your average cost per
application request could be lower.
Having detailed per-call cost metrics can help guide your use of AI
tremendously. This can inform decisions about how you prompt AI internally by
providing more accurate cost estimates. Cost-per-call metrics can also inform
model switching and other optimizations to make more efficient use of AI.
## 7. Semantic cache performance
_What are the cost-per-call metrics with and without semantic caching?_
[Semantic caching](https://thenewstack.io/what-is-semantic-caching/), or LLM
caching, is similar to traditional caching but for AI models. It stores
responses based on previous prompts and serves them when semantically similar
prompts match what's stored in the cache.
Semantic caching can
[greatly reduce tool calls](/blog/what-is-semantic-caching) and
LLM calls, reducing unnecessary waste while improving speed. For instance,
[AWS analyzed](https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/semantic-caching-benchmarks.html)
over 60,000 chatbot queries and found that individual cache hits were as much as
59x faster. They found semantic caching also yielded 51.9% cost savings while
maintaining 92.6% accuracy.
Others report similar results.
[Early testing from Portkey](https://portkey.ai/blog/reducing-llm-costs-and-latency-semantic-cache/)
found that semantic caching for simple Q&A queries achieved a roughly 20% cache
hit rate while maintaining 99% accuracy.
While the gains of semantic caching seem evident, the accuracy of cached results
should be continually evaluated. Savings should also be weighed against the cost
of the caching layer itself to ensure it's paying off. This can be done by
keeping tabs on cost per call with and without semantic caching.
Semantic caching can reduce unnecessary frontier AI model expenditure while
retaining similar application results. But tracking cache hit rates, accuracy,
latency, and cost savings validates whether semantic caching is actually a
cost-effective and performant addition to your AI architecture.
## AI visibility happens at the gateway layer
As enterprises mature their AI strategies, observability will increasingly
become a necessary ingredient. However, collecting, computing, and storing all
the metrics described above is easier said than done.
In practice, enterprises will likely need durable, pre-built infrastructure to
enable this sort of deep visibility, paired with instrumentation to automate
things like cost-per-call calculations, semantic caching, and model routing.
This is where an [AI gateway](/ai-gateway) can help. Since LLM
traffic passes through the gateway, it creates a natural control and observation
point for governing AI resource consumption and directing LLM usage.
This visibility can also inform actual business decisions around AI usage.
Organizations can set limits by company, department, team, or role, track
spending and token consumption, and use those metrics to inform automated model
routing.
AI gateways can also enable capabilities like
[semantic caching](/examples/semantic-caching) while monitoring
whether they're actually delivering the expected savings.
## AI gateways and the future of enterprise AI
[According to Gartner](https://www.gartner.com/en/newsroom/press-releases/2026-05-12-gartner-predicts-40-percent-of-organizations-deploying-ai-will-use-ai-observability-to-monitor-model-performance-by-2028),
40% of organizations deploying AI will use dedicated AI observability tools to
monitor model performance by 2028. And AI gateways will likely aid here. In
fact, the infrastructure is already being put into practice.
Another
[2026 Gartner survey](https://konghq.com/resources/reports/gartner-complimentary-report)
found that 68% of software engineering leaders were engaged with AI gateways:
15% had already deployed them, 31% were deploying them, and 22% were evaluating
the technology.
All in all, LLMs are helping agents, developers, and users reach newfound
capabilities. But this comes at a cost, and evaluating the performance, expense,
and security implications of LLMs in practice is important. As such, AI gateways
stand to serve an important function in the future of enterprise AI governance.
To see what that looks like in practice, start with the
[Zuplo AI Gateway introduction](https://zuplo.com/docs/ai-gateway/overview).