AI Gateway Analytics
- Applies to
- Gatewayruntime for API, AI, and MCP gateways
- Hosting
- Managed EdgeManaged DedicatedSelf-Hosted
Every request to an AI Gateway app records its model, provider, token counts, estimated cost, latency, and cache outcome. The Zuplo Portal turns those records into dashboards, so you can see what each app spends, which models carry the traffic, and how fast the providers answer, without adding any policy or code.
Where to find it
Open your AI Gateway project and select the Observability tab. It has two sub-tabs:
- Logs shows request logs and messages written by policies. See Logging.
- Analytics holds the dashboards described on this page. A sidebar lists its sections.
| Section | What it shows |
|---|---|
| Overview | A summary of tokens, cost, latency, consumers, and cache hit rate, plus model usage, latency distribution, and cache lookups. |
| Requests | Request volume, errors, and gateway latency for every request the gateway handled. See Requests. |
| Consumers | Traffic by caller. See Consumers. |
| Agents | Traffic from AI agent clients such as ChatGPT, Claude.ai, and Cursor. See Agents. |
| AI & LLMs | Requests, tokens, estimated cost, and median latency, broken down by model, provider, and token type, plus blocked requests and warnings. |
| Pools & Apps | Spend for the gateway, each pool, and each app, drawn as a tree against their budgets. See Budget alerts. |
| Cache | Semantic cache hits, misses, stores, saved tokens, and lookup latency. |
The time range, filters, and selected section live in the URL, so you can share a view by copying the link. How far back you can look depends on your plan. See Data retention.
Overview
The Overview section is the landing page for an AI Gateway project. Its summary cards show:
- Total Tokens, split into input and output tokens.
- Estimated Cost.
- Avg Latency, the average gateway request duration.
- Consumers, the number of distinct callers.
- Cache Hit Rate, hits divided by hits plus misses for the semantic cache.
Below the cards, Model Usage charts the busiest models over time, Latency Distribution shows the p50, p90, p95, and p99 latency of AI Gateway requests, and Cache Lookups charts semantic cache hits and misses. A Consumers table lists the callers with the most traffic.
AI & LLMs
The AI & LLMs section focuses on model traffic and spend.
| Panel | What it shows |
|---|---|
| Summary | Total Requests, Total Tokens (input and output), Estimated Cost, and Median Latency (p50). |
| Model Usage | The most-used models by share of AI Gateway requests, over time. |
| Model Share | Each model's share of cost, tokens, and requests, labeled provider/model. |
| Token Breakdown | Prompt, completion, and embedding tokens over time. |
| Provider Breakdown | Requests by provider over time. |
| Blocked & Warnings | Requests that a policy blocked, such as a guardrail or an exhausted budget, and budget warnings, by type and over time. |
Click a model, a provider, or a blocked-request type to filter the whole section to that value; click it again to clear the filter. Every panel, including the summary cards, reloads for the filter. To compare latency between models, click each model in turn and read its Median Latency.
Estimated Cost uses the provider's reported price when the provider returns one, and otherwise multiplies the token counts by the model's rate card. To set your own rates, see Custom pricing.
Cache
The Cache section reports on the Semantic Cache policy: Hit Rate, Hits, Misses, Stores, and Saved Tokens (the prompt and completion tokens a hit didn't send to the provider), plus hits and misses over time and p50, p95, and p99 cache lookup latency. A hit costs no provider tokens, so it adds no cost.
App and pool overviews
Each app and pool has an Overview tab with a Metrics frame for the last 30 days: Spend, Tokens, Requests, and Cache Hit Rate, each with its change from the previous 30 days, followed by tokens and cost over time. Select View in Analytics to open the AI & LLMs section filtered to that app or pool. See Monitoring usage.
How latency is measured
The dashboards report two kinds of latency. Both stop the clock when the gateway has the response and starts sending it. For a streaming response, that's when the stream starts, so the latency shows how quickly the model begins to answer.
| Metric | Starts | Ends |
|---|---|---|
| AI Gateway latency: Median Latency on AI & LLMs and Latency Distribution on Overview | When the AI Gateway handler receives the request, after the app's policy chain has run | When the provider's response is ready to return. For a streaming response, when the provider starts streaming |
| Gateway request duration: Avg Latency on Overview and the latency charts on Requests | When the gateway receives the request | When the gateway has the response to send, after every policy has run |
AI Gateway latency covers successful responses. It includes every provider call the request needed, such as a retry on a fallback model. For a request that the semantic cache answers, it's the time the cache took to find and return the stored response.
Send the data elsewhere
- To follow a request through the gateway's policies and into the provider call, add OpenTelemetry tracing. See AI Gateway tracing.
- To send logs to Datadog, Splunk, or another logging service, add a log plugin.
- To learn which fields the analytics record, and which features store prompt and completion content, see AI Gateway data handling.