Cache
- Applies to
- Gatewayruntime for API, AI, and MCP gateways
- Hosting
- Managed EdgeManaged DedicatedSelf-Hosted
The Cache section reports every lookup that a caching policy makes: how often the cache answers a request, how fast each lookup is, and, for AI Gateway semantic caching, how many tokens the cache saved you. It covers both HTTP caching with the Caching policy and semantic caching with the Semantic Cache and AI Gateway Semantic Cache policies. It appears for every project and at account scope.
When to use this
- Check how often each cache answers requests, by hit rate and by lookup volume.
- See how many LLM tokens semantic cache hits kept from reaching the provider.
- Compare HTTP and semantic caching, or one caching policy against another.
- Find the routes that hit the cache most, and the ones that rarely do.
Summary KPIs
| Name | What it measures |
|---|---|
| Hit Rate | Hits ÷ (hits + misses) in the window. Shows a dash when there were no lookups. |
| Hits | Lookups the cache answered. |
| Misses | Lookups that went on to the backend or the model provider. |
| Stores | Responses written into the cache for later lookups. |
| Saved Tokens | Tokens that AI Gateway Semantic Cache hits kept from reaching the provider. Secondary: input · output split. These tokens add $0 to consumption. Appears when the window has at least one saved token. |
Saved Tokens comes from the token usage in each cached response, so it counts hits on AI Gateway Semantic Cache. The general Semantic Cache policy can cache any JSON route and doesn't report saved tokens.
Charts
Cache Lookups Over Time. Hits and misses per interval.
Lookup Latency. P50, P95, and P99 cache lookup latency in the window.
Cache Latency Over Time. P50, P95, and P99 lookup latency per interval. What to look for: a rise in lookup latency adds to every request on the route, hit or miss. Filter by cache type or policy to find which cache it comes from.
Cache Types. Lookup volume split between HTTP and Semantic, with a By Cache Type table of hits, misses, and hit rate for each.
Tables
Policies. One row per caching policy, by policy name.
| Column | Notes |
|---|---|
| Policy | The policy name from your policy configuration. |
| Type | HTTP or Semantic. |
| Hits | Lookups the policy answered. |
| Misses | Lookups the policy passed on. |
| Hit Rate | Hits ÷ (hits + misses). |
Top Routes. The routes with the most cache hits, with the same Hits, Misses, and Hit Rate columns.
Both tables sort on any column (default: hits descending). Click a cache type, policy, or route to filter the whole section to it, and click it again to clear the filter.
Filters
The filter bar applies. See Shared controls.
Troubleshooting
The section is empty. No caching policy made a lookup in the selected window. Add a caching policy to a route and send traffic through it. See Cache at the gateway.
The Saved Tokens card is missing. No AI Gateway Semantic Cache hit saved tokens in the window. Hits on the general Semantic Cache policy and on the Caching policy don't report tokens.
The hit rate is low. Each lookup that misses still has to reach the backend. For HTTP caching, see how the cache key affects the hit rate in Shape the cache key with headers. For semantic caching, see Isolate semantic cache entries by tenant.