Semantic cache
ai-gateway-semantic-cache-v2-inbound
The semantic cache answers a request from a previous response when the new prompt is semantically similar to one already seen, skipping the provider entirely. One policy handles both halves: it checks the cache on the way in and, on a miss, stores the provider's response on the way out. There's no separate outbound policy to add.
A cache hit answers the request, so entries after it in the chain don't run. Place it after Budgets and Costs so hits still count toward the app's request limits.
Options
| Option | Default | What it does |
|---|---|---|
semanticTolerance | 0.2 | How loose a match to accept. A response is reused at similarity ≥ 1 − tolerance, so 0.2 needs 0.8 |
expirationSecondsTtl | 3600 | How long a cached response stays valid, up to 30 days |
maxConversationLength | 3 | Longest conversation to cache, in messages |
endpoints | Chat Completions, Anthropic Messages | Which API shapes to cache |
namespace | The app's ID | Cache partition. Only applies when the request has no app configuration |
What isn't cached
- Conversations longer than
maxConversationLength—three messages by default—aren't cached at all. - The Responses API isn't supported. The cache covers Chat Completions and Anthropic Messages.
- The cache always fails open: if the cache is unavailable, the request goes to the provider as normal.
One app can never read another's cached responses. The namespace is fixed to the
app's ID whenever the request resolves to an app, so the namespace option only
matters on a gateway that doesn't identify apps.
Reading cache outcomes
Each response carries the outcome as a Cache-Status header under the cache
name zp-aigw-sem-cache, alongside x-ai-gateway-cache (HIT or MISS) and
x-ai-gateway-cache-similarity.