# Semantic cache

`ai-gateway-semantic-cache-v2-inbound`

The semantic cache answers a request from a previous response when the new
prompt is semantically similar to one already seen, skipping the provider
entirely. One policy handles both halves: it checks the cache on the way in and,
on a miss, stores the provider's response on the way out. There's no separate
outbound policy to add.

A cache hit answers the request, so entries after it in the chain don't run.
Place it after [Budgets and Costs](./metering.mdx) so hits still count toward
the app's request limits.

## Options

| Option                  | Default                              | What it does                                                                                        |
| ----------------------- | ------------------------------------ | --------------------------------------------------------------------------------------------------- |
| `semanticTolerance`     | `0.2`                                | How loose a match to accept. A response is reused at similarity ≥ 1 − tolerance, so `0.2` needs 0.8 |
| `expirationSecondsTtl`  | `3600`                               | How long a cached response stays valid, up to 30 days                                               |
| `maxConversationLength` | `3`                                  | Longest conversation to cache, in messages                                                          |
| `endpoints`             | Chat Completions, Anthropic Messages | Which API shapes to cache                                                                           |
| `namespace`             | The app's ID                         | Cache partition. Only applies when the request has no app configuration                             |

## What isn't cached

- Conversations longer than `maxConversationLength`—three messages by
  default—aren't cached at all.
- The Responses API isn't supported. The cache covers Chat Completions and
  Anthropic Messages.
- The cache always fails open: if the cache is unavailable, the request goes to
  the provider as normal.

One app can never read another's cached responses. The namespace is fixed to the
app's ID whenever the request resolves to an app, so the `namespace` option only
matters on a gateway that doesn't identify apps.

## Reading cache outcomes

Each response carries the outcome as a `Cache-Status` header under the cache
name `zp-aigw-sem-cache`, alongside `x-ai-gateway-cache` (`HIT` or `MISS`) and
`x-ai-gateway-cache-similarity`.