# AI Gateway Semantic Cache Policy

:::note{title="AI Gateway Policy"}

This policy is for use with the [AI Gateway](/docs/ai-gateway/introduction). See
the AI Gateway documentation to learn how to configure and govern AI models
with Zuplo.

:::

The AI Gateway Semantic Cache returns a previous response when a new prompt is
semantically similar, avoiding another provider call. One inbound policy checks
the cache and stores successful responses on a miss; no outbound policy is
required.

Place it after Metering so cache hits still count toward application request
limits. Resolved applications are isolated by configuration ID; set `namespace`
to partition traffic that has no configuration ID. Failures always pass the
request through to the provider.

:::info{title="Enterprise Feature"}

This policy is only available as part of our enterprise plans. It's free to try only any plan for development only purposes. If you would like to use this in production reach out to us: [sales@zuplo.com](mailto:sales@zuplo.com)

:::

## Configuration

The configuration shows how to configure the policy in the 'policies.json' document.

```json title="config/policies.json"
{
  "name": "my-ai-gateway-semantic-cache-v2-inbound-policy",
  "policyType": "ai-gateway-semantic-cache-v2-inbound",
  "handler": {
    "export": "AIGatewaySemanticCacheV2InboundPolicy",
    "module": "$import(@zuplo/runtime)",
    "options": {
      "semanticTolerance": 0.2,
      "expirationSecondsTtl": 3600,
      "recentMessageCount": 1
    }
  }
}
```

### Policy Configuration

- `name` <code className="text-green-600">&lt;string&gt;</code> - The name of your policy instance. This is used as a reference in your routes.
- `policyType` <code className="text-green-600">&lt;string&gt;</code> - The identifier of the policy. This is used by the Zuplo UI. Value should be `ai-gateway-semantic-cache-v2-inbound`.
- `handler.export` <code className="text-green-600">&lt;string&gt;</code> - The name of the exported type. Value should be `AIGatewaySemanticCacheV2InboundPolicy`.
- `handler.module` <code className="text-green-600">&lt;string&gt;</code> - The module containing the policy. Value should be `$import(@zuplo/runtime)`.
- `handler.options` <code className="text-green-600">&lt;object&gt;</code> - The options for this policy. [See Policy Options](#policy-options) below.

### Policy Options

The options for this policy are specified below. All properties are optional unless specifically marked as required.

- `semanticTolerance` <code className="text-green-600">&lt;number&gt;</code> - The semantic similarity threshold for semantic cache matches. Values closer to 0 require closer similarity, while larger values allow more flexible matching. Default is 0.2. Defaults to `0.2`.
- `expirationSecondsTtl` <code className="text-green-600">&lt;integer&gt;</code> - How long a cached response lives, in whole seconds. Defaults to 3600 (1 hour); maximum 2592000 (30 days). Defaults to `3600`.
- `namespace` <code className="text-green-600">&lt;string&gt;</code> - Partitions the cache when the authenticated application has no configuration id. Applications with a configuration id always use it, so cache entries stay isolated per application.
- `recentMessageCount` <code className="text-green-600">&lt;integer&gt;</code> - How many of the conversation's most recent messages (input items on the Responses endpoint) form the cache key, both when storing a response and when matching later requests. Earlier messages are ignored, so a multi-turn conversation can hit an entry cached from a shorter one; system messages are always included. Requests whose keyed window contains no user message (for example a conversation ending in an assistant prefill) are not cached. Defaults to 1 (match on the newest message only). Defaults to `1`.
- `maxConversationLength` <code className="text-green-600">&lt;number&gt;</code> - Skip caching entirely when the conversation has more than this many messages. Unset by default (no limit). Set a bound to guard against topic drift: the cache key covers only the last recentMessageCount messages, so the deeper a conversation goes, the more likely a match on recent messages alone ignores earlier context that should change the answer.
- `endpoints` <code className="text-green-600">&lt;string[]&gt;</code> - The endpoint shapes to cache. Defaults to all shapes: \['openai-chat', 'anthropic-messages', 'openai-responses'\].
- `onUnknownShape` <code className="text-green-600">&lt;string&gt;</code> - Unused by the cache (a cache miss is never unsafe, so it always fails open). Present for interface consistency. Allowed values are `deny`, `skip`. Defaults to `"skip"`.

## Using the Policy

# AI Gateway Semantic Cache

AI Gateway Semantic Cache answers a request from a previous response when the
new prompt is semantically similar to one already seen. A hit skips the provider
call. One inbound policy handles both directions: it checks the cache on the way
in and, on a miss, stores a successful provider response on the way out.

A cache hit returns immediately, so later inbound policies do not run. Place
Semantic Cache after Metering so hits still count toward application request
limits. Put guardrails before the cache when they must inspect every request.

## How matching works

The cache builds an embedding from the system context and the most recent
conversation messages. It then requires the endpoint format, model, system
context, and non-conversation request parameters to match exactly before
accepting a semantic match. This prevents a similar prompt from crossing model,
parameter, or system-prompt boundaries.

`semanticTolerance` controls how loose the semantic match may be. A response is
reused when similarity is at least `1 - semanticTolerance`; the default `0.2`
therefore requires a similarity of at least `0.8`.

All three text-based AI Gateway shapes are supported: Chat Completions,
Responses, and Anthropic Messages. Streaming responses are accumulated for
storage and can be replayed as streams on a later hit.

## Options

| Option                  | Default                             | What it does                                                                                                        |
| ----------------------- | ----------------------------------- | ------------------------------------------------------------------------------------------------------------------- |
| `semanticTolerance`     | `0.2`                               | Controls the minimum semantic similarity. Smaller values require a closer match.                                    |
| `expirationSecondsTtl`  | `3600`                              | Keeps a cached response for this many seconds, up to 30 days.                                                       |
| `recentMessageCount`    | `1`                                 | Uses this many recent conversation messages in the semantic key. System and developer messages are always included. |
| `maxConversationLength` | Unset                               | Skips conversations longer than this many messages.                                                                 |
| `endpoints`             | All three shapes                    | Limits caching to selected endpoint shapes.                                                                         |
| `namespace`             | Resolved configuration ID, or unset | Partitions entries when no application configuration ID is available.                                               |

Increasing `recentMessageCount` makes the cache consider more conversational
context. Setting `maxConversationLength` can reduce topic-drift risk when only a
small recent window is used.

## What isn't cached

- Requests whose selected message window has no user message.
- Selected messages containing images, files, audio, tool calls, tool results,
  or other content that cannot be represented safely as text.
- Conversations longer than `maxConversationLength`, when that option is set.
- Provider responses whose status is not `200`.
- Empty, filtered, malformed, or otherwise unreadable responses.

Earlier multimodal messages outside the selected recent window do not prevent a
text-only recent prompt from being cached.

The cache always fails open. If a lookup, match, or store operation fails, the
request continues to the provider rather than being rejected.

## Application isolation

When a request resolves to an application, its configuration ID fixes the
namespace and isolates its cached responses from other applications. The
`namespace` option only applies when the gateway has no application
configuration ID. Set it explicitly when unidentified traffic must be isolated
from other requests that also lack a configuration ID.

## Read cache outcomes

Responses report the outcome in the RFC 9211 `Cache-Status` header under the
cache name `zp-aigw-sem-cache`. Responses also include these headers:

- `x-ai-gateway-cache`: `HIT` or `MISS`.
- `x-ai-gateway-cache-similarity`: the similarity score for a hit.

Read more about [how policies work](/articles/policies)