Copy page

AI Gateway

AI Gateway Semantic Cache Policy

The AI Gateway Semantic Cache returns a previous response when a new prompt is semantically similar, avoiding another provider call. One inbound policy checks the cache and stores successful responses on a miss; no outbound policy is required.

Place it after Metering so cache hits still count toward application request limits. Resolved applications are isolated by configuration ID; set namespace to partition traffic that has no configuration ID. Failures always pass the request through to the provider.

Configuration

The configuration shows how to configure the policy in the 'policies.json' document.

Code
{
  "name": "my-ai-gateway-semantic-cache-v2-inbound-policy",
  "policyType": "ai-gateway-semantic-cache-v2-inbound",
  "handler": {
    "export": "AIGatewaySemanticCacheV2InboundPolicy",
    "module": "$import(@zuplo/runtime)",
    "options": {
      "semanticTolerance": 0.2,
      "expirationSecondsTtl": 3600,
      "recentMessageCount": 1
    }
  }
}

Policy Configuration

• name <string> - The name of your policy instance. This is used as a reference in your routes.
• policyType <string> - The identifier of the policy. This is used by the Zuplo UI. Value should be ai-gateway-semantic-cache-v2-inbound.
• handler.export <string> - The name of the exported type. Value should be AIGatewaySemanticCacheV2InboundPolicy.
• handler.module <string> - The module containing the policy. Value should be $import(@zuplo/runtime).
• handler.options <object> - The options for this policy. See Policy Options below.

Policy Options

The options for this policy are specified below. All properties are optional unless specifically marked as required.

• semanticTolerance <number> - The semantic similarity threshold for semantic cache matches. Values closer to 0 require closer similarity, while larger values allow more flexible matching. Default is 0.2. Defaults to 0.2.
• expirationSecondsTtl <integer> - How long a cached response lives, in whole seconds. Defaults to 3600 (1 hour); maximum 2592000 (30 days). Defaults to 3600.
• namespace <string> - Partitions the cache when the authenticated application has no configuration id. Applications with a configuration id always use it, so cache entries stay isolated per application.
• recentMessageCount <integer> - How many of the conversation's most recent messages (input items on the Responses endpoint) form the cache key, both when storing a response and when matching later requests. Earlier messages are ignored, so a multi-turn conversation can hit an entry cached from a shorter one; system messages are always included. Requests whose keyed window contains no user message (for example a conversation ending in an assistant prefill) are not cached. Defaults to 1 (match on the newest message only). Defaults to 1.
• maxConversationLength <number> - Skip caching entirely when the conversation has more than this many messages. Unset by default (no limit). Set a bound to guard against topic drift: the cache key covers only the last recentMessageCount messages, so the deeper a conversation goes, the more likely a match on recent messages alone ignores earlier context that should change the answer.
• endpoints <string[]> - The endpoint shapes to cache. Defaults to all shapes: ['openai-chat', 'anthropic-messages', 'openai-responses'].
• onUnknownShape <string> - Unused by the cache (a cache miss is never unsafe, so it always fails open). Present for interface consistency. Allowed values are deny, skip. Defaults to "skip".

Using the Policy

AI Gateway Semantic Cache

AI Gateway Semantic Cache answers a request from a previous response when the new prompt is semantically similar to one already seen. A hit skips the provider call. One inbound policy handles both directions: it checks the cache on the way in and, on a miss, stores a successful provider response on the way out.

A cache hit returns immediately, so later inbound policies do not run. Place Semantic Cache after Metering so hits still count toward application request limits. Put guardrails before the cache when they must inspect every request.

How matching works

The cache builds an embedding from the system context and the most recent conversation messages. It then requires the endpoint format, model, system context, and non-conversation request parameters to match exactly before accepting a semantic match. This prevents a similar prompt from crossing model, parameter, or system-prompt boundaries.

semanticTolerance controls how loose the semantic match may be. A response is reused when similarity is at least 1 - semanticTolerance; the default 0.2 therefore requires a similarity of at least 0.8.

All three text-based AI Gateway shapes are supported: Chat Completions, Responses, and Anthropic Messages. Streaming responses are accumulated for storage and can be replayed as streams on a later hit.

Options

Option	Default	What it does
semanticTolerance	0.2	Controls the minimum semantic similarity. Smaller values require a closer match.
expirationSecondsTtl	3600	Keeps a cached response for this many seconds, up to 30 days.
recentMessageCount	1	Uses this many recent conversation messages in the semantic key. System and developer messages are always included.
maxConversationLength	Unset	Skips conversations longer than this many messages.
endpoints	All three shapes	Limits caching to selected endpoint shapes.
namespace	Resolved configuration ID, or unset	Partitions entries when no application configuration ID is available.

Increasing recentMessageCount makes the cache consider more conversational context. Setting maxConversationLength can reduce topic-drift risk when only a small recent window is used.

What isn't cached

• Requests whose selected message window has no user message.
• Selected messages containing images, files, audio, tool calls, tool results, or other content that cannot be represented safely as text.
• Conversations longer than maxConversationLength, when that option is set.
• Provider responses whose status is not 200.
• Empty, filtered, malformed, or otherwise unreadable responses.

Earlier multimodal messages outside the selected recent window do not prevent a text-only recent prompt from being cached.

The cache always fails open. If a lookup, match, or store operation fails, the request continues to the provider rather than being rejected.

Application isolation

When a request resolves to an application, its configuration ID fixes the namespace and isolates its cached responses from other applications. The namespace option only applies when the gateway has no application configuration ID. Set it explicitly when unidentified traffic must be isolated from other requests that also lack a configuration ID.

Read cache outcomes

Responses report the outcome in the RFC 9211 Cache-Status header under the cache name zp-aigw-sem-cache. Responses also include these headers:

• x-ai-gateway-cache: HIT or MISS.
• x-ai-gateway-cache-similarity: the similarity score for a hit.

Read more about how policies work