Isolate semantic cache entries by tenant

The Semantic Cache policy (semantic-cache-inbound) answers a request from an earlier response when the new request means the same thing, even when the wording differs. That makes it a good fit in front of an LLM or a natural-language search API. When several tenants share a route, give each tenant its own namespace so that one tenant's cached answer never reaches another tenant.

A namespace partitions the cache. A lookup only matches entries stored in the same namespace, and the policy stores a response in the namespace of the request that produced it.

Set the namespace for each request

Set cacheBy to function and point cacheByFunction at a function in your project. The policy calls the function on each request. The function returns the cache key and, optionally, settings for that request only:

A value the function returns overrides the policy option of the same name for that request. A value it leaves out falls back to the policy option. With no namespace from either, the policy uses the default namespace.

This function uses the question in the request body as the key and the authenticated caller as the namespace:

modules/semantic-cache-key.ts
import { ZuploContext, ZuploRequest } from "@zuplo/runtime";

export async function tenantCacheKey(
  request: ZuploRequest,
  context: ZuploContext,
  policyName: string,
) {
  // Clone so the body is still available for the backend
  const body = await request.clone().json();

  return {
    cacheKey: body.question,
    namespace: `tenant:${request.user?.sub ?? "anonymous"}`,
  };
}

Read the body from request.clone(), as in the example. Reading the original request consumes the body before it reaches your backend.

If the function throws, or returns no string cacheKey, the policy logs an error and the request continues to the backend without caching.

Configure the policy

Reference the function in the policy options:

config/policies.json
{
  "name": "semantic-cache",
  "policyType": "semantic-cache-inbound",
  "handler": {
    "export": "SemanticCacheInboundPolicy",
    "module": "$import(@zuplo/runtime)",
    "options": {
      "cacheBy": "function",
      "cacheByFunction": {
        "module": "$import(./modules/semantic-cache-key)",
        "export": "tenantCacheKey"
      },
      "expirationSecondsTtl": 3600,
      "returnCacheStatusHeader": true
    }
  }
}

Add the policy to the route after your authentication policy, so that the function can read request.user:

config/routes.oas.json
"policies": {
  "inbound": ["api-key-inbound", "semantic-cache"]
}

With returnCacheStatusHeader set to true, every response carries a zp-semantic-cache header with HIT or MISS. Set cacheStatusHeaderName to use a different header name.

Check the results

Send a question twice with one API key, then once with a key for a different consumer:

To watch hit rates over time, open Observability → Analytics in your project and select the Cache section. The Policies table shows hits, misses, and hit rate for each caching policy.

AI Gateway apps

For traffic through the AI Gateway, use the AI Gateway Semantic Cache policy in the app's policy chain. It keys on the prompt for you and keeps each app's cached responses apart, and the tokens its hits save appear as Saved Tokens in the Cache analytics.

Related

• Semantic Cache policy reference
• Cache at the gateway
• Caching overview