# Build a custom caching policy Custom caching code is the most expensive caching you can ship. You own it, you debug it, and the failure mode is never a slow API. It's one customer reading another customer's data. Reach for it only after a built-in policy has failed to express what you need. ## Check the built-in policies first Most caching requirements are already a configuration option somewhere. Find your row before writing anything. | You want | Use this instead | | ----------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- | | Cache responses at the gateway so repeats skip your backend | [Cache at the gateway](./gateway-caching.mdx) | | A key that varies by `Accept` or `Accept-Language` | The Caching policy's `headers` option | | A key that varies by the raw `Authorization` header | The Caching policy includes it by default | | Flush every cached response at once | The [Caching policy's `cacheId`](./gateway-caching.mdx#invalidate-with-cacheid), bound to an environment variable | | Cache at the CDN so requests never reach the gateway at all | [Cache at the CDN](./cdn-caching.mdx) | | A different TTL per route, or per response | The [`cacheConfig` function](./dynamic-cache-rules.mdx) | | Purge tags for targeted CDN invalidation | The CDN policy's `tags` option | | Cache GraphQL query results by query and variables | [`graphql-cache-inbound`](../policies/graphql-cache-inbound.mdx) | | Cache LLM completions by prompt similarity | [`semantic-cache-inbound`](../policies/semantic-cache-inbound.mdx) | | Store an arbitrary JSON value with a TTL from your own code | [`ZoneCache`](../programmable-api/zone-cache.mdx) | ## When custom code is the right answer What survives that table is a short list. These are the cases where no combination of built-in options gets you there: - **The key depends on a decoded value, not a raw header.** Two JWTs for the same tenant are different strings, so keying on `Authorization` gives a near-zero hit rate. The `org_id` claim gives one entry per tenant. - **You cache only some status codes, or only small responses.** A 5 MB export isn't worth a cache slot; a 4 KB catalog page is. - **The TTL comes from the upstream response.** The backend knows how volatile each record is and says so in a header. - **Each tenant needs its own namespace with independent invalidation.** One tenant's write should drop that tenant's entries and leave every other tenant's untouched. - **You want soft purge or stale-while-revalidate behavior.** Serve the stale copy now and refresh it in the background. Everything else, which is most of it, belongs in a built-in policy. ## An inbound and outbound policy pair A gateway cache is two policies on one route. The inbound policy looks for a stored copy and returns a `Response` to short-circuit the pipeline on a hit. The outbound policy stores the response on the way back out. Client Inbound: cache.match Backend Outbound: cache.put Both exports live in one module so they share the key function and the options type. Register the read side: ```json title="config/policies.json" { "policies": [ { "name": "tenant-cache-read", "policyType": "custom-code-inbound", "handler": { "export": "tenantCacheInbound", "module": "$import(./modules/tenant-cache)", "options": { "namespace": "catalog", "defaultTtlSeconds": 300, "maxTtlSeconds": 3600, "maxBytes": 262144, "cacheableStatusCodes": [200, 203], "varyByQuery": ["page", "pageSize", "sort"] } } } ] } ``` Add a second entry named `tenant-cache-write` with `policyType` set to `custom-code-outbound`, `export` set to `tenantCacheOutbound`, and the same `options` object. The two policies have to agree on the namespace and on every input to the key, or the write lands somewhere the read never looks. Order matters on the route: authentication runs before the cache read, because the key depends on the authenticated identity. ```json title="config/routes.oas.json" { "policies": { "inbound": ["jwt-auth", "tenant-cache-read"], "outbound": ["tenant-cache-write"] } } ``` ## Build the cache key The Cache API keys entries by `Request`, so a custom key is a synthetic URL you build yourself. Keep the function pure: same request in, same string out. ```ts title="modules/tenant-cache.ts" import type { ZuploRequest } from "@zuplo/runtime"; export interface CacheOptions { namespace: string; defaultTtlSeconds: number; maxTtlSeconds: number; maxBytes: number; cacheableStatusCodes: number[]; varyByQuery: string[]; } export type TenantRequest = ZuploRequest<{ UserData: { org_id?: string } }>; /** The tenant this request reads for, or undefined when unauthenticated. */ export function getTenantId(request: TenantRequest): string | undefined { return request.user?.data?.org_id; } /** Stable, collision-free cache key. Pure: no clocks, no randomness. */ export function cacheKey( request: TenantRequest, tenantId: string, options: CacheOptions, ): string { const url = new URL(request.url); // Allowlist parameters, lowercase the names, and sort, so ?sort=name&page=2 // and ?page=2&sort=name produce one entry instead of two. const params = [...url.searchParams] .map(([name, value]) => [name.toLowerCase(), value] as [string, string]) .filter(([name]) => options.varyByQuery.includes(name)) .sort((a, b) => a[0].localeCompare(b[0]) || a[1].localeCompare(b[1])); const search = new URLSearchParams(params).toString(); const query = search ? `?${search}` : ""; const tenant = encodeURIComponent(tenantId); const path = url.pathname.toLowerCase(); return `https://cache.invalid/${options.namespace}/${tenant}${path}${query}`; } ``` Three rules do the real work here. **Determinism.** Sorting and lowercasing collapse the same logical request onto one entry. An unsorted key turns one cacheable resource into a pile of near-duplicates that each get read once. **Bounded cardinality.** `varyByQuery` is an allowlist, not a blocklist. A free-text `q=` parameter or a client-supplied timestamp has unbounded values, so every request writes an entry nobody reads again — a 0% hit rate that still costs a write per request. **Namespacing.** The tenant id sits in a fixed segment, URL-encoded so a value containing `/` cannot escape it. Dropping one tenant's entries becomes a prefix operation, and no two tenants can produce the same string. :::danger A cache key that omits the caller identity will serve one tenant's data to another. This is not a theoretical risk — it is the default outcome. If `getTenantId` returns `undefined`, the inbound policy must skip the cache entirely rather than fall back to a shared key. Never derive the tenant from a client-supplied header such as `X-Tenant-Id`; derive it from the authenticated `request.user` an authentication policy populated. Before shipping, add a test that sends one URL with two tenants' tokens and asserts the bodies differ. ::: ## Read the cache on the way in The inbound policy returns a `Response` to short-circuit, or the original `ZuploRequest` to continue to the backend. ```ts title="modules/tenant-cache.ts" import { ContextData, environment, type ZuploContext } from "@zuplo/runtime"; const CACHE_KEY = "tenant-cache-key"; const cacheEnabled = () => environment.CACHE_DISABLED !== "true"; export async function tenantCacheInbound( request: TenantRequest, context: ZuploContext, options: CacheOptions, ): Promise { const tenantId = getTenantId(request); // No identity, no shared cache. Never fall back to a tenant-less key. if (!cacheEnabled() || request.method !== "GET" || !tenantId) { return request; } const key = cacheKey(request, tenantId, options); ContextData.set(context, CACHE_KEY, key); const cache = await caches.open(options.namespace); const stored = await cache.match(new Request(key)); if (!stored) { context.log.debug("cache miss", { key: await keyDigest(key) }); return request; } // Copy into a fresh Response so the headers are mutable, then label the hit. const storedAt = Number(stored.headers.get("x-cache-stored-at") ?? 0); const age = Math.max(0, Math.floor((Date.now() - storedAt) / 1000)); const response = new Response(stored.body, { status: stored.status, headers: stored.headers, }); response.headers.set("cache-status", "zuplo; hit"); response.headers.set("age", String(age)); response.headers.delete("x-cache-stored-at"); // The stored Cache-Control is the Cache API's expiry clock, not advice for // the caller. A tenant-scoped body must not be shareable downstream. response.headers.set("cache-control", "private, no-store"); context.log.debug("cache hit", { key: await keyDigest(key), age }); return response; } ``` Returning the stored response ends the pipeline: the backend is never called and the outbound policies never run. That is the point, and it is also why the hit path has to set its own headers — nothing downstream will do it for you. ## Write the cache on the way out The outbound policy decides whether the response is worth storing, then stores a clone. ```ts title="modules/tenant-cache.ts" /** Clamp an upstream TTL so one bad header cannot pin a copy for a year. */ function resolveTtl(response: Response, options: CacheOptions): number { const raw = response.headers.get("x-cache-ttl"); const upstream = raw === null ? Number.NaN : Number.parseInt(raw, 10); return Number.isFinite(upstream) && upstream > 0 ? Math.min(upstream, options.maxTtlSeconds) : options.defaultTtlSeconds; } /** Re-wrap a response so its headers are mutable, then label it. */ function label(response: Response, status: string): Response { const copy = new Response(response.body, { status: response.status, headers: response.headers, }); copy.headers.set("cache-status", status); return copy; } export async function tenantCacheOutbound( response: Response, request: TenantRequest, context: ZuploContext, options: CacheOptions, ): Promise { const key = ContextData.get(context, CACHE_KEY); const size = Number(response.headers.get("content-length") ?? Number.NaN); const storable = cacheEnabled() && options.cacheableStatusCodes.includes(response.status) && Number.isFinite(size) && size <= options.maxBytes; if (key === undefined || !storable) { return label(response, "zuplo; fwd=miss"); } // Clone before storing. The original body still has to reach the client. const copy = response.clone(); const headers = new Headers(copy.headers); headers.set("cache-control", `max-age=${resolveTtl(response, options)}`); headers.set("x-cache-stored-at", Date.now().toString()); // Reusing upstream headers means inheriting upstream cache headers. Drop the // ones that would compete with the Cache-Control just set. headers.delete("expires"); headers.delete("etag"); headers.delete("last-modified"); const toStore = new Response(copy.body, { status: copy.status, headers }); const cache = await caches.open(options.namespace); void cache .put(new Request(key), toStore) .catch((err) => context.log.error("cache write failed", { err })); return label(response, "zuplo; fwd=miss; stored"); } ``` `Cache-Control` on the stored copy carries the TTL — the Cache API reads it back on `match`, so a stored response without it has no expiry you control. It is internal machinery, not advice for the caller, which is why the hit path overwrites it with `private, no-store` before returning: a tenant-scoped body must never carry a header that invites a browser or a shared proxy to hold onto it. A response with no `Content-Length`, such as a streamed or chunked body, fails the size check and is skipped, which is correct: you cannot bound what you cannot measure. :::warning Outbound policies run on every response, not only successful ones, so a `502` from a failing backend reaches this code. The `cacheableStatusCodes` allowlist is the only thing standing between an error and a cached error, so keep it explicit and keep it narrow. It also stops a `204` or `206` from being stored by accident. ::: ## Pass state between the two policies The outbound policy must not recompute the key. The request may have been rewritten by then, and any drift between the two computations means you write to a key nobody reads. Compute it once inbound and hand it forward with [`ContextData`](../programmable-api/context-data.mdx), which is scoped to one request and garbage collected when that request completes. Setting the key inbound doubles as the signal for the outbound policy. When `ContextData.get` returns `undefined`, the inbound policy skipped the cache — unauthenticated request, non-`GET` method, or kill switch on — so the outbound policy skips it too, with no second copy of that logic to keep in sync. ## Do not block the response on the cache write The `void cache.put(...)` above is deliberately not awaited. A client waiting on a successful response should not also wait on a cache write; the write is an optimization for the _next_ request. The `.catch` on it is mandatory, not stylistic. A floating promise that rejects becomes an unhandled rejection, and an unhandled rejection can fail the request that was already on its way out the door — a cache write error turning into a `500` for a response that was otherwise fine. Catching and logging keeps the failure in your logs and invisible to the caller. ## Correctness checklist Walk this before the pull request goes up. - **Never store a response whose body was already consumed.** `response.json()` or `response.text()` drains the stream, and everything downstream gets a `body-used` error. See [Safely clone a request or response](../programmable-api/safely-clone-a-request-or-response.mdx). - **Clone before storing.** `response.clone()` gives you a second readable body: store the clone, return the original. - **Never cache an error.** Check the status against an explicit allowlist. A cached `502` outlives the outage that caused it. - **Never let one tenant's key collide with another's.** URL-encode the tenant id and keep it in a fixed position in the key. - **Bound the key space.** Allowlist the query parameters that vary the response and ignore everything else. - **Always have a kill switch.** `CACHE_DISABLED=true` turns both policies into pass-throughs. Flipping an [environment variable](../articles/environment-variables.mdx) and redeploying takes minutes and needs no code edit or review while the cache is serving something wrong. ## Measure the hit rate The `Cache-Status` header set above ([RFC 9211](https://www.rfc-editor.org/rfc/rfc9211.html)) makes every response self-describing, so a developer chasing stale data reads the answer off their network tab instead of asking you. For aggregate hit rate, log a short digest of the key rather than the key itself — the key holds the tenant id and the query string, and neither belongs in a log line: ```ts title="modules/tenant-cache.ts" /** Short, stable, non-reversible label for a cache key. */ async function keyDigest(key: string): Promise { const bytes = new TextEncoder().encode(key); const digest = await crypto.subtle.digest("SHA-256", bytes); return [...new Uint8Array(digest)] .slice(0, 8) .map((byte) => byte.toString(16).padStart(2, "0")) .join(""); } ``` Hits divided by hits plus misses gives the hit rate, and grouping by digest surfaces the keys that never repeat — the ones to stop caching. The runtime supports the [Web Crypto API](../programmable-api/web-crypto-apis.mdx) directly, so this needs no dependency. ## Related If the reason you are here is that only part of each response is cacheable, read [Cache part of a response](./partial-response-caching.mdx) before building any of this. Caching a shared fragment and fetching only the caller-specific remainder is the same machinery one level down, and it usually beats caching the whole payload. - [Caching in Zuplo](./overview.mdx) — which layer to reach for, and why. - [Cache at the gateway](./gateway-caching.mdx) — the built-in policy this page replaces, and the options to exhaust before writing any code. - [Cache API](../programmable-api/cache.mdx) — the `caches.open`, `match`, and `put` reference behind the code above. - [ContextData](../programmable-api/context-data.mdx) — how the cache key travels from the inbound policy to the outbound one.