Build a custom caching policy

Custom caching code is the most expensive caching you can ship. You own it, you debug it, and the failure mode is never a slow API. It's one customer reading another customer's data. Reach for it only after a built-in policy has failed to express what you need.

Check the built-in policies first

Most caching requirements are already a configuration option somewhere. Find your row before writing anything.

When custom code is the right answer

What survives that table is a short list. These are the cases where no combination of built-in options gets you there:

• The key depends on a decoded value, not a raw header. Two JWTs for the same tenant are different strings, so keying on Authorization gives a near-zero hit rate. The org_id claim gives one entry per tenant.
• You cache only some status codes, or only small responses. A 5 MB export isn't worth a cache slot; a 4 KB catalog page is.
• The TTL comes from the upstream response. The backend knows how volatile each record is and says so in a header.
• Each tenant needs its own namespace with independent invalidation. One tenant's write should drop that tenant's entries and leave every other tenant's untouched.
• You want soft purge or stale-while-revalidate behavior. Serve the stale copy now and refresh it in the background.

Everything else, which is most of it, belongs in a built-in policy.

An inbound and outbound policy pair

A gateway cache is two policies on one route. The inbound policy looks for a stored copy and returns a Response to short-circuit the pipeline on a hit. The outbound policy stores the response on the way back out.

Both exports live in one module so they share the key function and the options type. Register the read side:

Code
{
  "policies": [
    {
      "name": "tenant-cache-read",
      "policyType": "custom-code-inbound",
      "handler": {
        "export": "tenantCacheInbound",
        "module": "$import(./modules/tenant-cache)",
        "options": {
          "namespace": "catalog",
          "defaultTtlSeconds": 300,
          "maxTtlSeconds": 3600,
          "maxBytes": 262144,
          "cacheableStatusCodes": [200, 203],
          "varyByQuery": ["page", "pageSize", "sort"]
        }
      }
    }
  ]
}

Add a second entry named tenant-cache-write with policyType set to custom-code-outbound, export set to tenantCacheOutbound, and the same options object. The two policies have to agree on the namespace and on every input to the key, or the write lands somewhere the read never looks.

Order matters on the route: authentication runs before the cache read, because the key depends on the authenticated identity.

Code
{
  "policies": {
    "inbound": ["jwt-auth", "tenant-cache-read"],
    "outbound": ["tenant-cache-write"]
  }
}

Build the cache key

The Cache API keys entries by Request, so a custom key is a synthetic URL you build yourself. Keep the function pure: same request in, same string out.

Code
import type { ZuploRequest } from "@zuplo/runtime";

export interface CacheOptions {
  namespace: string;
  defaultTtlSeconds: number;
  maxTtlSeconds: number;
  maxBytes: number;
  cacheableStatusCodes: number[];
  varyByQuery: string[];
}

export type TenantRequest = ZuploRequest<{ UserData: { org_id?: string } }>;

/** The tenant this request reads for, or undefined when unauthenticated. */
export function getTenantId(request: TenantRequest): string | undefined {
  return request.user?.data?.org_id;
}

/** Stable, collision-free cache key. Pure: no clocks, no randomness. */
export function cacheKey(
  request: TenantRequest,
  tenantId: string,
  options: CacheOptions,
): string {
  const url = new URL(request.url);

  // Allowlist parameters, lowercase the names, and sort, so ?sort=name&page=2
  // and ?page=2&sort=name produce one entry instead of two.
  const params = [...url.searchParams]
    .map(([name, value]) => [name.toLowerCase(), value] as [string, string])
    .filter(([name]) => options.varyByQuery.includes(name))
    .sort((a, b) => a[0].localeCompare(b[0]) || a[1].localeCompare(b[1]));

  const search = new URLSearchParams(params).toString();
  const query = search ? `?${search}` : "";
  const tenant = encodeURIComponent(tenantId);
  const path = url.pathname.toLowerCase();

  return `https://cache.invalid/${options.namespace}/${tenant}${path}${query}`;
}

Three rules do the real work here.

Determinism. Sorting and lowercasing collapse the same logical request onto one entry. An unsorted key turns one cacheable resource into a pile of near-duplicates that each get read once.

Bounded cardinality. varyByQuery is an allowlist, not a blocklist. A free-text q= parameter or a client-supplied timestamp has unbounded values, so every request writes an entry nobody reads again — a 0% hit rate that still costs a write per request.

Namespacing. The tenant id sits in a fixed segment, URL-encoded so a value containing / cannot escape it. Dropping one tenant's entries becomes a prefix operation, and no two tenants can produce the same string.

Read the cache on the way in

The inbound policy returns a Response to short-circuit, or the original ZuploRequest to continue to the backend.

Code
import { ContextData, environment, type ZuploContext } from "@zuplo/runtime";

const CACHE_KEY = "tenant-cache-key";

const cacheEnabled = () => environment.CACHE_DISABLED !== "true";

export async function tenantCacheInbound(
  request: TenantRequest,
  context: ZuploContext,
  options: CacheOptions,
): Promise<ZuploRequest | Response> {
  const tenantId = getTenantId(request);

  // No identity, no shared cache. Never fall back to a tenant-less key.
  if (!cacheEnabled() || request.method !== "GET" || !tenantId) {
    return request;
  }

  const key = cacheKey(request, tenantId, options);
  ContextData.set(context, CACHE_KEY, key);

  const cache = await caches.open(options.namespace);
  const stored = await cache.match(new Request(key));

  if (!stored) {
    context.log.debug("cache miss", { key: await keyDigest(key) });
    return request;
  }

  // Copy into a fresh Response so the headers are mutable, then label the hit.
  const storedAt = Number(stored.headers.get("x-cache-stored-at") ?? 0);
  const age = Math.max(0, Math.floor((Date.now() - storedAt) / 1000));

  const response = new Response(stored.body, {
    status: stored.status,
    headers: stored.headers,
  });
  response.headers.set("cache-status", "zuplo; hit");
  response.headers.set("age", String(age));
  response.headers.delete("x-cache-stored-at");

  // The stored Cache-Control is the Cache API's expiry clock, not advice for
  // the caller. A tenant-scoped body must not be shareable downstream.
  response.headers.set("cache-control", "private, no-store");

  context.log.debug("cache hit", { key: await keyDigest(key), age });
  return response;
}

Returning the stored response ends the pipeline: the backend is never called and the outbound policies never run. That is the point, and it is also why the hit path has to set its own headers — nothing downstream will do it for you.

Write the cache on the way out

The outbound policy decides whether the response is worth storing, then stores a clone.

Code
/** Clamp an upstream TTL so one bad header cannot pin a copy for a year. */
function resolveTtl(response: Response, options: CacheOptions): number {
  const raw = response.headers.get("x-cache-ttl");
  const upstream = raw === null ? Number.NaN : Number.parseInt(raw, 10);

  return Number.isFinite(upstream) && upstream > 0
    ? Math.min(upstream, options.maxTtlSeconds)
    : options.defaultTtlSeconds;
}

/** Re-wrap a response so its headers are mutable, then label it. */
function label(response: Response, status: string): Response {
  const copy = new Response(response.body, {
    status: response.status,
    headers: response.headers,
  });
  copy.headers.set("cache-status", status);
  return copy;
}

export async function tenantCacheOutbound(
  response: Response,
  request: TenantRequest,
  context: ZuploContext,
  options: CacheOptions,
): Promise<Response> {
  const key = ContextData.get<string>(context, CACHE_KEY);
  const size = Number(response.headers.get("content-length") ?? Number.NaN);

  const storable =
    cacheEnabled() &&
    options.cacheableStatusCodes.includes(response.status) &&
    Number.isFinite(size) &&
    size <= options.maxBytes;

  if (key === undefined || !storable) {
    return label(response, "zuplo; fwd=miss");
  }

  // Clone before storing. The original body still has to reach the client.
  const copy = response.clone();
  const headers = new Headers(copy.headers);
  headers.set("cache-control", `max-age=${resolveTtl(response, options)}`);
  headers.set("x-cache-stored-at", Date.now().toString());

  // Reusing upstream headers means inheriting upstream cache headers. Drop the
  // ones that would compete with the Cache-Control just set.
  headers.delete("expires");
  headers.delete("etag");
  headers.delete("last-modified");

  const toStore = new Response(copy.body, { status: copy.status, headers });
  const cache = await caches.open(options.namespace);
  void cache
    .put(new Request(key), toStore)
    .catch((err) => context.log.error("cache write failed", { err }));

  return label(response, "zuplo; fwd=miss; stored");
}

Cache-Control on the stored copy carries the TTL — the Cache API reads it back on match, so a stored response without it has no expiry you control. It is internal machinery, not advice for the caller, which is why the hit path overwrites it with private, no-store before returning: a tenant-scoped body must never carry a header that invites a browser or a shared proxy to hold onto it. A response with no Content-Length, such as a streamed or chunked body, fails the size check and is skipped, which is correct: you cannot bound what you cannot measure.

Pass state between the two policies

The outbound policy must not recompute the key. The request may have been rewritten by then, and any drift between the two computations means you write to a key nobody reads. Compute it once inbound and hand it forward with ContextData, which is scoped to one request and garbage collected when that request completes.

Setting the key inbound doubles as the signal for the outbound policy. When ContextData.get returns undefined, the inbound policy skipped the cache — unauthenticated request, non-GET method, or kill switch on — so the outbound policy skips it too, with no second copy of that logic to keep in sync.

Do not block the response on the cache write

The void cache.put(...) above is deliberately not awaited. A client waiting on a successful response should not also wait on a cache write; the write is an optimization for the next request.

The .catch on it is mandatory, not stylistic. A floating promise that rejects becomes an unhandled rejection, and an unhandled rejection can fail the request that was already on its way out the door — a cache write error turning into a 500 for a response that was otherwise fine. Catching and logging keeps the failure in your logs and invisible to the caller.

Correctness checklist

Walk this before the pull request goes up.

• Never store a response whose body was already consumed. response.json() or response.text() drains the stream, and everything downstream gets a body-used error. See Safely clone a request or response.
• Clone before storing. response.clone() gives you a second readable body: store the clone, return the original.
• Never cache an error. Check the status against an explicit allowlist. A cached 502 outlives the outage that caused it.
• Never let one tenant's key collide with another's. URL-encode the tenant id and keep it in a fixed position in the key.
• Bound the key space. Allowlist the query parameters that vary the response and ignore everything else.
• Always have a kill switch. CACHE_DISABLED=true turns both policies into pass-throughs. Flipping an environment variable and redeploying takes minutes and needs no code edit or review while the cache is serving something wrong.

Measure the hit rate

The Cache-Status header set above (RFC 9211) makes every response self-describing, so a developer chasing stale data reads the answer off their network tab instead of asking you.

For aggregate hit rate, log a short digest of the key rather than the key itself — the key holds the tenant id and the query string, and neither belongs in a log line:

Code
/** Short, stable, non-reversible label for a cache key. */
async function keyDigest(key: string): Promise<string> {
  const bytes = new TextEncoder().encode(key);
  const digest = await crypto.subtle.digest("SHA-256", bytes);
  return [...new Uint8Array(digest)]
    .slice(0, 8)
    .map((byte) => byte.toString(16).padStart(2, "0"))
    .join("");
}

Hits divided by hits plus misses gives the hit rate, and grouping by digest surfaces the keys that never repeat — the ones to stop caching. The runtime supports the Web Crypto API directly, so this needs no dependency.

Related

If the reason you are here is that only part of each response is cacheable, read Cache part of a response before building any of this. Caching a shared fragment and fetching only the caller-specific remainder is the same machinery one level down, and it usually beats caching the whole payload.

• Caching in Zuplo — which layer to reach for, and why.
• Cache at the gateway — the built-in policy this page replaces, and the options to exhaust before writing any code.
• Cache API — the caches.open, match, and put reference behind the code above.
• ContextData — how the cache key travels from the inbound policy to the outbound one.