Cookbook: Proxy the OpenAI Realtime API

The OpenAI Realtime API streams speech and text over a WebSocket. Server-side clients authenticate with an Authorization: Bearer header that carries an OpenAI API key. This cookbook puts a Zuplo route in front of that WebSocket so that:

• Clients authenticate with a Zuplo API key, and your OpenAI key stays on the gateway.
• The gateway sets the model in the backend URL, so the model a client puts in its own URL has no effect.
• A message policy ends any session in which the client sends its own instructions or a system message.

The route uses the WebSocket Pipeline Handler, which is an Enterprise feature.

Prerequisites

• A Zuplo project on Managed Edge.
• An OpenAI API key with access to a Realtime model.
• An API Key Authentication consumer for each client that connects through the gateway.

1. Store the OpenAI key and model

Add two environment variables to your project:

2. Add the policies

Add an API Key policy to authenticate clients, and a Set Headers policy that replaces the client's Authorization header with your OpenAI key. Add both to config/policies.json:

config/policies.json
{
  "policies": [
    {
      "name": "api-key-inbound",
      "policyType": "api-key-inbound",
      "handler": {
        "export": "ApiKeyInboundPolicy",
        "module": "$import(@zuplo/runtime)",
        "options": {
          "allowUnauthenticatedRequests": false
        }
      }
    },
    {
      "name": "set-openai-key",
      "policyType": "set-headers-inbound",
      "handler": {
        "export": "SetHeadersInboundPolicy",
        "module": "$import(@zuplo/runtime)",
        "options": {
          "headers": [
            {
              "name": "Authorization",
              "value": "Bearer $env(OPENAI_API_KEY)"
            }
          ]
        }
      }
    }
  ]
}

The API Key policy runs first and reads the client's Zuplo key from the Authorization header. The Set Headers policy then overwrites that header, so OpenAI receives your OpenAI key and never sees the client's key.

3. Write the message policy

Create modules/realtime-policies.ts. This inbound policy reads each event the client sends. Clients can change instructions in several ways: session.update, response.create with its own instructions, or a conversation.item.create that adds a system message. The policy checks every event for an instructions field or a system-role item at any depth. When it finds one, it closes the client connection, and the handler then closes the connection to OpenAI. Every other event passes through unchanged.

modules/realtime-policies.ts
import { ZuploContext, ZuploRequest } from "@zuplo/runtime";

// True when any object in the event sets instructions or adds a system message
function overridesInstructions(value: unknown): boolean {
  if (Array.isArray(value)) {
    return value.some(overridesInstructions);
  }
  if (value !== null && typeof value === "object") {
    const obj = value as Record<string, unknown>;
    if ("instructions" in obj || obj.role === "system") {
      return true;
    }
    return Object.values(obj).some(overridesInstructions);
  }
  return false;
}

export async function protectInstructions(
  data: string,
  target: WebSocket,
  source: WebSocket,
  request: ZuploRequest,
  context: ZuploContext,
) {
  // Forward anything that isn't a JSON event unchanged
  let event: unknown;
  try {
    event = JSON.parse(data);
  } catch {
    return data;
  }

  if (overridesInstructions(event)) {
    context.log.warn("Client tried to replace the instructions", {
      consumer: request.user?.sub,
    });
    source.send(
      JSON.stringify({
        type: "error",
        error: { message: "This gateway doesn't allow custom instructions." },
      }),
    );
    // `source` is the client connection in an inbound policy
    source.close(4003, "Custom instructions aren't allowed");
    return undefined; // don't forward the event to OpenAI
  }

  return data;
}

4. Add the route

Add a GET route that uses the WebSocket Pipeline Handler. The rewritePattern sets the model from the environment variable, so the model a client puts in its own URL has no effect.

config/routes.oas.json
"/v1/realtime": {
  "x-zuplo-path": {
    "pathMode": "open-api"
  },
  "get": {
    "summary": "OpenAI Realtime API",
    "operationId": "openai-realtime",
    "x-zuplo-route": {
      "corsPolicy": "none",
      "handler": {
        "export": "webSocketPipelineHandler",
        "module": "$import(@zuplo/runtime)",
        "options": {
          "rewritePattern": "wss://api.openai.com/v1/realtime?model=${env.OPENAI_REALTIME_MODEL}",
          "policies": {
            "inbound": [
              {
                "module": "$import(./modules/realtime-policies)",
                "export": "protectInstructions"
              }
            ]
          }
        }
      },
      "policies": {
        "inbound": ["api-key-inbound", "set-openai-key"]
      }
    }
  }
}

The route-level policies run once, when the client opens the connection. The policies inside handler.options run on every message. See Two different policies blocks.

5. Connect a client

Connect to your gateway URL with a Zuplo API key instead of an OpenAI key. This Node.js example uses the ws package:

Code
import WebSocket from "ws";

const ws = new WebSocket("wss://my-gateway.zuplo.app/v1/realtime", {
  headers: {
    Authorization: `Bearer ${process.env.ZUPLO_API_KEY}`,
  },
});

ws.on("open", () => {
  console.log("Connected through Zuplo");
});

ws.on("message", (message) => {
  console.log(JSON.parse(message.toString()));
});

ws.on("close", (code, reason) => {
  console.log(`Closed: ${code} ${reason.toString()}`);
});

Replace my-gateway.zuplo.app with your gateway's hostname. A request without a valid Zuplo API key gets a 401 response and no WebSocket connection. A client that sends an event with instructions, such as session.update or response.create, receives the error event and then a close with code 4003.

Next steps

• Add a Rate Limiting policy to the route-level inbound array to limit how many sessions each consumer opens.
• Run code when a session starts with the onOpen connection hook.
• Read more about message policies in the WebSocket Pipeline Handler reference.