Copy page

Observability

Zuplo OpenTelemetry

Applies to

Gatewayruntime for API, AI, and MCP gateways

Hosting

Managed EdgeManaged DedicatedSelf-Hosted

Zuplo ships with an OpenTelemetry plugin (@zuplo/otel) that instruments your API and exports traces and logs in OpenTelemetry format. The quickest way to use it is Zuplo's built-in tracing: add the plugin and your traces are stored by Zuplo and shown in the portal's Observability tab, with no collector to run or backend to host. Built-in tracing is available on every plan. You can also export to your own OpenTelemetry backend (an Enterprise feature), or to both at once.

Tracing

Tracing helps you monitor performance, identify bottlenecks, and troubleshoot issues in your Zuplo API. The OpenTelemetry plugin automatically instruments your API, so you get timings for each request along with spans for policies, handlers, and subrequests. It supports trace propagation (W3C headers by default), so you can follow a request from the client through to your backend.

What's Traced?

By default, when the OpenTelemetry plugin is enabled, the following is traced:

• Request: The entire request lifecycle is traced, including the time taken to process the request and send the response.
• Inbound Policies: The time taken to execute all inbound policies as well as each policy is traced.
• Handler: The request handler is traced.
• Outbound Policies: The time taken to execute all outbound policies as well as each policy is traced.
• Subrequests: Any use of fetch within your custom policies or handlers is traced.
• MCP operations: Every MCP message that the MCP Server handler or the MCP Gateway handles gets a span that follows the OpenTelemetry semantic conventions for MCP. See MCP tracing.
• AI Gateway calls: Every call the AI Gateway makes to an LLM provider gets a GenAI inference span with the model, token usage, and cost. See AI Gateway tracing.

Limitations

One important limitation to keep in mind is that the clock will only increment when performing I/O operations (for example when calling fetch, using the Cache APIs, etc.). This is a limitation imposed as a security measure due to Zuplo's serverless, multi-tenant architecture. In practice this shouldn't impact your ability to trace, as virtually any code that isn't I/O bound is fast.

Setup

Add the OpenTelemetryPlugin in your zuplo.runtime.ts file. Where it sends data is up to you: Zuplo's built-in storage, your own backend, or both.

Send Traces to Zuplo

Add the plugin with no configuration. It sends traces to Zuplo and names the service after your project:

zuplo.runtime.ts
import { OpenTelemetryPlugin } from "@zuplo/otel";
import { RuntimeExtensions } from "@zuplo/runtime";

export function runtimeInit(runtime: RuntimeExtensions) {
  runtime.addPlugin(new OpenTelemetryPlugin());
}

When you created your project, Zuplo added this plugin with a sampling ratio already set.

Deploy your project and open the Observability tab to see traces.

Each trace is tagged with the account, project, deployment, and environment it ran in, plus the request ID (the zp-rid value) that also appears on your logs, so you can move between a request's logs and its trace. How long traces are kept depends on your plan — see data retention.

Export to Your Own Backend

To send traces to an OpenTelemetry service such as Honeycomb, Middleware, Dynatrace, Jaeger, or others, configure an exporter with the service's url and headers. It's common for providers to use a header for authorization. Configuring an exporter sends traces there instead of Zuplo.

zuplo.runtime.ts
import { OpenTelemetryPlugin } from "@zuplo/otel";
import { RuntimeExtensions, environment } from "@zuplo/runtime";

export function runtimeInit(runtime: RuntimeExtensions) {
  runtime.addPlugin(
    new OpenTelemetryPlugin({
      exporter: {
        url: "https://otel-collector.example.com/v1/traces",
        headers: {
          "api-key": environment.OTEL_API_KEY,
        },
      },
      service: {
        name: "my-api",
        version: "1.0.0",
      },
    }),
  );
}

Sampling and Post-Processing

The plugin supports additional options for advanced use cases, including head sampling and post-processing of spans before export.

zuplo.runtime.ts
import { OpenTelemetryPlugin } from "@zuplo/otel";
import { RuntimeExtensions, environment } from "@zuplo/runtime";

export function runtimeInit(runtime: RuntimeExtensions) {
  runtime.addPlugin(
    new OpenTelemetryPlugin({
      exporter: {
        url: "https://otel-collector.example.com/v1/traces",
        headers: { "api-key": environment.OTEL_API_KEY },
      },
      service: {
        name: "my-api",
        version: "1.0.0",
      },
      // Optional post processor to modify spans before export
      postProcessor: (spans) => {
        for (const span of spans) {
          (
            span as unknown as { attributes: Record<string, unknown> }
          ).attributes["post.processed"] = true;
        }
        return spans;
      },
      sampling: {
        headSampler: {
          ratio: 0.1, // Sample 10% of requests
        },
      },
    }),
  );
}

Logs and Traces Together

To export logs as well as traces, use the top-level traceUrl, logUrl, and headers properties instead of the exporter object. The plugin supports both shapes, but they're mutually exclusive: exporter configures tracing only, while traceUrl and logUrl configure tracing and logging together with a shared set of headers. See Logging for how to emit log records.

zuplo.runtime.ts
import { OpenTelemetryPlugin } from "@zuplo/otel";
import { RuntimeExtensions, environment } from "@zuplo/runtime";

export function runtimeInit(runtime: RuntimeExtensions) {
  runtime.addPlugin(
    new OpenTelemetryPlugin({
      logUrl: "https://otel-collector.example.com/v1/logs",
      traceUrl: "https://otel-collector.example.com/v1/traces",
      headers: { "api-key": environment.OTEL_API_KEY },
      service: {
        name: "my-api",
        version: "1.0.0",
      },
      sampling: {
        headSampler: {
          ratio: 0.1, // Sample 10% of requests
        },
      },
    }),
  );
}

Send Traces to Zuplo and Your Own Backend

You aren't limited to one destination. To deliver traces to Zuplo and your own backend at the same time, add a span processor for each. Zuplo is represented by a ZuploSpanExporter. The processors batch and flush independently, so a slow or failing backend doesn't hold up the other:

zuplo.runtime.ts
import {
  OpenTelemetryPlugin,
  OTLPSpanExporter,
  BatchTraceSpanProcessor,
  ZuploSpanExporter,
} from "@zuplo/otel";
import { RuntimeExtensions, environment } from "@zuplo/runtime";

export function runtimeInit(runtime: RuntimeExtensions) {
  runtime.addPlugin(
    new OpenTelemetryPlugin({
      service: { name: "my-api" },
      spanProcessors: [
        new BatchTraceSpanProcessor(
          new OTLPSpanExporter({
            url: "https://otel-collector.example.com/v1/traces",
            headers: { "api-key": environment.OTEL_API_KEY },
          }),
        ),
        new BatchTraceSpanProcessor(new ZuploSpanExporter()),
      ],
    }),
  );
}

MCP Tracing

When the OpenTelemetry plugin is registered, the MCP Server handler and the MCP Gateway record a span for every MCP request and notification they handle. The spans follow the OpenTelemetry semantic conventions for MCP, so a backend that understands GenAI and MCP telemetry recognizes the span names, kinds, and attributes without any mapping.

There's nothing extra to enable. Register the plugin as described in Setup and MCP spans appear in the Observability tab and in any backend you export to. The plugin's sampling settings apply to MCP spans the same way they apply to the rest of the trace.

All MCP spans use the instrumentation scope mcp, whether they come from an MCP Server route or the MCP Gateway, so one filter selects every MCP span in a project.

Span Names and Kinds

Each span is named after the JSON-RPC method, plus the tool or prompt name when there is one:

Every method that MCP defines up to protocol revision 2026-07-28 keeps its name, including server/discover and subscriptions/listen. Only a method MCP doesn't define becomes _OTHER.

The tool or prompt name becomes part of the span name only when the server actually has that tool or prompt. A call to a tool that doesn't exist produces a span named tools/call, and the name the client sent is still recorded in gen_ai.tool.name. This keeps span names bounded, so a client can't create a new span name for every request. Resource URIs never appear in span names. On the MCP Gateway, span names don't include the tool or prompt name; read it from the gen_ai.tool.name or gen_ai.prompt.name attribute.

The MCP Server handler records a SERVER span. The MCP Gateway records two spans for each message it proxies: a SERVER span for the message as the gateway received it from the MCP client, and a CLIENT span, its child, for the message as the gateway forwarded it to the upstream MCP server.

Span Attributes

Values the MCP client chooses, such as tool names, resource URIs, request IDs, and error messages, are cut to 256 characters.

Errors

An MCP operation can fail in several ways. The span records each one with an error.type attribute and an error status:

Errors caused by the client's request aren't errors of the server, so SERVER spans record rpc.response.status_code for them but no error.type and no error status. That covers parse errors (-32700), invalid requests (-32600), unknown methods (-32601), invalid parameters, including a call to a tool that doesn't exist (-32602), unknown resources (-32002), and requests the transport rejects with an HTTP 4xx status, such as an unsupported protocol version. On the MCP Gateway CLIENT span, every JSON-RPC error counts as an error, because it's the upstream's answer to a request the gateway sent.

A tool that fails still returns HTTP 200 with an isError result, so the request's entry span reports success while the tools/call span reports tool_error. Filter on the MCP span to find failed tool calls.

Trace Structure

MCP spans sit inside the request's trace, under the handler span. On an MCP Server route, the tool's own route runs inside the tools/call span, so the policies, handler, and outbound fetch calls of that route appear as its children:

Code
|--- POST /mcp                                  (entry span)
|    |
|    |--- handler:mcpServerHandler
|    |    |
|    |    |--- tools/call get_weather           (SERVER)
|    |    |    |
|    |    |    |--- policy:rate-limit-inbound
|    |    |    |
|    |    |    |--- handler:urlForwardHandler
|    |    |    |    |
|    |    |    |    |--- GET https://api.example.com/weather

On the MCP Gateway, the CLIENT span holds the call to the upstream MCP server, including a retry after the upstream answers 401:

Code
|--- POST /mcp/linear                           (entry span)
|    |
|    |--- policies:inbound
|    |    |
|    |    |--- policy:auth0-managed-oauth
|    |    |
|    |    |--- policy:mcp-token-exchange-linear
|    |
|    |--- handler:McpProxyHandler
|    |    |
|    |    |--- tools/call                       (SERVER, gen_ai.tool.name=create_issue)
|    |    |    |
|    |    |    |--- tools/call                  (CLIENT, server.address=mcp.linear.app)
|    |    |    |    |
|    |    |    |    |--- POST https://mcp.linear.app/mcp

The MCP Gateway also passes the trace on to the upstream MCP server. It adds the CLIENT span's traceparent and tracestate to the forwarded message's params._meta, alongside the standard HTTP traceparent header, so an upstream server that reads MCP trace context continues the same trace. The gateway adds them only to sampled requests, and forwards the original bytes unchanged when the body is larger than 256 KB or carries a body digest header such as Content-Digest.

Trace Context From MCP Clients

MCP clients can send trace context inside the request's params._meta (traceparent and tracestate), separate from the HTTP headers. Zuplo uses it as follows:

• When params._meta.traceparent belongs to the same trace as the HTTP request, the MCP span becomes a child of the client's span, and the span it would otherwise have had as a parent is attached as a link.
• When it belongs to a different trace, the MCP span stays in the HTTP request's trace and links to the client's span. The link has the attribute mcp.context.source set to params._meta.

The HTTP request's sampling decision always applies: trace context in params._meta can't turn sampling on or off. Baggage in params._meta isn't propagated to the requests that tool routes make.

Capture Tool Call Content

By default, MCP spans don't record what a tool received or returned. To record tool call content, set telemetry.captureToolCallContent to true in the handler options of an MCP Server route or an MCP Gateway route.

On an MCP Server route:

config/routes.oas.json
{
  "handler": {
    "export": "mcpServerHandler",
    "module": "$import(@zuplo/runtime)",
    "options": {
      "name": "example-mcp-server",
      "version": "1.0.0",
      "telemetry": {
        "captureToolCallContent": true
      }
    }
  }
}

On an MCP Gateway route:

config/routes.oas.json
{
  "handler": {
    "export": "McpProxyHandler",
    "module": "$import(@zuplo/runtime/mcp-gateway)",
    "options": {
      "rewritePattern": "https://mcp.example.com/mcp",
      "telemetry": {
        "captureToolCallContent": true
      }
    }
  }
}

With this option on, each sampled tools/call span records:

• gen_ai.tool.call.arguments: the arguments the client sent. On an MCP Server route, they're recorded once they pass validation. A call without arguments records {}.
• gen_ai.tool.call.result: the tool result, for successful calls only. A result with isError: true isn't recorded.

Both are JSON strings capped at 8,192 characters.

On the MCP Gateway, both the SERVER and the CLIENT span of a forwarded tools/call carry the attributes, and the result is the one the upstream MCP server returned. A response the gateway builds itself, such as a request to connect the upstream, is never recorded as a tool result. With the option on, the gateway reads tools/call responses up to 1 MiB, instead of the usual 64 KiB, so it can record larger results. A result in a JSON response, or in the event-stream event that carries it, larger than 1 MiB still reaches the client but isn't recorded. The gateway turns capture on only for the boolean true. Any other value, including the string "true", leaves it off.

Query MCP Spans in Your Backend

Because the attributes follow the semantic conventions, you can query MCP activity directly in any backend that receives your traces. For example:

• Find slow tool calls: filter on gen_ai.operation.name = execute_tool and sort by duration, grouped by gen_ai.tool.name.
• Find failing tools: filter on error.type = tool_error.
• Compare protocol versions in use: group by mcp.protocol.version.
• Follow one client session: filter on mcp.session.id.
• Compare gateway overhead with upstream latency: compare a SERVER span's duration with its CLIENT child's duration.

Performance and Sampling

Tracing isn't free. The plugin instruments every request: it opens spans for the request, each policy, the handler, and any fetch subrequests, and holds them in memory until the request finishes. Spans are exported after the response is sent, so tracing adds little to any single response time, but the CPU and memory it consumes per request reduce the headroom available for the next one. On high-throughput APIs, that shows up as lower peak throughput and higher tail latency.

Sample traces on any API that runs at a high request rate — high-RPS production traffic, latency-sensitive service-to-service calls, or an API you're about to load test. A ratio of 0.1, which keeps 10% of traces, is the standard starting point: enough traffic to see how the API behaves, without paying the export cost on every request. Tracing every request is a reasonable default during development and for APIs at moderate request rates.

Sample a Percentage of Traces

Set sampling.headSampler.ratio to the fraction of traces to keep. The ratio applies to every destination, including Zuplo's built-in storage. When you created your project, Zuplo set it up this way: every request traced in your working copy, 10% everywhere else.

modules/zuplo.runtime.ts
import { OpenTelemetryPlugin } from "@zuplo/otel";
import { environment, RuntimeExtensions } from "@zuplo/runtime";

export function runtimeInit(runtime: RuntimeExtensions) {
  const isWorkingCopy = environment.ZUPLO_ENVIRONMENT_STAGE === "working-copy";

  runtime.addPlugin(
    new OpenTelemetryPlugin({
      sampling: {
        headSampler: { ratio: isWorkingCopy ? 1 : 0.1 },
      },
    }),
  );
}

Set ratio to a constant to use one value everywhere. Zuplo populates ZUPLO_ENVIRONMENT_STAGE, which reads working-copy in your working copy and production, preview, or local elsewhere, so local development (zuplo dev) samples at 10% along with deployed environments. See system environment variables for the full list.

By default, traces whose root span ends in an error are exported whatever the ratio, so errors stay debuggable on a heavily sampled API. Supplying your own sampling.tailSampler replaces that behavior.

Sampling cuts the volume of trace data exported and stored, which lowers export overhead and backend cost. The per-request instrumentation still runs on every request, so sampling reduces the cost of tracing rather than removing it.

Disable Tracing

Sampling leaves some per-request work in place. Where you need none of it — measuring the gateway's raw capacity, or squeezing the last of the headroom out of a high-RPS environment — turn tracing off instead.

To remove the overhead entirely, don't register the plugin. Gate it on an environment variable so you can control tracing per environment without changing code:

zuplo.runtime.ts
import { OpenTelemetryPlugin } from "@zuplo/otel";
import { RuntimeExtensions, environment } from "@zuplo/runtime";

export function runtimeInit(runtime: RuntimeExtensions) {
  if (environment.TRACING_ENABLED === "true") {
    runtime.addPlugin(new OpenTelemetryPlugin());
  }
}

Set TRACING_ENABLED to true on the environments you want traced and leave it unset everywhere else. Tracing is off unless you opt in, so an environment built for maximum throughput — or for a load test — carries no tracing overhead until you ask for it.

Where the plugin doesn't run, traces stop appearing in the Observability tab for that environment. Request analytics and logs come from a separate pipeline, so they aren't affected.

Account for Logging Too

Logging costs compute and memory per request the same way tracing does. On a high-throughput API, log the events you'd act on rather than everything that happens, and keep custom log properties small. Sampling traces while logging verbosely on every request moves the cost rather than removing it. See Performance Under High Load.

Logging

The plugin can also export logs in OpenTelemetry format. Logs are sent to your own endpoint configured with the logUrl property (see Logs and Traces Together); Zuplo's built-in storage covers traces today, with managed logs and metrics planned for future releases.

To emit OpenTelemetry logs from your handlers and policies, use the context.log object:

Code
import { ZuploContext, ZuploRequest } from "@zuplo/runtime";

export default async function policy(
  request: ZuploRequest,
  context: ZuploContext,
) {
  context.log.info("Hello World");
  return request;
}

You can also set additional custom log properties using context.log.setLogProperties!:

Code
import { ZuploContext, ZuploRequest } from "@zuplo/runtime";

export default async function policy(
  request: ZuploRequest,
  context: ZuploContext,
) {
  context.log.setLogProperties!({ customProperty: "value" });
  context.log.info("Hello World");
  return request;
}

After setting a custom property, all subsequent log messages will include that property. These logs are exported to the configured log endpoint in OpenTelemetry format: the log message is in the message field and the custom properties are in the attributes field.

Custom Tracing

You can add custom tracing to your Zuplo API using the OpenTelemetry API. The example below shows how to implement tracing in a custom policy.

Code
import { ZuploContext, ZuploRequest } from "@zuplo/runtime";
import { trace } from "@opentelemetry/api";

export default async function policy(
  request: ZuploRequest,
  context: ZuploContext,
) {
  const tracer = trace.getTracer("my-tracer");
  return tracer.startActiveSpan("my-span", async (span) => {
    span.setAttribute("key", "value");

    try {
      const results = await Promise.all([
        fetch("https://api.example.com/hello"),
        fetch("https://api.example.com/world"),
      ]);
      // ...

      return request;
    } finally {
      span.end();
    }
  });
}

This will result in the following spans:

Code
|--- my-policy
|    |
|    |--- my-span
|    |    |
|    |    |--- GET https://api.example.com/hello
|    |    |
|    |    |--- GET https://api.example.com/world