Copy page

Observability

Zuplo OpenTelemetry

Zuplo ships with an OpenTelemetry plugin (@zuplo/otel) that instruments your API and exports traces and logs in OpenTelemetry format. The quickest way to use it is Zuplo's built-in tracing: add the plugin and your traces are stored by Zuplo and shown in the portal's Observability tab, with no collector to run or backend to host. You can also export to your own OpenTelemetry backend, or to both at once.

Tracing

Tracing helps you monitor performance, identify bottlenecks, and troubleshoot issues in your Zuplo API. The OpenTelemetry plugin automatically instruments your API, so you get timings for each request along with spans for policies, handlers, and subrequests. It supports trace propagation (W3C headers by default), so you can follow a request from the client through to your backend.

What's Traced?

By default, when the OpenTelemetry plugin is enabled, the following is traced:

• Request: The entire request lifecycle is traced, including the time taken to process the request and send the response.
• Inbound Policies: The time taken to execute all inbound policies as well as each policy is traced.
• Handler: The request handler is traced.
• Outbound Policies: The time taken to execute all outbound policies as well as each policy is traced.
• Subrequests: Any use of fetch within your custom policies or handlers is traced.

Limitations

One important limitation to keep in mind is that the clock will only increment when performing I/O operations (for example when calling fetch, using the Cache APIs, etc.). This is a limitation imposed as a security measure due to Zuplo's serverless, multi-tenant architecture. In practice this shouldn't impact your ability to trace, as virtually any code that isn't I/O bound is fast.

Setup

Add the OpenTelemetryPlugin in your zuplo.runtime.ts file. Where it sends data is up to you: Zuplo's built-in storage, your own backend, or both.

Send Traces to Zuplo

Add the plugin with no configuration. It sends traces to Zuplo and names the service after your project:

zuplo.runtime.ts
import { OpenTelemetryPlugin } from "@zuplo/otel";
import { RuntimeExtensions } from "@zuplo/runtime";

export function runtimeInit(runtime: RuntimeExtensions) {
  runtime.addPlugin(new OpenTelemetryPlugin());
}

Deploy your project and open the Observability tab to see traces.

Each trace is tagged with the account, project, deployment, and environment it ran in, plus the request ID (the zp-rid value) that also appears on your logs, so you can move between a request's logs and its trace. How long traces are kept depends on your plan.

Export to Your Own Backend

To send traces to an OpenTelemetry service such as Honeycomb, Middleware, Dynatrace, Jaeger, or others, configure an exporter with the service's url and headers. It's common for providers to use a header for authorization. Configuring an exporter sends traces there instead of Zuplo.

zuplo.runtime.ts
import { OpenTelemetryPlugin } from "@zuplo/otel";
import { RuntimeExtensions, environment } from "@zuplo/runtime";

export function runtimeInit(runtime: RuntimeExtensions) {
  runtime.addPlugin(
    new OpenTelemetryPlugin({
      exporter: {
        url: "https://otel-collector.example.com/v1/traces",
        headers: {
          "api-key": environment.OTEL_API_KEY,
        },
      },
      service: {
        name: "my-api",
        version: "1.0.0",
      },
    }),
  );
}

Sampling and Post-Processing

The plugin supports additional options for advanced use cases, including head sampling and post-processing of spans before export.

zuplo.runtime.ts
import { OpenTelemetryPlugin } from "@zuplo/otel";
import { RuntimeExtensions, environment } from "@zuplo/runtime";

export function runtimeInit(runtime: RuntimeExtensions) {
  runtime.addPlugin(
    new OpenTelemetryPlugin({
      exporter: {
        url: "https://otel-collector.example.com/v1/traces",
        headers: { "api-key": environment.OTEL_API_KEY },
      },
      service: {
        name: "my-api",
        version: "1.0.0",
      },
      // Optional post processor to modify spans before export
      postProcessor: (spans) => {
        for (const span of spans) {
          (
            span as unknown as { attributes: Record<string, unknown> }
          ).attributes["post.processed"] = true;
        }
        return spans;
      },
      sampling: {
        headSampler: {
          ratio: 0.1, // Sample 10% of requests
        },
      },
    }),
  );
}

Logs and Traces Together

To export logs as well as traces, use the top-level traceUrl, logUrl, and headers properties instead of the exporter object. The plugin supports both shapes, but they're mutually exclusive: exporter configures tracing only, while traceUrl and logUrl configure tracing and logging together with a shared set of headers. See Logging for how to emit log records.

zuplo.runtime.ts
import { OpenTelemetryPlugin } from "@zuplo/otel";
import { RuntimeExtensions, environment } from "@zuplo/runtime";

export function runtimeInit(runtime: RuntimeExtensions) {
  runtime.addPlugin(
    new OpenTelemetryPlugin({
      logUrl: "https://otel-collector.example.com/v1/logs",
      traceUrl: "https://otel-collector.example.com/v1/traces",
      headers: { "api-key": environment.OTEL_API_KEY },
      service: {
        name: "my-api",
        version: "1.0.0",
      },
      sampling: {
        headSampler: {
          ratio: 0.1, // Sample 10% of requests
        },
      },
    }),
  );
}

Send Traces to Zuplo and Your Own Backend

You aren't limited to one destination. To deliver traces to Zuplo and your own backend at the same time, add a span processor for each. Zuplo is represented by a ZuploSpanExporter. The processors batch and flush independently, so a slow or failing backend doesn't hold up the other:

zuplo.runtime.ts
import {
  OpenTelemetryPlugin,
  OTLPSpanExporter,
  BatchTraceSpanProcessor,
  ZuploSpanExporter,
} from "@zuplo/otel";
import { RuntimeExtensions, environment } from "@zuplo/runtime";

export function runtimeInit(runtime: RuntimeExtensions) {
  runtime.addPlugin(
    new OpenTelemetryPlugin({
      service: { name: "my-api" },
      spanProcessors: [
        new BatchTraceSpanProcessor(
          new OTLPSpanExporter({
            url: "https://otel-collector.example.com/v1/traces",
            headers: { "api-key": environment.OTEL_API_KEY },
          }),
        ),
        new BatchTraceSpanProcessor(new ZuploSpanExporter()),
      ],
    }),
  );
}

Logging

The plugin can also export logs in OpenTelemetry format. Logs are sent to your own endpoint configured with the logUrl property (see Logs and Traces Together); Zuplo's built-in storage covers traces today, with managed logs and metrics planned for future releases.

To emit OpenTelemetry logs from your handlers and policies, use the context.log object:

Code
import { ZuploContext, ZuploRequest } from "@zuplo/runtime";

export default async function policy(
  request: ZuploRequest,
  context: ZuploContext,
) {
  context.log.info("Hello World");
  return request;
}

You can also set additional custom log properties using context.log.setLogProperties!:

Code
import { ZuploContext, ZuploRequest } from "@zuplo/runtime";

export default async function policy(
  request: ZuploRequest,
  context: ZuploContext,
) {
  context.log.setLogProperties!({ customProperty: "value" });
  context.log.info("Hello World");
  return request;
}

After setting a custom property, all subsequent log messages will include that property. These logs are exported to the configured log endpoint in OpenTelemetry format: the log message is in the message field and the custom properties are in the attributes field.

Custom Tracing

You can add custom tracing to your Zuplo API using the OpenTelemetry API. The example below shows how to implement tracing in a custom policy.

Code
import { ZuploContext, ZuploRequest } from "@zuplo/runtime";
import { trace } from "@opentelemetry/api";

export default async function policy(
  request: ZuploRequest,
  context: ZuploContext,
) {
  const tracer = trace.getTracer("my-tracer");
  return tracer.startActiveSpan("my-span", async (span) => {
    span.setAttribute("key", "value");

    try {
      const results = await Promise.all([
        fetch("https://api.example.com/hello"),
        fetch("https://api.example.com/world"),
      ]);
      // ...

      return request;
    } finally {
      span.end();
    }
  });
}

This will result in the following spans:

Code
|--- my-policy
|    |
|    |--- my-span
|    |    |
|    |    |--- GET https://api.example.com/hello
|    |    |
|    |    |--- GET https://api.example.com/world