Run a Load Test Like a Production Launch

A large load test looks like an attack to every automated defense in its path. It can also burn thousands of dollars in generator time and egress, produce millions of billing events, and affect shared infrastructure.

We load test Zuplo constantly, and the technical part is rarely what makes a large run painful. The expensive failures are operational: a provider was not told, the first run used the whole budget, or the environment disappeared before anyone finished reading the evidence.

Treat the test like a production launch. Book the window, name the decision makers, write down the limits, and keep the environment available until the review is done.

Notify every provider in the request path

WAFs, bot systems, abuse controls, and DDoS protection are designed to block the traffic pattern a load test creates: a rapid ramp, high request rate, and a small set of source IPs.

Review the testing policy for every provider in the path. Akamai’s published guidance requires advance coordination for certain tests. AWS distinguishes load testing from simulated DDoS events and applies different rules to each.

Contact providers about a week before the planned run. Put these details in one thread:

• Date, start time, end time, and timezone
• Source IP ranges for every generator
• Target hostnames and environments
• Maximum RPS and ramp shape
• Expected response volume and payload size
• Monitoring contact and abort contact

Ask each provider to confirm allowlisting, monitoring, and any temporary capacity changes. If the window moves, update the confirmation instead of assuming it still applies.

Define success, failure, and abort conditions in writing

Write a short test contract before provisioning infrastructure. It should answer:

• What decision will this test support?
• What are the target and maximum RPS?
• What ramp, steady-state, soak, and recovery phases will run?
• Which environment is under test, and is it production-sized?
• What are the latency and error thresholds for each status class?
• Which business systems must remain correct?
• Who can stop the test, and how will they do it?

Separate a target from an authorized maximum. The target is the rate you need to validate. The maximum is the rate the test must never exceed without a new approval. Treating the maximum as a suggestion is how a useful run becomes an incident.

Do not load test a development environment and extrapolate the result to production. Preview environments also vary: some use production infrastructure, while others use smaller pools and stricter limits. Verify the environment’s capacity model with the provider.

Budget for the test, the retries, and the investigation

The budget should include more than generator runtime. We have had large distributed runs cost thousands of dollars before anyone opened a chart, so the invoice is part of our test design, not an unpleasant surprise afterward.

• Virtual-user or request-based load-testing charges
• Generator and target egress
• Temporary target capacity
• Observability ingestion and retention
• Staff coverage during the window
• At least one rerun after correcting the test itself
• Time to keep the environment available for investigation

Open workload models can require many more virtual users when the target slows down. Estimate VUs from the degraded latency you plan to observe, not the happy path. At 20,000 VUs, even a short window can become expensive. Use the provider’s current pricing calculator instead of copying an old list price into the plan. In our experience, the first run often finds a problem with the test, and the result worth sharing comes from the second or third.

Response size matters too. A small request body with a large response can make target egress the largest variable cost. Include request and response bytes in the estimate.

Do not destroy the test environment when the generator stops. Keep the generators, dashboards, logs, traces, and target infrastructure until everyone has completed analysis. A day of idle infrastructure is usually cheaper than rerunning a test because the evidence disappeared.

Validate metering and billing with a soak test

This is the part of the series I care most about. A capacity test can prove that the API serves traffic while the business systems behind it quietly produce bad data. Metering, billing, analytics, and log pipelines need volume over time, not only a short spike.

Run a steady soak before the spike when correctness depends on downstream events. The soak should produce enough records to expose duplication, loss, late delivery, null fields, and aggregation errors. Reconcile counts across the generator, gateway, event pipeline, and billing system before moving on.

k6 can schedule both profiles without overlapping them:

export const options = {
  discardResponseBodies: true,
  scenarios: {
    soak: {
      executor: "constant-arrival-rate",
      rate: 400,
      timeUnit: "1s",
      duration: "24h",
      preAllocatedVUs: 200,
      maxVUs: 400,
    },
    spike: {
      executor: "ramping-arrival-rate",
      startRate: 0,
      timeUnit: "1s",
      preAllocatedVUs: 1000,
      maxVUs: 3000,
      startTime: "24h10m",
      stages: [
        { target: 5000, duration: "1m" },
        { target: 5000, duration: "5m" },
        { target: 0, duration: "1m" },
      ],
    },
  },
};

At 400 RPS, a 24-hour soak produces about 34.6 million requests. After it finishes, verify:

• Generator requests against gateway and origin counts
• Metering events against accepted billable requests
• Duplicate and missing event rates
• Billing records and aggregation totals
• Analytics counts and dimensions
• Log and trace completeness under sustained volume

Mark synthetic traffic so it cannot create real invoices or contaminate business metrics. A dedicated header or tenant ID is easier to audit than a time-window exclusion.

Use synthetic traffic for questions production cannot answer

Production observability describes traffic you already have. A load test helps answer questions about traffic you do not have yet, such as a launch spike or a migration to a new gateway.

Synthetic traffic is still an approximation. Uniform payloads, a few cache keys, and generator-specific connection behavior can exercise the stack differently from real clients. Improve realism with:

• Shadow traffic that replays production-shaped requests against an isolated target, using tools such as ShadowReader
• Traffic shifting that gradually moves real requests to the new system
• Synthetic request markers that exclude test traffic from billing and product analytics
• Request keys and payload sizes sampled from production distributions

Use breakpoint tests when you need the system’s actual capacity boundary. Ramp until latency or errors cross the agreed threshold, then stop. The point where behavior changes is more useful for planning than a pass at one arbitrary rate.

For systems that change frequently, run smaller load tests continuously. Slack has described its approach to continuous load testing, which catches capacity regressions before launch week.

Convert the result into an operating plan

A load test should end with a decision, not a screenshot of a dashboard. A calendar entry to pre-provision capacity before a known event is a perfectly good outcome. You do not get extra points for turning every capacity problem into an architecture project.

Record:

• Sustainable RPS at the agreed latency and error thresholds
• The breakpoint and the first component that saturated
• Autoscaling reaction time and the capacity available before scaling completes
• Regional capacity and failover assumptions
• Required headroom above forecast peak traffic
• Manual capacity changes needed for scheduled events
• The next test date and the change that should trigger an earlier rerun

Do not count autoscaling as available capacity until you have observed it react to the tested ramp. If the traffic spike ends before new instances serve requests, pre-provisioning is part of the launch plan.

Regional results need the same scrutiny. A global test can accidentally send all traffic to one region. The article on multi-region generator placement explains how to test distribution separately from regional capacity.

API load-testing launch checklist

One week before the run:

1. Notify every provider and confirm the test window.
2. Share generator IPs, target hostnames, maximum RPS, and ramp shape.
3. Confirm the environment is production-sized.
4. Approve the budget for the first run, a rerun, and evidence retention.

Before traffic starts:

5. Review success, failure, and abort thresholds by status class.
6. Verify dashboards for generators, intermediaries, origin, and business pipelines.
7. Confirm synthetic traffic cannot create real billing records.
8. Test the abort procedure at low volume.

After the run:

9. Keep the infrastructure and telemetry available.
10. Reconcile request, metering, billing, analytics, and log counts.
11. Publish the capacity boundary, bottleneck, and operating changes.
12. Schedule the rerun or continuous test that keeps the result current.

The rest of this series covers the measurement details: workload models and percentiles, generator placement, testing the full intermediary path, and using tracing to explain the result.

If Zuplo is in the path, send us the window, target RPS, and generator IP ranges before the run. We can coordinate allowlisting and monitor the gateway side of the test with you.