Run a Load Test Like a Production Launch

Somebody on our free tier once decided to find out what our gateway could do. They spiked to roughly 300,000 RPS and pushed more than 15 million requests through in a short window, without telling anyone. Our abuse systems suspended the account outright rather than throttling it. By the time a human looked at it and worked out that this was a legitimate test by a legitimate user, their test window was long gone and they had learned nothing about our gateway except that we take 300,000 RPS from an unknown free-tier account seriously.

To be clear, that is the right kind of work. Presumably they wanted to know whether the platform would hold before betting on it, which is exactly what you should want to know. The only mistake was treating a 300,000 RPS test as something you can just do on a Tuesday night, rather than as an operational event that other people need to know about in advance.

Because at real scale, that is what it is. A load test looks like an attack to nearly every automated system between your generators and your origin. It ramps in seconds, comes from a handful of IPs, and looks nothing like a human audience, which is pretty much the exact signature WAFs, bot rules, and abuse systems exist to catch. It is also a billing event: you pay for generator time and egress, and every provider in the path spends real capacity absorbing it.

The earlier posts in this series were about measurement. This one is the operational checklist - how to run a big load test the way you would run a production launch.

Tell every provider in the path

Provider load-testing policies are typically more specific than teams expect. One major CDN’s published policy wants a support ticket at least 24 hours ahead of every test, caps the ramp at zero-to-full in 15 minutes, and confines tests to an overnight window. AWS draws a line between network stress testing, allowed against prepared targets under conditions, and simulated DDoS, prohibited from EC2 and handled under separate approval. If you haven’t read every layer’s policy, your test probably violates at least one.

The suspension story at the top of this post is the normal outcome, not an unusually harsh one. Most abuse systems read an unannounced spike exactly the same way, and by the time a human reviews the case your window has passed.

The frustrating part is how cheap the alternative is. Vendors are almost always happy to join a test they know about. Tell us ahead of time and we can allowlist your generator IPs, watch the run live from our side, and pre-provision if you’re pointing at something unusual. None of that happens automatically at 2am, and all of it happens off the back of one email.

Agree success criteria and max RPS in writing

Before the first request, everyone involved (your team, your gateway vendor, anyone whose infrastructure the test crosses) should agree in writing on:

• The target max RPS and the ramp shape to reach it
• The test window, with dates and timezone
• The environment under test, confirmed production-sized
• Success criteria as thresholds by status class (i.e. 2xx vs 4xx vs 5xx), not one aggregate number
• Who can call an abort, and how

Load test arguments are almost always about what the test was supposed to show, and a one-paragraph agreement settles most of them in advance. Treat the quoted max as a floor rather than a ceiling - once a test is running and passing, someone will want twice the number, so provision for the overshoot and ask your providers to do the same.

That environment line deserves emphasis: never load test dev environments. They’re typically sized for functional testing (ours included), so their numbers under load say little about production. Preview environments vary by provider: some are sized like dev, others deploy to production infrastructure. Check which kind you have first.

Budget the test like an infrastructure project

We load test our own platform constantly, and the invoices are a design input: one large distributed run can cost thousands of dollars in generator time and egress before anyone reads a chart. Hosted platforms typically price by the virtual user hour (VUh), and a large test under an open workload model needs a lot of them: the tool adds VUs to hold the arrival rate when the target slows down. Grafana Cloud’s pricing page currently lists $0.15 per VUh as the starting paid rate for k6, so a two-hour window at 20,000 VUs is 40,000 VUh, about $6,000, though Grafana’s published volume tiers bring the per-hour rate down materially at that scale. The TLDR is that generator time is priced like compute, so check the current list price before committing to a test matrix.

Egress is a line item on both sides: the generators’ provider charges for request bytes going out, your side for response bytes coming back, easy to underestimate when responses are large and the run is long. Budget for more than one run: in our experience the first mostly finds problems with the test itself, and the report comes out of the second or third.

Don’t tear down the test infrastructure the moment the run ends. During one long testing campaign, runs kept hitting a suspected limit somewhere in the path and nobody could investigate: the clusters were already gone. Leave everything up until the forensics are done, and treat the extra day of instance cost as part of the budget.

Load test the business pipeline behind the API

This is the one I would most like people to take away from this series. Most load tests point at the data plane - can the API serve the traffic - and skip the business systems behind it entirely, which is backwards the moment requests turn into invoices.

On one monetized API we worked on, the team did it the other way around. Before any spike testing at all, they held their expected steady-state rate for a full 24 hours purely to validate metering and billing end to end. It is a boring run by design and it is not the run anyone wants to sit through. It found duplicate metering events and null fields in a small slice of records - data-quality bugs that every functional test had passed straight over, and that would have corrupted real invoices at production scale.

The alternative is that the API passes its spike test, everyone signs off, you launch, and six weeks later you’re reconciling a billing run by hand and explaining to customers why they were charged twice. Nobody puts that in a load test report because by then nobody connects it to the load test.

That ordering, soak first, spike second, tends to be right when billing correctness matters. The soak is modest by design: data-quality bugs are usually rate phenomena that show up only after millions of records, and there’s no point asking whether the API survives a sudden spike while the pipeline behind it is still producing corrupt records. In k6 both profiles are just the options block of the skeleton from the first post of this series; the imports and the export default function carry over unchanged:

export const options = {
  discardResponseBodies: true,
  scenarios: {
    // Soak: modest, sustained, boring on purpose. This is the run that
    // exercises metering, billing, and log pipelines at real volume.
    soak: {
      executor: "constant-arrival-rate",
      rate: 400, // RPS, set this to your expected steady-state traffic
      timeUnit: "1s",
      duration: "24h",
      preAllocatedVUs: 200,
      maxVUs: 400,
    },
    // Spike: short and steep. This run checks survival under sudden load.
    spike: {
      executor: "ramping-arrival-rate",
      startRate: 0,
      timeUnit: "1s",
      preAllocatedVUs: 1000,
      maxVUs: 3000,
      startTime: "24h10m", // never overlap the soak window
      stages: [
        { target: 5000, duration: "1m" }, // jump to 5,000 RPS
        { target: 5000, duration: "5m" }, // hold
        { target: 0, duration: "1m" }, // drop
      ],
    },
  },
};

After the soak, audit metering events, billing records, analytics counts, and log volumes. A day at 400 RPS is roughly 35 million requests, and metering all of them cleanly validates something a spike test never touches. The spike asks the other question. Ten minutes after the soak ends it jumps to 5,000 RPS, holds for five minutes, and is done by 24h17m, which tests survival rather than volume over time.

Know what a load test can’t tell you

On Hacker News, someone always argues that synthetic tests are mostly waste and you should watch production instead. That’s about half right. Observability describes the traffic you already have; a load test asks about traffic you don’t have yet. Before a launch, a marketing event, or a migration, watching production tells you nothing about the load you’re worried about.

The right half is that synthetic traffic stays synthetic: uniform payloads, a handful of cache keys, and generator-shaped connection behavior make a test exercise every layer in the path differently than real clients do. There is a middle ground:

• Shadow traffic, i.e. replaying production request logs against the new system, with tools like ShadowReader.
• Traffic shifting, where a growing share of real traffic moves to the system under test.
• Request tainting, where synthetic requests carry a marker header so downstream systems exclude them from business metrics and billing.

Two more habits push the same way. Prefer breakpoint tests, which ramp until the system degrades, over pass/fail runs at a single number: the knee matters more than surviving one arbitrary rate. And treat load testing as a standing practice rather than a launch-week scramble: Slack’s engineering team has written about running simulated clients continuously against shared environments, which turns capacity questions from an annual event into a query.

Turn load test results into a capacity plan

One customer we work with has traffic driven entirely by scheduled events: a steady baseline most days, then peaks roughly a third above it whenever an event runs. They could not trust autoscaling to react inside the spike, so the fix ended up being operational rather than architectural. They schedule short-term capacity boosts days ahead of known events, sized directly from load-test results. No new architecture, no rewrite, just a number from a test and a calendar entry. Capacity planning off the back of a load test is mostly peak-versus-baseline arithmetic plus a calendar, and that is a perfectly good answer.

That autoscaler point generalizes: if you have never watched yours react to a real spike, you probably shouldn’t count on it during one. A load test measures the reaction time between load arriving and new capacity serving requests. If the spike completes before the scaler reacts, the scaler is not part of your answer. Regional capacity assumptions deserve the same suspicion. The generator-placement post covers a test where generators clustered near one metro sent every request to one region of a two-region deployment sized for its share of the total, not all of it. Our guide to scaling APIs for increased traffic covers the architectural options.

The playbook, in one page

If you take nothing else from this series, take this list and run it before your next big test:

1. Tell every provider in the path, about a week out. Get the window, the generator IP allowlist, and monitoring on their side confirmed in one thread.
2. Write down the success criteria, the max RPS, the ramp shape, the environment, and who can call an abort. Then provision above the quoted max, because somebody will ask for twice the number once it passes.
3. Never load test a dev environment. Check whether your preview environment is dev-sized or production-sized before you trust anything it tells you.
4. Budget it like an infrastructure project. Generator time and egress on both sides, and enough headroom for two or three runs, because the first one mostly finds problems with the test.
5. Soak before you spike, and audit metering, billing, analytics, and logs after the soak. This is the step that catches the expensive bugs.
6. Leave the infrastructure standing until everyone’s forensics are done.

None of that is technically difficult. All of it is the difference between a test that produces a capacity plan and a test that produces a suspended account, a surprise invoice, or a billing incident six weeks after launch.

That closes the path this series walked, from generator to origin: choosing a workload model and statistics that mean something, why the generators’ location decides what you measure, what a test through gateways, CDNs, and load balancers actually measures, and using distributed tracing to explain the results. This post is the wrapper around all of it, because even a perfectly designed measurement is wasted if the operation around it goes wrong.

So: if you’re planning a big test against a Zuplo gateway, tell us first. Send us the window, the target RPS, and your generator IP ranges. We would very much rather watch the run with you and help you read the results than have our abuse systems decide the question for both of us at 2am.