What actually falls over at peak — written a week after the peak

The web server is the one thing that usually holds

Peak season is over, the numbers are in, and this is the right week to write this down — in March nobody remembers what actually happened. The recurring surprise, year after year, is where the failure was not. It was not the web server. PHP workers and a reverse proxy are the easiest part of the stack to add more of, everybody knows this, and consequently everybody has already over-provisioned it. When a shop dies at peak, it dies behind the web server, at something that cannot be cloned by adding a box.

There are three usual suspects and they are the same three every year: the database, the ERP sync, and somebody else's rate limit. All three share a property that makes them nasty — they degrade rather than fail. The shop does not go down. It gets slow, then slower, then the checkout times out for one customer in twenty, and because it is not down nobody escalates for forty minutes. Availability monitoring says green throughout.

The database goes first, and usually because of one query

Load does not scale a database gracefully. It falls off a cliff. Everything is fine at eight times normal traffic and unusable at twelve, because a query that was doing an acceptable amount of work per request is now doing that work often enough to saturate the disk, and the connection pool fills while everyone waits. The query is nearly always something ordinary: a listing page with a filter combination nobody tested, a stock check that runs uncached, a search that falls back to a LIKE across the product table.

Two things reliably help and neither is bought on the day. First, know your slow queries before December — turn on the slow query log in the autumn, when there is time to read it, and fix the top three. Second, understand what your cache actually covers, because the pages that get hammered on a sale day are the ones the cache does not help with: the cart, the checkout, anything with a customer-specific price, anything with a live stock number. Your cache hit rate in November tells you very little about your cache hit rate during a promotion where everyone is logged in.

The ERP sync is a queue that has never seen a queue

This one causes the most damage per incident because it fails silently and the consequences are commercial rather than technical. Your order export was built for a normal Tuesday. It moves orders one at a time, it holds a transaction while it waits for an ERP that was never fast, and it has a retry that assumes the failure was transient. Feed it twenty times the orders and it does not crash — it falls behind. Two hours later the ERP has yesterday's stock, the shop is still selling from a number that is no longer real, and you are overselling.

Overselling on a sale day is expensive in a way that does not show up in any technical postmortem: cancellations, refunds, support hours, and customers who found out on their first order with you that your stock numbers are decoration. The mitigations are unglamorous. Decouple the export from the order — the customer's order must be written and confirmed whether or not the ERP is reachable. Give the queue a monitor that alerts on depth and age, not on errors, because a backlog is not an error. And know in advance which non-essential jobs you switch off when the queue grows: the newsletter export, the review reminder, the recommendation rebuild. Peak day is not the day to be recalculating anything.

The limit you cannot scale is someone else's

You can add servers. You cannot add allowance on your payment provider's API, your shipping label service, your address validator, your tax service or your marketplace connector. Each has a rate limit, and on a normal day you are nowhere near it, so nobody has ever read that page of the documentation. On a peak day you hit it, and the failure lands in the worst possible place — a customer who has decided to buy, has entered their card, and gets an error.

Go and read the limits now, while it is calm, and write down what your shop does when each one says no. Most integrations, left to their default, retry immediately and in parallel, which is the precise behaviour that turns a brief throttle into a sustained outage. Retry with a backoff, queue what can be queued — a shipping label does not have to be created in the request that takes the money — and make sure the customer-facing failure text says something a human can act on. Also: tell your providers you expect a spike. Some of them will raise a limit if you ask in October. None of them will at nine on the evening of the sale.

• Read every third-party rate limit and write down the behaviour when it is hit.
• Retry with backoff — the default immediate retry is what turns a throttle into an outage.
• Ask providers for headroom in October; nobody grants it during the sale.
• Queue what is not needed to take the money — labels, exports, mails.

Rehearse in October, or improvise in November

Nothing above is discovered by thinking. It is discovered by generating load against something that resembles production and watching what bends first — and by doing it in October, when a broken evening is free. A load test that only requests the homepage tells you the thing you already knew, that your cache works. The test worth running is a whole customer: list, filter, product page, add to cart, checkout, pay against the provider's sandbox. That is the path that touches the database, the sync and the third parties, and it is the only path anybody is paying you for.

The other half of the rehearsal is not technical. Who is watching, on what, at what hour. What the threshold is at which somebody switches off recommendations or turns the sync down to essential orders only. Who is allowed to decide to stop selling a product that has oversold, at eleven at night, without calling anyone. Write that down on one page in October. The teams that get through peak calmly are not the ones with heroic engineers on the night — they are the ones where the boring decisions were already made and the capacity was already bought.

Frequently asked questions