Peak Day Was Three Weeks Away. Capacity Fell Short.

A retail peak-readiness story about finding the failure before customers do.

The sale was three weeks away when the forecast arrived.

The retail team expected 11,440 concurrent calls at the busiest point of the event. Operations added a 20% safety margin. That moved the requirement to 13,728.

Someone in the review asked the obvious question.

“How many can we support today?”

“About 9,600.”

The room went quiet for a moment.

Production was stable. There was no incident ticket, no queue backing up, and no angry customer waiting in support. Yet the arithmetic had already found a problem.

Customer details have been anonymized.

The Question Wasn’t “Is Production Healthy?”

The first story in this series followed an incident where healthy infrastructure hid a broken business workflow. This time, nothing had broken. The team was trying to answer a harder question: would the platform remain healthy under traffic it had not seen before?

That answer lived in several places. The commercial forecast held the expected demand. Kubernetes defined how far the application could scale. A provider account imposed its own concurrency quota. Autoscaling settings determined when new capacity would arrive. Each answer looked reasonable on its own.

Together, they did not work.

The application was capped at 12 replicas when the forecast called for 18. The provider quota stopped at 10,000 concurrent calls, short of the 13,728 requirement. No pre-scaling configuration had been prepared. CPU autoscaling would wait until 80% utilization, which left little room for a sharp traffic rise.

None of those settings caused trouble on an ordinary day. Peak day would make all four matter at once.

Watch the Readiness Check

This short video walks through the assessment and shows how an OpsRabbit agent checks the forecast against the controls that determine real capacity.

The useful part is the evidence trail. The agent can show which requirement it tested, where it found the current setting, and why a control passed or failed. An engineer can challenge the result instead of accepting a vague readiness score.

A Finding Still Needs an Owner

The assessment did not make the platform ready by itself. Someone still had to confirm the forecast, raise the provider quota, change the replica ceiling, decide whether to pre-scale, and test the revised settings before the event.

That is where AAIC’s managed operations team comes in.

OpsRabbit gathers the operational evidence and repeats the checks. AAIC’s DevOps and SRE engineers review the findings, work through the changes with the customer, and keep track of what remains unresolved. If a dependency belongs to another team or vendor, the finding has an owner and a due date instead of disappearing into a meeting note.

This also changes the next readiness review. The team does not have to rebuild the checklist from memory. The same controls can be checked again after a release, a forecast change, or a provider update. New checks can be added when the architecture changes.

The Better Meeting Happens Before the Event

A useful peak-readiness meeting should be pleasantly dull. The forecast is clear. The capacity assumptions have evidence behind them. Failed checks have owners. Unknowns are visible while there is still time to investigate them.

The alternative is familiar: a busy incident bridge, five dashboards, and somebody asking why a quota that had been in place for months was never tested against the sale forecast.

Peak readiness is one part of AAIC Managed AI Operations. The same operating model supports incident investigation, release readiness, CloudOps, ITOps, SRE, DevOps, and FinOps work. OpsRabbit assists with the investigation and recurring checks. AAIC engineers own the operational response.

For the other side of this story, read The Dashboards Were Green. The Business Wasn’t., which follows an investigation after a retail workflow had already failed.