DevOps | Applied AI Consulting

Peak Day Was Three Weeks Away. Capacity Fell Short.

Fri, 11 Sep 2026 11:00:00 -0500

A retail peak-readiness story about finding the failure before customers do.

The sale was three weeks away when the forecast arrived.

The retail team expected 11,440 concurrent calls at the busiest point of the event. Operations added a 20% safety margin. That moved the requirement to 13,728.

Someone in the review asked the obvious question.

“How many can we support today?”

“About 9,600.”

The room went quiet for a moment.

Production was stable. There was no incident ticket, no queue backing up, and no angry customer waiting in support. Yet the arithmetic had already found a problem.

Customer details have been anonymized.

The Question Wasn’t “Is Production Healthy?”

The first story in this series followed an incident where healthy infrastructure hid a broken business workflow. This time, nothing had broken. The team was trying to answer a harder question: would the platform remain healthy under traffic it had not seen before?

That answer lived in several places. The commercial forecast held the expected demand. Kubernetes defined how far the application could scale. A provider account imposed its own concurrency quota. Autoscaling settings determined when new capacity would arrive. Each answer looked reasonable on its own.

Together, they did not work.

The forecast and the platform told different stories

Forecast 11,440 concurrent calls

Safety margin 20% above forecast

Required 13,728 concurrent calls

Supported 9,600 current capacity

4,128-call capacity gapvisible before peak traffic arrived

The application was capped at 12 replicas when the forecast called for 18. The provider quota stopped at 10,000 concurrent calls, short of the 13,728 requirement. No pre-scaling configuration had been prepared. CPU autoscaling would wait until 80% utilization, which left little room for a sharp traffic rise.

None of those settings caused trouble on an ordinary day. Peak day would make all four matter at once.

Watch the Readiness Check

This short video walks through the assessment and shows how an OpsRabbit agent checks the forecast against the controls that determine real capacity.

The useful part is the evidence trail. The agent can show which requirement it tested, where it found the current setting, and why a control passed or failed. An engineer can challenge the result instead of accepting a vague readiness score.

A Finding Still Needs an Owner

The assessment did not make the platform ready by itself. Someone still had to confirm the forecast, raise the provider quota, change the replica ceiling, decide whether to pre-scale, and test the revised settings before the event.

That is where AAIC’s managed operations team comes in.

OpsRabbit gathers the operational evidence and repeats the checks. AAIC’s DevOps and SRE engineers review the findings, work through the changes with the customer, and keep track of what remains unresolved. If a dependency belongs to another team or vendor, the finding has an owner and a due date instead of disappearing into a meeting note.

How the readiness work is divided

OpsRabbit

Checks the evidence

Compares forecast demand with quotas, replica limits, scaling controls, and the latest configuration.

AAIC engineers

Own the readiness work

Validate findings, coordinate changes, retest controls, and keep unresolved risks visible to the customer.

This also changes the next readiness review. The team does not have to rebuild the checklist from memory. The same controls can be checked again after a release, a forecast change, or a provider update. New checks can be added when the architecture changes.

The Better Meeting Happens Before the Event

A useful peak-readiness meeting should be pleasantly dull. The forecast is clear. The capacity assumptions have evidence behind them. Failed checks have owners. Unknowns are visible while there is still time to investigate them.

The alternative is familiar: a busy incident bridge, five dashboards, and somebody asking why a quota that had been in place for months was never tested against the sale forecast.

Peak readiness is one part of AAIC Managed AI Operations. The same operating model supports incident investigation, release readiness, CloudOps, ITOps, SRE, DevOps, and FinOps work. OpsRabbit assists with the investigation and recurring checks. AAIC engineers own the operational response.

For the other side of this story, read The Dashboards Were Green. The Business Wasn’t., which follows an investigation after a retail workflow had already failed.

The Dashboards Were Green. The Business Wasn't.

Fri, 11 Sep 2026 10:30:00 -0500

A Managed AI Operations story about the gap between seeing an alert and understanding what actually happened.

The alert arrived at a bad time. A retail team was preparing for peak traffic, and the usual checks had been done. Kubernetes looked healthy, the application was responding, and Argo CD showed the deployment as synchronized.

Nothing on the infrastructure dashboards suggested a serious problem. But the loyalty settlement extract had stopped, a reporting team could no longer query an authorized view, and a deployment check was blocked.

The incident bridge opened.

“Anything in Kubernetes?”

“Looks normal.”

“Did something change in the deployment?”

“Argo is clean.”

There was no shortage of data. The problem was finding the right evidence across several systems while the clock was running.

The dialogue is reconstructed from the incident sequence, and customer details have been anonymized.

The Incident Behind the Incident

Anyone who has spent time around production operations knows what tends to happen next. One engineer checks application logs. Someone else opens the cloud console. The database team gets pulled in. Another person searches an old ticket because the symptoms look familiar.

Each person has a piece of the picture. Connecting those pieces takes time.

When the evidence is everywhere

Alert

MonitoringLogsKubernetesDeployments IAM and accessDatabaseTicketsRunbooks

Manual correlationAcross tools and teams

Working hypothesis

Most operations teams already have the evidence. The expensive part is finding the right pieces and connecting them quickly enough to act.

AAIC’s managed operations engineers continued using the customer’s existing monitoring, cloud, ticketing, and deployment systems. OpsRabbit worked across that environment as an AI-assisted investigation and operational intelligence layer.

The first useful finding was not what had failed. It was what had not. The cluster was healthy, the service was reachable, and the deployment looked fine. The team could stop spending time there.

Then the investigation surfaced two changes. A reporting service account had lost a required BigQuery permission. An Oracle credential had also been rotated, and a dependent workflow was failing authentication.

Two changes in different places had landed close enough together to look like one messy outage. Once the evidence was connected, the investigation became specific.

Now the Team Had Something to Test

AAIC’s engineers reviewed the findings and narrowed the response to a small set of checks: restore the required permission, correct the credential issue, rerun the application checks, test access to the reporting view, verify the settlement workflow, and confirm that operational signals returned to normal.

OpsRabbit did not decide to change production. The engineers did. AI helped the team move from “something is broken” to evidence-backed hypotheses. AAIC remained responsible for validating the evidence, deciding what to change, communicating with the customer, and confirming that the business workflow was restored.

That is how AAIC uses AI in managed operations.

What Changed?

The useful breakthrough came when events that looked unrelated could be examined as part of the same investigation.

A real investigation, simplified

Operational signals

What teams could see

Datadog incident alert
Prometheus and Grafana healthy
Reporting query failed

Correlated evidence

What OpsRabbit surfaced

Platform and deployment healthy
BigQuery permission removed
Oracle credential rotation failed

Engineer-owned response

What AAIC validated

Restore required access
Correct and verify credential
Rerun workflow checks

Engagement result: 75% faster RCA during the outageFewer escalation loops and a repeatable investigation record

An anonymized example of operational evidence brought together during an investigation. Results are engagement-specific.

In this documented engagement, the OpsRabbit-assisted workflow produced RCA 75% faster during the outage. That result is specific to this incident, not a universal performance guarantee. The more durable value was a repeatable way to gather evidence, rule out healthy components, record missing signals, and give engineers a concrete validation plan.

The More Interesting Question Came After the Incident

Once the immediate problem was resolved, the team could ask a broader question: what happens to the operating model when every operations engineer has this kind of investigative support?

A DevOps engineer may be tracing a pipeline failure. An SRE may be responding to a Kubernetes memory alert. CloudOps may be investigating an infrastructure change, while ITOps handles a recurring application issue and FinOps examines an unexpected shift in spend.

The symptoms differ, but the work often follows the same pattern: find the relevant evidence, understand what changed and what depends on it, compare what happened with what should have happened, and decide what to do next.

AAIC Managed AI Operations provides the accountable engineers and managed service. OpsRabbit gives those engineers an additional investigation layer. Existing systems such as Datadog, Grafana, CloudWatch, ServiceNow, Jira, Kubernetes, and CI/CD platforms remain the operational sources of truth.

AAIC Managed AI Operations model

Your operations environment

ObservabilityCloud and infrastructureCI/CDITSM and ticketsLogs and metricsRunbooksCloud cost data

OpsRabbitInvestigation and Operational IntelligenceConnect evidence, investigate, correlate context, build hypotheses, recommend checks

AAIC Managed Operations

ITOpsDevOpsCloudOpsSREFinOps

InvestigateDecideResolveLearnImprove

OpsRabbit supports the investigation. AAIC engineers remain accountable for the operational work and customer outcomes.

AI Operations Is Not Another Monitoring Category

Monitoring, observability, ITSM, cloud management, and CI/CD remain essential. AI becomes useful when it helps engineers make better use of the evidence those systems already contain.

For AAIC, that applies across ITOps application support, DevOps pipeline and release workflows, CloudOps infrastructure changes, SRE incident and reliability work, and FinOps cost investigations. DevOps engineering and cloud engineering remain part of the delivery foundation.

The technology matters when it shortens the distance between an operational signal and an informed engineering decision. That is the role OpsRabbit plays inside AAIC Managed AI Operations. AAIC owns the customer outcome.

This incident raised a second question: could the same operating model find a capacity problem before a major retail event? Continue with Peak Day Was Three Weeks Away. Capacity Fell Short.