The KloudMate AI Assistant answers questions about your systems and investigates your alerts. Version 2 is a ground-up rebuild. We kept one rule through all of it: every change gets measured against real incidents.
An AI assistant is easy to demo and hard to measure. A good answer on Monday says very little about Tuesday's incident. So before we touched the assistant, the team built a benchmark that only moves when the assistant actually gets better.
What changed, at a glance
- 2.7 times more root causes found than earlier, on the same incidents.
- 4 times faster chat. Answers that took 98 seconds now take 24.
- 82% of investigations name the root cause, and fewer than 2% are confidently wrong.
- 88% of expected facts met on live failures, with the exact root cause named in more than nine runs out of ten.
- 75% lower price. A chat answer is now $0.05 and a full investigation is $0.25.
The benchmark comes first
We record each incident at one instant, so every version of the assistant sees exactly the same telemetry. The incidents come from three places: our own production systems, faults we injected into a Kubernetes test cluster, and failures we injected into the OpenTelemetry Astronomy Shop, a live online shop with more than 15 services. We also run live tests to keep the recordings honest.
Every scenario lists the facts a good answer has to state, like the failed component, the start time and a fix. It also lists what the answer must not claim, like a decoy cause or an old event presented as current. An independent model grades each answer, and we read the graded answers ourselves at every milestone.
- Recordthe incident at one instant
- Replayit for every version
- Gradeagainst the fact list
- Comparewith 95% confidence intervals
- Shiponly if nothing got worse
Each change runs side by side with the previous version, two or three times per scenario. It only ships if the 95% confidence interval shows nothing got worse. So far that's 19,565 runs, and 13,153 of them have been graded by the judge.
Four rules behind every investigation
- 1
Accuracy
We set the targets before we started: at least 80% of expected facts as the floor, and 90% as the goal. A change ships only if accuracy holds or improves.
80% floor90% goal - 2
Evidence before conclusions
Like a good SRE, the assistant forms hypotheses and tests each one against traces, logs, metrics and Kubernetes state. Then a separate review checks the conclusion. That review can lower the confidence, but never raise it. If the evidence isn't there, the report says inconclusive.
- 3
General tools for every stack
The assistant can query any field in your telemetry, the way an engineer would, so it isn't tied to Kubernetes. In our tool-selection test it picks the right tool and fields about nine times out of ten.
- 4
Reports that show their work
Each one has the causal path, a timeline, the causes it ruled out, and the evidence and query behind every claim. Every fix comes with a step to check that it worked.
'Confidently wrong' is the number we watch
An assistant that's sometimes unsure is still useful. One that's confidently wrong sends you down the wrong path. So we track that on its own: how often it names a cause with confidence when the real cause is something else.
We ran the new investigation engine against the old one on the same investigations, three runs each, both on our production model. Right outcomes went from 29% to 90%. The share of runs that found the cause went from 33% to 88%. And confidently wrong answers went from 10% to zero: the new engine named a cause in every run, and none of them was confidently wrong. Across the full benchmark, fewer than 2% of investigations are confidently wrong.
That zero comes from one matched comparison. It doesn't mean the assistant can never be confidently wrong.
Live failures that cascade
A single fault with a clear alert is the easy case. Real incidents cascade. A database stops, and the alert fires from a service three hops away at the edge. So we injected failures into the live Astronomy Shop: services scaled to zero, crash loops, memory limits, bad feature flags, DNS faults, and a database with no free connections. Each investigation started from a different place - the failing component, a downstream service, the edge, or an engineer's question with no alert at all.
- 1The failing component
- 2A downstream service
- 3The application edge
- 4An engineer's question, no alert
Scaled to zeroCrash loopsMemory limitsBad feature flagsDNS faultsNo free DB connections
"Exact root cause" is a strict bar. "There's a problem with the cart" fails. "The cart's cache has no pods because its deployment was scaled to zero" passes. On that bar the assistant got 88% of the expected facts, and named the exact root cause in 92% of runs.
There's a problem with the cart.
The cart's cache has no pods because its deployment was scaled to zero.
The 82% is a different measurement. It's the share of runs that named the root cause on the wider investigation benchmark, not the exact root cause on the live failures. Different tests, different numbers.
The decoy test
Our favourite test plants a red herring. We pushed the ad service's CPU to 3.7 cores. Ten minutes later, a feature flag made some product lookups in product-catalog fail, and the error-rate alert fired at the edge. An assistant that chases the biggest recent anomaly blames the ad service.
- 02:58Ad service CPU to 3.7 coresThe decoy
- 03:08Feature flag: GetProduct calls failThe real start
- 03:11Error-rate alert fires at the edgeThe symptom
product-catalog→frontend→frontend-proxyThe KloudMate AI Assistant followed the failed requests from the edge back to the source. It found the flag, put the start at 03:08, and ignored the ad service. Ten facts out of ten.
Why it costs so little
Most of an investigation is fast telemetry queries, not model calls. On average it runs 291 queries and 15 model calls, and takes about four minutes. The model plans the queries and reads the results. Model calls are the expensive part of any AI answer, so using only a handful of them is what makes the new price possible.
A chat answer averages 39 queries and six model calls. It takes about 25 seconds, and the first words show up in two to three seconds. Chat answers meet 85% of the expected facts. The four-times-faster number is a separate comparison with the old version, on scenarios both ran: 98 seconds down to 24.
Where we're still improving
We'd rather tell you the gaps ourselves. Slow resource exhaustion is the hardest pattern we test. A database that gradually uses up its connections looks like ordinary slowness until new connections get refused. Managed cloud services are another one. If the failure is inside a managed service, we have no internal traces to follow, so those score lower.
Slow resource exhaustion, like a database using up its connections.
Failures inside managed services, where we have no internal traces.
And it's our own benchmark. We test on systems we know well, and a model does the grading. We review the grades ourselves and keep adding scenarios from real incidents. Next, we're going after larger incidents, where one fault ripples through many services before anyone notices.
The new price
An AI credit now costs $0.05, down from $0.20. A chat answer still uses one credit and an investigation still uses 5. So nothing changes in how you use it. Each credit just costs a quarter of what it did. We're also raising the included monthly credits fourfold. A $10 credit pack now buys 200 credits instead of 50.
At $1 for an investigation, you save root cause analysis for the big incidents. At 25 cents, you can run it on every alert.
- A team asking 20 questions and running 2 investigations every working day pays about $33 a month. It used to be $132.
- Automatically investigating 50 alert groups a day costs about $375 a month. It used to be $1,500.
Try it on your next alert.