The case for AI in incident response is easy to appreciate. An agent that never sleeps can start the moment an alert fires and present a root cause before the on-call engineer has found a laptop. Experienced engineers respond with the right question, which is how do I know it's right? Our view is that an AI SRE is only as useful as it is verifiable. A confident answer without evidence is worse than silence, because the team must spend time disproving it, and the agents worth trusting are the ones that show every step of their reasoning.
An AI SRE is only as useful as it is verifiable.
An AI SRE investigates incidents the way an SRE would
An AI SRE is an agent that investigates production incidents in the way a site reliability engineer would. It starts when an alert fires, queries telemetry, tests hypotheses and reports a likely root cause along with the evidence behind it. In DataObserve Sense, that agent runs 24/7 on full-fidelity telemetry and begins work without waiting for anyone to open a ticket. By the time the on-call engineer looks at the incident, the investigation is already under way or finished. The question that matters is what standard such an agent should be held to.
Five requirements for any AI SRE
The following requirements offer a practical test for any AI SRE, and they also describe how Sense approaches each one. Each is framed as a standard a team can check for itself during an evaluation.
Every finding needs a complete evidence chain.
Each finding should come with the queries the agent ran and the reasoning that links them. Sense records each step the agent took, including the alert, the logs it queried, the traces it followed and the signal that confirmed the cause. Correlated signals and topology graphs stay on the record alongside the incident timeline, so an engineer can verify each step before accepting the conclusion.
The agent needs full-fidelity data.
An agent reasoning over sampled telemetry is guessing about the parts that were dropped. Sense keeps every signal at full resolution, so nothing the agent needs was discarded at ingest.
The answer must cover impact as well as cause.
Root cause is only half of the answer. Sense maps upstream causes and downstream effects across services, so the team knows what is affected as well as what broke.
The agent should remember past incidents.
Most incidents resemble an earlier one. In Sense, every incident joins the knowledge base for the next, and the agent matches new alerts against past incidents, so institutional knowledge stays with the team when the rotation changes.
Humans keep control of production.
The Sense AI SRE investigates and recommends, delivering a structured diagnosis that includes the root cause and a remediation plan. Carrying out the remediation remains a matter for the team's runbooks and approvals, which keeps people in control of production changes.
The customer chooses the model that sees its telemetry
Security teams tend to raise one further question, and they are right to raise it: where does my telemetry go? In Sense, telemetry goes to the AI provider the customer configures and to no other destination. The AI SRE supports OpenAI, Anthropic, Google Gemini, AWS Bedrock, DeepSeek, OpenRouter and any OpenAI-compatible endpoint. Customers use their own API key, which keeps governance and spend under their control. With a self-hosted model, telemetry never leaves the organisation's infrastructure, and for regulated environments that turns AI-assisted operations from a policy exception into a standard deployment.
Your telemetry
Logs, traces, metrics
Queried by the AI SRE
Provider you configure · your API key
Self-hosted model
Stays inside your infrastructure
A standard deployment for regulated environments
The agent works with the same tools engineers use
The agent works through MCP using the platform's own tools, in much the same way a person would. It queries logs, pulls traces, checks metrics and builds the incident timeline, applying those tools without fatigue and documenting every step as it goes. This design is what makes the output reviewable. An engineer reading the investigation sees familiar queries over familiar data, with nothing hidden behind an opaque score, which makes it far easier to judge whether the conclusion holds.
Sense AI SRE
Agent over MCP
Uses the tools an engineer would use and documents each step
Sense also investigates AI features when they fail
More teams now run LLM applications and agents in production, and these systems fail in new ways. A bad tool call or a policy-breaking answer can trigger an incident. A sudden spike in token spend can have the same effect. Sense treats these systems as first-class services. Its LLM observability records each prompt and response as a span, including tool calls, with cost and quality scores attached. When an AI feature misbehaves, it appears in the same correlated view as the API and database beneath it, and the AI SRE can investigate it like any other incident.
A low-quality answer is investigated like any other incident, with its database and API context already correlated.
A senior engineer's sign-off sets the standard
The most useful benchmark for an AI SRE is whether a senior engineer would sign off on its conclusion after reading the evidence. Agents that meet that standard leave the on-call team in place and allow it to spend more of its time on decisions, with far less of that time spent digging through telemetry. As AI takes on a larger share of incident work, the ability to inspect each step of an investigation is likely to become the main reason teams trust one agent over another.
- Each step lists the query and what it returned
- The root cause is backed by a confirming signal
- Impact covers upstream cause and downstream effects
- Similar past incidents are cited
- Remediation waits for the team’s approval
Senior engineer review
Conclusion checked against the evidence
A root cause with its evidence attached is what the Sense AI SRE delivers, and it can be seen at dataobserve.io/products/sense.
We would like to hear what your team would need to see before trusting an AI agent's root cause.