🎥 The YouTube version is also available:
Transform the First 15 Minutes of Incident Response | Proactive Investigation Agent

When a system failure or anomaly occurs, the first tasks performed by an on-call engineer are usually the same:
- Check the alert details
- Open dashboards
- Search error logs
- Review recent deployments and configuration changes
- Read related tickets and change histories
- Look for other anomalies occurring during the same time window
- Estimate the scope of impact
This initial investigation is essential, but it takes time. During an incident, information is scattered across multiple places, and the engineer must form hypotheses under severe time pressure.
This is where a Proactive Investigation Agent becomes valuable.
When it detects an incident, an anomaly, or a decline in answer quality, it begins collecting logs, change histories, and related Decision Traces before the on-call engineer starts investigating. It then produces an evidence-backed initial investigation report.
However, this AI is not intended to perform automatic recovery.
Its role is not to fix the incident on its own. Its role is to create the conditions in which people can begin making safe decisions sooner.
The first fifteen minutes are often the most costly
In incident response, time is needed before corrective action can even begin—time to understand what is happening.
For example, imagine that the API error rate rises sharply. At that point, several possible causes may exist:
- A recent application update
- Latency or an outage in an external API
- Exhausted database connections
- A sudden surge in traffic from a particular tenant
- A configuration or secret update
- Backlog in background jobs
- A network or authentication infrastructure issue
The alert itself only says, “The error rate has increased.” The on-call engineer must gather related information and narrow down the possibilities.
The Proactive Investigation Agent starts collecting investigation material as soon as it receives the alert:
- Logs immediately before and after the event
- Affected APIs, jobs, tenants, and regions
- Error types and the conditions under which they increased
- Recent deployments, configuration changes, and Feature Flag changes
- The operational status of dependent services
- Similar past incidents and relevant runbooks
- Related Decision Traces and approval histories
Instead of requiring people to search for everything from scratch, the agent provides a structured starting point for investigation.
The AI’s role is not to determine the cause conclusively
When using AI in incident response, it is tempting to think: “Why not let the AI recover the system automatically?”
But recovery actions in production can involve serious risks: data loss, secondary failures, security issues, customer impact, and contractual consequences.
For example, it may be dangerous for an AI to automatically perform actions such as:
- Restarting a production database
- Deleting or replaying queues
- Disabling a Feature Flag
- Executing a rollback
- Sending emails to customers
- Removing access restrictions
- Significantly increasing infrastructure resources and costs
As a rule, a Proactive Investigation Agent does not perform such changes.
Instead, it presents hypotheses and evidence in a form such as:
Since 10:03 JST, the 5xx error rate for the payment API has increased.
At 10:01 JST, a configuration change to the authentication service was applied, followed immediately by a sharp increase intoken_validation_failederrors.
The same authentication error has also been observed across three services other than the payment API.
The temporal correlation with the configuration change is strong, but causality has not yet been confirmed. The approved rollback procedure is documented in Runbook v3.2.
In this way, the AI does not treat speculation as a conclusion. It separates observed facts, relevant relationships, and uncertainty.
What an initial investigation report should contain
An investigation report should not merely be a summary of a large volume of logs. It must be structured so that the on-call engineer can make an initial judgment.
| Item | Contents |
|---|---|
| Incident summary | What happened, when, and to what extent |
| Scope of impact | Affected functions, users, regions, and tenants |
| Timeline | Sequence of alerts, changes, and anomalies before and after the event |
| Observed evidence | Links to logs, metrics, traces, and monitoring dashboards |
| Related changes | Deployments, configuration changes, Feature Flags, and dependency changes |
| Related Decision Trace | Actor, approver, rules, tools, and execution results |
| Hypotheses | Evidence-backed possible causes and their confidence levels |
| Unknowns | Missing logs, insufficient permissions, or inaccessible information |
| Recommended next checks | Additional logs, stakeholders, and runbooks to review |
| Prohibited or deferred actions | Actions that must not be performed without human approval |
The objective of this report is not for AI to become the incident manager. It is to help engineers reach the necessary evidence quickly and begin making decisions.
Connecting change history with the Decision Trace
Some form of change has often occurred shortly before an incident.
But simply reviewing a list of Git commits or deployment histories is not enough. What matters is understanding why the change was made, who approved it, and under which rules it was executed.
With a Decision Trace, the full chain of a change can be examined:
- Who proposed the change
- Which design documents, tickets, or incident reports supported it
- Whether AI participated in proposing or reviewing it
- Which test and validation results were checked
- Who approved it
- When and to which environment it was deployed
- What monitoring results were observed afterward
This prevents the investigation from beginning with the vague assumption that “the most recent change looks suspicious.”
This configuration change was applied through an approved workflow at 10:01 JST.
No authentication errors were observed in staging.
However, tenant-specific settings enabled only in production had been excluded from the validation scope.
This makes it possible to examine not only the change itself, but also the judgment and validation process behind it.
Preserving “unknown” correctly
In an initial investigation, the most dangerous failure mode is for AI to fill information gaps with plausible-sounding explanations.
For example, if monitoring logs are missing, detailed logs from an external service are unavailable, or data access is restricted, the AI must state this explicitly.
Job execution logs between 09:50 and 10:10 JST could not be retrieved because of an incorrectly configured retention policy.
Therefore, it cannot currently be determined whether delays in background jobs contributed to the incident.
Such unknowns must not be treated as normal conditions or as irrelevant.
A trustworthy investigation AI is not one that reaches conclusions quickly. It is one that can accurately distinguish between what has been verified and what has not.
Declining answer quality should also be investigated
The scope of a Proactive Investigation Agent is not limited to infrastructure incidents.
A decline in the quality of AI systems themselves should also be subject to detection and initial investigation.
For example, the following may be warning signs:
- An increase in responses without supporting citations
- Search results containing an excessive number of outdated documents
- Higher rates of response rejection or repeated user questions
- A sharp increase in Unknown outcomes in a particular business domain
- A higher failure rate in tool calls
- More attempts to access unintended data sources
In such cases, the agent can collect and organize relevant information about recent model changes, prompts, search indexes, permission settings, policy updates, and data integration status.
If organizations operate AI systems, they must also monitor the quality of the AI itself.
Easier to introduce because it does not perform automatic recovery
There is no need to design a Proactive Investigation Agent as an automatic recovery agent from the outset.
In fact, limiting it to investigation and reporting—without granting change authority—makes it safer and easier to introduce.
| Function | Treatment by the Proactive Investigation Agent |
|---|---|
| Receive alerts | Performed |
| Collect logs and metrics | Performed within authorized scope |
| Review change history and Decision Traces | Performed |
| Search similar incidents and runbooks | Performed |
| Organize possible causes | Performed |
| Create an initial investigation report | Performed |
| Change production settings | Not performed |
| Restart systems or execute rollbacks | Not performed |
| Send external notifications to customers | Generally not performed |
| Make recovery decisions | Performed by humans |
The first step is to introduce an AI that “investigates, but does not change.”
With this boundary clearly defined, operations teams can benefit from AI-generated insights while retaining responsibility and final authority over recovery actions.
Conclusion: AI should become an investigator before it becomes a recovery operator
The most valuable AI in incident response is not necessarily an AI that restores systems on its own.
A more realistic and useful approach is an AI that collects the necessary evidence the moment an anomaly is detected, connects change history with Decision Traces, and creates a state in which the on-call engineer can begin making decisions immediately.
- What is happening?
- Where is the impact?
- What changed immediately beforehand?
- Which evidence supports each possible cause?
- What remains unconfirmed?
- What should be checked next?
The AI presents these answers with evidence, while leaving judgment and recovery operations to people.
A Proactive Investigation Agent is not a mechanism for handing operations over to AI.
It is an initial-investigation foundation for the AI era—one that enables people to make faster, calmer, and more explainable decisions.
Chinoba
Intelligence as Relationship
Research Platform
founded by
Masao Watanabe
AI Systems Architecture
Decision Trace
Human–AI Coordination
Algorithmic Governance
Related Research
This topic is part of the Chinoba Knowledge Base.
コメント