🎥 The YouTube version is also available:
Trust Infrastructure: Building the Foundation for Human–AI Collaboration
This question has suddenly become practical with the spread of AI coding assistance. The faster code generation becomes, the scarcer reviewers’ time becomes. Even if the volume of generated code increases tenfold, human reading speed does not increase tenfold.
The issue is therefore shifting from “Is AI-generated code good?” to “How can we create changes that can be trusted without requiring humans to read everything?”
In this context, the idea demonstrated by Bend 2 is important. Let AI write not only the code, but also a proof of its correctness; do not allow the change through unless that proof can be mechanically verified. Rather than relying on samples covered by tests, verify that specified properties hold across all executions.
But it would be a missed opportunity to confine this idea to programming languages. In an age where AI makes decisions and takes action within society and organizations, code correctness alone is not enough.
What we need is verifiable autonomy.
The Problem of Autonomy Exists Outside the Code
Consider an AI system that supports maintenance operations in manufacturing.
The AI reads equipment sensor data, maintenance records, parts inventory, past failure histories, and worker qualification information. It proposes inspection plans. For minor inspections, it may even automatically issue work orders, reserve parts, and notify the site.
In this setting, having software that operates without bugs is necessary, but it is not sufficient.
What truly matters are questions such as these:
- May an AI determine whether safety-critical equipment can continue operating without human approval?
- When sensor data is missing, may it issue a work instruction based on estimated values?
- To what extent may it reserve inventory—across which plants, budgets, and urgency levels?
- When the evidence required for statutory inspections or shutdown decisions is incomplete, should the AI stop, escalate, or conduct further investigation?
- If an accident or quality problem occurs later, can the data, rules, responsible parties, and exception handling behind the decision be reconstructed?
These are not merely specifications for code. They are specifications for organizational intent, responsibility, and boundaries.
The Idea of Decision Law
This is where I propose the concept of Decision Law.
Decision Law is not simply a business rule. It is a machine-verifiable expression of the conditions under which an AI may act autonomously, the conditions under which it must ask a human, and the conditions under which it must stop execution.
For a maintenance-support AI, it might be expressed as follows:
- A decision to continue operating safety-critical equipment must not be finalized without approval from a qualified human.
- If a required sensor value or recent inspection record is missing, the system must not issue an automatic “safe” determination.
- Automatic procurement of replacement parts is permitted only when the approved budget, supplier, and spending limit are all satisfied.
- Any proposal intended to avoid an equipment shutdown must include alternatives, a risk assessment, and references to supporting evidence.
- If the inputs, applicable rules, evidence, executor, and outcome of a decision cannot be retained in a Trace, the action must not be executed.
The key point is not that the AI provides a plausible explanation in prose. Before execution, a verifier must confirm that the required evidence, authority, and boundaries have been satisfied.
Verifying Act / Ask / Stop
From the perspective of a Runtime OS, autonomy is not binary.
Rather than forcing a choice between full automation and full manual operation, each decision can be classified into three states:
- Act: The Decision Law is satisfied, and the action may be executed automatically.
- Ask: Evidence, authority, risk, or an exception requires human confirmation.
- Stop: A boundary has been violated, required evidence is missing, or the uncertainty is not permissible; execution must not proceed.
This classification must not be determined solely by a model’s confidence score. Highly confident errors are possible. What matters is whether the decision satisfies the laws defined in advance.
In other words, autonomy is not a blanket authority granted to an AI.
Autonomy exists only when intent, evidence, authority, boundaries, and verification are bound together at runtime.
What Can—and Cannot—Be Proven
Formal methods also have a long-standing limitation here.
A verifier can precisely check the laws that have been written. It cannot discover laws that have not been written. Unless the law “safety-critical equipment requires human approval” is defined, the verifier cannot identify its absence as a problem.
This is not a failure. It makes the locus of design responsibility explicit.
Bugs do not disappear from implementation. They are decomposed into implementation errors, specification errors, and specification omissions. Of these, machines can directly detect only violations of laws that have already been defined.
That is why Decision Law must not end as a fixed rulebook.
Exceptions encountered in the field, rejections at Human Gates, near misses, audit findings, and decisions with poor outcomes should all be accumulated as Traces. The organization can then revisit which laws were missing and which boundaries were defined too coarsely. This is where Decision Trace and Knowledge Flow become connected to the design of autonomy.
From Knowledge to Law, from Law to Evidence
Maintenance records, quality-incident reports, procedures, engineering changes, and audit records scattered across an organization contain fragments of decision laws.
Semantic Chunk and Evidence-Grounded Ontology organize these fragments into meaningful units—equipment, parts, processes, anomalies, responses, responsibilities, and outcomes—while preserving links back to the original sources. On that basis, recurring conditions and exceptions can be extracted as candidates for Decision Law.
However, candidates extracted by AI must never become laws automatically. They must be reviewed and approved with an accountable owner, scope of application, validity period, exceptions, and verification method.
The flow can be expressed as follows:
Raw Data → Semantic Chunk → Ontology / Policy Candidate → Human Review → Decision Law → Runtime Verification → Decision Trace → Law Revision
Knowledge does not exist merely for search. It exists to define the boundaries within which decisions may be made, verify execution, and improve the laws afterward.
Start by Writing the First Law
There is no need to formalize every aspect of the business. Instead, begin with the one point that is most important and most difficult to reverse if wrong.
In a manufacturing environment, for example:
For safety-critical equipment, if any required observation value, inspection record, or authorization check is missing, the AI must not autonomously finalize a decision to continue operation.
This one sentence affects implementation, data design, authorization design, user interfaces, audit logs, and Human Gates. That is precisely why it is valuable.
Mature autonomy in the AI era is not a state in which “AI can do anything.” It is a state in which the range of permitted autonomous action is explicit, and in which every execution can be checked to ensure that the AI does not exceed that range.
More testing and better human review both matter. But when agents begin to participate seriously in organizational execution, they are not enough.
What will be needed is not only competition over AI capability.
We must describe what must be protected as Decision Law, and retain a Trace showing that it was protected.
The foundation for doing so is an Autonomy Assurance Runtime.
Chinoba
Intelligence as Relationship
Research Platform
founded by
Masao Watanabe
AI Systems Architecture
Decision Trace
Human–AI Coordination
Algorithmic Governance
Related Research
This topic is part of the Chinoba Knowledge Base.
コメント