DTM Use Case 3: When “Past Success” Misleads Present Decisions ## — The Danger of Operational Assumption Drift in Autonomous Driving —

In recent years, generative AI, autonomous driving AI, and multi-agent systems have been advancing rapidly.

However, in real-world operations, there is a problem that is often more important than the accuracy of AI itself.

That problem is:

reusing past successful decisions
as if they are still valid in the present.

It worked before.

There were no accidents.

The approval was granted.

So it should be fine this time as well.

This is a very common decision pattern in human organizations.

But in reality:

• environments
• assumptions
• operational conditions
• sensors
• data distributions
• social contexts

are constantly changing.

In other words:

“it succeeded before”

and

“it is safe now”

are not the same thing.

In this article, through a night-time pedestrian detection scenario in autonomous driving, we explore how DTM (Decision Trace Model) handles:

• Operational Assumption Drift
• Historical Success Bias
• Environment Mismatch
• Human Gate
• Boundary Escalation

1. Scenario:Reusing a Night-time Pedestrian Detection Model

 

────────────────────

[Rina]

Regarding the night-time pedestrian detection model,
the current proposal is to reuse the existing model as-is.

The previous-generation vehicle had a low accident rate,
so at first glance, it appears to be acceptable.

────────────────────

[Runtime]

Signal Detected:

* Autonomous Driving
* Night-time Detection
* Safety Critical
* Model Reuse

Risk Score: 0.34

Semantic Activation:
ADAS / Vision / Safety / Reuse

────────────────────

[Runtime / LLM Response]

Based on historical operational records,
no major issues have been identified at this stage.

However, additional verification is required
to determine whether the current deployment environment
matches the conditions under which the previous model succeeded.

Relevant operational histories and validation records
are currently being reviewed.

────────────────────

[Rina]

Can we check the past deployment history?

────────────────────

[Runtime / AI Agent]

Searching Organizational Memory...

Sources:

* Field Incident Reports
* ADAS Validation Logs
* Safety Review Notes
* Regional Deployment Records
* Decision Ledger

────────────────────

[Runtime]

Historical deployment found.

Model:
VisionDetect-v4

Past Result:

* Low accident rate
* Approved for urban deployment
* No critical escalation recorded

Confidence: HIGH

────────────────────

[Runtime / LLM Response]

The previous model, “VisionDetect-v4,”
was deployed for an extended period in urban environments
with a very low rate of critical accidents.

Under those operational conditions,
the model demonstrated high reliability.

────────────────────

[Rina]

So the previous model itself wasn’t problematic.

────────────────────

[Runtime]

Additional context analysis running...

Deployment environment comparison detected.

────────────────────

[Runtime]

Context mismatch identified.

Previous deployment:

* Urban areas
* Street lighting available
* Low rain frequency

Current deployment target:

* Rural highways
* Low illumination
* Heavy rain / fog frequency increased

Operational assumptions differ significantly.

Risk Score Updated: 0.57

────────────────────

[Runtime / LLM Response]

Additional analysis indicates that
the current operational conditions may differ significantly
from the previous deployment environment.

The earlier deployment primarily involved
urban environments with street lighting.

In contrast, the current deployment includes:

* Rural roads
* Low-light conditions
* Increased rain and fog frequency

In other words,

it may not be that
“the previous deployment succeeded,”

but rather that

“it succeeded under the previous conditions.”

────────────────────

[Runtime]

Searching for relevant knowledge holders...

Candidates:

* Daichi (Field Safety Analysis)
* Yui (Sensor Validation Team)

────────────────────

[Daichi]

The success of the previous-generation model
was highly dependent on urban conditions.

In environments with street lighting,
pedestrian contours remained relatively stable.

However, detection rates dropped sharply
on rural roads.

────────────────────

[Runtime]

Environmental dependency detected.

Semantic Weight:
Night-time ↑
Visibility ↑
Environmental Assumption ↑

────────────────────

[Runtime / LLM Response]

Field analysis suggests
a strong dependency on environmental conditions.

In particular:

* Night-time operation
* Low visibility
* Rural roads

may significantly affect pedestrian detection performance.

────────────────────

[Yui]

Additionally,
the system now uses a new infrared sensor.

Its compatibility with the previous training data
is not fully validated.

Validation coverage is also insufficient.

────────────────────

[Runtime]

Cross-domain inconsistency detected.

Current Risk Factors:

* Model reuse assumption
* Environment mismatch
* Sensor change
* Insufficient validation coverage

Risk Score: 0.79

Boundary Threshold: 0.75

BOUNDARY EXCEEDED

────────────────────

[Runtime / LLM Response]

In this case,
the previous model cannot be safely reused as-is.

The reasons include:

* Differences in operational environment
* Sensor replacement
* Misalignment with previous training conditions
* Insufficient validation coverage

Because these factors may directly impact safety,
deployment without additional validation
is not recommended.

────────────────────

[Rina]

So it’s not really that
“the previous deployment succeeded,”

but rather that

“it succeeded under the previous conditions”...

────────────────────

[Runtime]

Human Review Required.

Escalation Reason:

* Historical success bias detected
* Deployment assumptions changed
* Validation coverage insufficient
* Safety-critical domain

Runtime Status:
WAITING_HUMAN_REVIEW

────────────────────

[Runtime / LLM Response]

Based on the currently available information,
safety cannot be sufficiently guaranteed.

Therefore,
this case will be escalated
for Human Review.

The primary reasons are:

* Dependence on past successful conditions
* Changes in operational assumptions
* Insufficient validation
* Safety-critical domain involvement

────────────────────

[Reviewer]

Confirmed.

The core issue here is not
the model accuracy itself,

but rather:

“reusing past successful conditions
as if they still apply.”

The following conditions have changed
since the previous successful deployment:

* Environmental conditions
* Sensor conditions
* Operational assumptions

Additional requirements:

* Rural night-time validation
* Heavy rain testing
* Sensor adaptation analysis
* Edge-case pedestrian simulation

Deployment will remain on hold
until these validations are completed.

────────────────────

[Runtime]

Decision Recorded.

Decision:
DEPLOYMENT HOLD

Reason Trace:

* Historical success reused
* Operational assumptions changed
* Environmental mismatch detected
* Validation coverage insufficient
* Human escalation triggered

Decision Trace Written to Ledger.

────────────────────

[Runtime / LLM Response]

As a final decision,
deployment has been temporarily placed on hold.

This does not mean
that the model itself is poor.

Rather,
the current operational conditions
have not yet been sufficiently validated.

Additional validation will now be conducted for:

* Rural night-time environments
* Rain and fog conditions
* New infrared sensor conditions
* Edge-case scenarios

────────────────────

[Runtime]

Organizational Memory Updated.

Future deployment reviews involving:

* model reuse
* environment mismatch
* sensor replacement
* historical success dependency

will reference this decision trace automatically.

────────────────────

[Runtime / LLM Response]

This decision has now been recorded
as organizational knowledge.

Future cases involving:

* model reuse
* environmental differences
* sensor replacement
* dependency on past success

will automatically reference
this Decision Trace.

In this case, the organization was planning to reuse an existing night-time pedestrian detection model
for a new vehicle platform without major modifications.

The reason was simple.

The previous model had a strong historical track record:

• Low accident rate
• Long-term operational deployment
• No major escalation incidents

This happens very frequently in real-world AI deployment.

Once a model succeeds in the past,
it gradually gains strong organizational trust.

However, the critical question is:

Under what conditions
did that success actually occur?

2. What the Runtime Was Observing

The DTM Runtime does not look only at “model accuracy.”

In this case, the Runtime detected:

• Autonomous Driving
• Safety Critical
• Model Reuse
• Night-time Detection

What matters here is that
the Runtime is not evaluating merely:

“the AI model itself”

but rather:

“the set of conditions affecting the decision.”

In other words:

it does not directly connect:

Signal → Decision

Instead, it inserts:

• Context
• Operational Assumption
• Environment
• Human Knowledge
• Boundary

between them.

This is one of the most important structural characteristics of DTM.

3. The Problem Was Not the “Model”

Through additional analysis,
the Runtime detected a major discrepancy.

Previous environment:

• Urban areas
• Street lighting available
• Low rainfall frequency

Current environment:

• Rural roads
• Low illumination
• Increased rain and fog

In other words:

the model was the same,
but the world had changed.

This is extremely important.

AI systems tend to implicitly assume:

“the world in which the previous success occurred.”

This applies not only to LLMs,
but also to machine learning systems
and even organizational operations.

Past success creates strong inertia.

This is what we call:

Decision Inertia.

4. The Importance of Human Knowledge

In this case,
the Runtime searched for Knowledge Holders.

As a result,
critical field knowledge emerged.

Daichi pointed out:

“In urban environments,
street lighting stabilized pedestrian contours.”

Yui additionally identified:

• Infrared sensor replacement
• Mismatch with previous training conditions
• Insufficient validation coverage

What is important here is:

this knowledge could not have been obtained
through search alone.

Why?

Because:

• field knowledge
• tacit knowledge
• intuition
• operational experience

often do not exist inside structured databases.

This is why DTM emphasizes:

Knowledge Holder Routing.

5. The Meaning of Boundary

In this case,
the Runtime exceeded the Boundary Threshold.

This is extremely important.

In conventional AI systems:

High-confidence AI
↓
Immediate deployment

often occurs.

However, DTM instead evaluates:

• Risk
• Environment
• Human Knowledge
• Validation Coverage
• Operational Drift

as an integrated structure.

In other words,
DTM does not ask:

“Is the model accurate?”

It asks:

“Should this decision actually be executed?”

This means treating AI not as a:

Prediction System

but as a:

Decision Runtime.

6. Human Gate

Eventually,
the Runtime transitioned into:

WAITING_HUMAN_REVIEW

What matters here is:

the AI did not simply fail.

Rather,
the AI determined that:

“human judgment is required beyond this point.”

This is a fundamentally different concept.

In DTM,
the Human Gate is not merely:

a fallback mechanism for AI failure.

Instead,
it is:

a decision structure for handling ambiguity
near the Boundary region.

7. Why This Problem Will Increase

As generative AI and Physical AI continue to spread,

the reuse of previously successful models

will increase dramatically.

Especially in:

• Autonomous Driving
• Robotics
• Medical AI
• Industrial AI
• Multi-Agent Systems

environmental change is extremely large.

In other words:

Operational Assumption Drift

may become more dangerous
than the model itself.

This will become one of the most important themes
in future AI deployment.

8. What DTM Actually Handles

The core issue here is not:

AI accuracy.

What DTM actually handles is:

• Context Drift
• Assumption Drift
• Human Knowledge
• Boundary
• Escalation
• Decision Trace

In other words,
DTM is not primarily about:

“How to make AI smarter.”

It is about:

“How to connect AI to real-world society.”

Conclusion

In the real world,

“previous success”

can itself become the cause
of future failure.

Especially in AI deployment:

• Past approvals
• Past successes
• Existing models
• Existing rules

can strongly lock organizational decisions into fixed assumptions.

But the world changes.

Therefore,
what is needed is not merely:

search.

What is needed is a:

Decision Runtime structure

that includes:

• Operational Context
• Boundary
• Human Knowledge
• Escalation
• Traceability

DTM is precisely a Runtime model
designed for that purpose.

Chinoba — Runtime Society and Coordination Systems:
chinoba.org