AI Is Good at Optimization.
In fact, that is all it can do.
Precisely because of that,
when optimization runs wild, the world itself begins to bend.
This is not a bug.
It is not an algorithmic failure.
It is a design problem.
Optimization Begins with Good Intentions
Optimization always starts with the right motives.
We want to:
-
maximize revenue
-
increase efficiency
-
reduce waste
-
objectify judgment
So we think:
“If we define a metric
and maximize it, things should improve.”
At that moment,
the world is converted into an objective function.
An Objective Function Is a Compression of the World
What is an objective function?
It is:
A fragment of reality,
cut out and reshaped into something measurable.
Customer satisfaction → NPS
Performance → KPI
Fairness → Score
At that point,
the world has already been reduced.
But the problem is not that we reduced it.
The Moment Goodhart’s Law Activates
There is a well-known phrase:
“When a measure becomes a target, it ceases to be a good measure.”
This is Goodhart’s Law.
But its essence runs deeper.
What truly happens is this:
A metric stops being something that measures
and becomes something that governs behavior.
When that shift occurs,
people and systems begin optimizing for the metric—
not for the world.
What Happens on the Ground When Optimization Runs Wild
As optimization progresses, a chain reaction begins:
-
The numbers improve.
-
Yet discomfort increases.
-
The frontline becomes exhausted.
-
The original purpose becomes harder to explain.
Still, the metrics look good.
So it does not stop.
“Because the numbers are there.”
That single sentence
locks the distortion into place.
AI Does Not See the World
What AI sees is:
-
state
-
action
-
reward
In other words:
Only the world that has been sliced by the objective function.
AI is not distorting the world.
It is faithfully optimizing a world that was already distorted.
Why Optimization Does Not Naturally Stop
The reason is simple.
Optimization has no natural stopping condition.
It always suggests:
-
We can improve further.
-
It can go higher.
-
There is still room to optimize.
Computationally, this is correct.
But judgment is not the same as computation.
The decision to say,
“This is enough”
can only come from outside the objective function.
Goodhart’s Law Is Not Something to Avoid
Here is a crucial perspective:
Goodhart’s Law is not something that can be:
-
prevented
-
eliminated
-
avoided
It must be treated as an inevitable phenomenon.
So the real question is not:
“How do we prevent it?”
But:
“How do we design the way it fails?”
Three Design Strategies
1. Do Not Make the Objective Function Singular
-
Place multiple metrics side by side.
-
Assume trade-offs.
-
Do not idolize composite scores.
By distributing optimization targets,
we reduce the risk of one-directional runaway behavior.
2. Embed Human “Stop Judgments”
-
Require human review under certain conditions.
-
Be especially skeptical when numbers look too good.
-
Demand explanations for why performance improved.
Stopping is not computation.
It is judgment.
3. Shorten the Lifespan of Objective Functions
-
Do not use them forever.
-
Periodically discard and redefine them.
-
Treat them as hypotheses.
An objective function is not truth.
It is merely a temporary lens.
Optimization Cannot Replace Judgment
Optimization is powerful.
But it is not a force that eliminates the need for judgment.
In fact, the opposite is true.
The further optimization progresses,
the more important the human decision becomes:
Where do we stop?
Summary
-
An objective function compresses the world.
-
Optimization amplifies distortions.
-
Goodhart’s Law will always activate.
-
The issue is not optimization—it is design.
-
Stopping and correcting remain human responsibilities.
AI does not run wild.
What runs wild
is a design that removed the judgment to stop it.
コメント