How Should We Evaluate Multi-Agent Systems?

As generative AI continues to evolve, AI is moving beyond standalone models toward multi-agent systems, where multiple AI agents collaborate to accomplish tasks.

Today, we are beginning to see architectures in which:

• Research Agents
• Reasoning Agents
• Execution Agents
• Monitoring Agents
• Evaluation Agents
• Human Support Agents

work together to process business operations and decision-making workflows.

This raises an important question:

How should we evaluate multi-agent systems?

Why Single-Model Evaluation Is Not Enough

Traditional AI evaluation has focused on metrics such as:

• Accuracy
• Precision
• Recall
• F1 Score
• BLEU
• ROUGE
• Hallucination Rate

These metrics work well for a relatively simple structure:

Input
↓
Model
↓
Output

However, multi-agent systems introduce additional factors that significantly affect system quality:

• Agent-to-agent communication
• Task delegation
• Decision handoffs
• Human-in-the-loop processes
• External system integration
• Exception handling

As a result, the evaluation target is shifting from:

Individual AI models

to

The entire decision-making system composed of multiple AI agents.

Multi-Agent Systems Are Connected to the Real World

More importantly, multi-agent systems are not merely AI-to-AI conversation systems.

In reality, they operate while interacting with:

• Humans
• Organizations
• Business processes
• Legal frameworks
• Safety requirements
• Operational procedures
• External systems

In other words, they are becoming:

Execution systems that make decisions within real-world environments.

In such environments, simply producing the correct answer is not enough.

The Importance of a Quality Engineering Perspective

Once multi-agent systems become connected to the real world, evaluation can no longer focus solely on AI reasoning performance.

What becomes important is whether the system can continue operating safely and reliably despite uncertainty, variability, and unexpected events.

In real operational environments, factors such as:

• Sensor anomalies
• Communication delays
• Component variability
• Human judgment variability
• Unexpected disturbances

occur regularly.

Therefore, what matters is not merely achieving high accuracy under ideal conditions.

What matters is maintaining expected behavior under uncertainty.

This perspective closely resembles the concept of Robust Design developed in quality engineering and manufacturing.

More importantly, in the real world:

“Correct on average” is not sufficient.

Even if a system performs correctly 99% of the time, the remaining 1% may result in:

• Crossing safety boundaries
• Missing critical anomalies
• Executing incorrect actions
• Bypassing required human review

Any of these failures may lead to serious consequences.

Therefore, multi-agent systems must be evaluated based on whether they:

• Remain stable under disturbances
• Fail safely under abnormal conditions
• Escalate appropriately to humans
• Support accountability
• Enable decision reconstruction

This is not merely an accuracy problem.

It is a quality engineering problem focused on evaluating the robustness of the overall system.

Why Output Evaluation Alone Is Insufficient

However, there is a challenge.

These characteristics cannot be evaluated simply by observing final outputs.

For example, even if a decision appears correct, it may not be clear whether:

• It was correct by chance
• It was correct because an appropriate process was followed

Likewise, it is impossible to determine from the final output alone whether:

• Safety mechanisms functioned properly
• Human review occurred when required
• Boundaries were respected

Therefore, multi-agent systems must be evaluated not only by:

What decision was produced

but also by:

How that decision was produced.

What Should Actually Be Evaluated?

In multi-agent systems, evaluating only the correctness of the final output is insufficient.

Two systems may produce the same answer while exhibiting very different levels of reliability, safety, and governance.

For example:

• Was the task routed to the correct agent?
• Did the system follow the appropriate review process?
• Was human approval obtained when necessary?
• Did the system respect operational boundaries?
• Was the decision process reproducible?

These questions cannot be answered by looking only at the final output.

Instead, evaluation must focus on the decision-making process itself.

Specifically, we need visibility into:

• What information was shared
• Which agent handled each task
• Where decision branches occurred
• Where execution stopped
• Why escalation occurred
• Whether human review functioned properly
• Whether boundaries were respected
• Whether the process can be reconstructed afterward

This represents a shift from output evaluation toward:

Decision Process Evaluation.

Key Evaluation Dimensions

1. Coordination Quality

Can agents collaborate effectively?

Evaluation includes:

• Appropriate routing
• Information completeness
• Prevention of infinite loops
• Reduction of unnecessary agent calls

The goal is to measure the quality of coordination rather than individual performance.

2. Boundary Compliance

Can the system stop appropriately when required?

Evaluation includes:

• Stopping high-risk operations
• Requesting human approval
• Rejecting dangerous actions
• Enforcing permission controls

In many real-world systems, the ability to stop safely is as important as the ability to act.

3. Escalation Accuracy

Can the system determine when control should return to humans?

Typical escalation conditions include:

• High uncertainty
• Conflicting information
• Legal or ethical concerns
• High-risk decisions

The objective is not full automation.

The objective is appropriate human involvement.

4. Decision Traceability

Can the decision process be reconstructed?

This includes understanding:

• Who made the decision
• What evidence was used
• Which boundaries were crossed
• Who approved the outcome

Without traceability, governance and accountability become impossible.

Runtime Must Be Evaluated

The evaluation dimensions described above share a common characteristic.

Coordination Quality,

Boundary Compliance,

Escalation Accuracy,

and Decision Traceability

cannot be achieved by individual agents alone.

For example:

Who controls routing?

Who determines when execution should stop?

Who decides when escalation is necessary?

Where are decision traces recorded?

These capabilities depend less on individual agents and more on the system-wide control structure.

Therefore, the evaluation target expands from individual agents to the mechanisms that coordinate them.

This is where the concept of a Runtime becomes critical.

A Runtime is the execution layer responsible for enabling multiple agents to operate safely and cooperatively.

It typically manages:

• Routing
• Boundary Control
• Permission Management
• Escalation
• Human Gates
• Decision Logging
• Coordination Flows

As a result, evaluating multi-agent systems requires evaluating not only AI models but also the:

AI Coordination Runtime

that governs them.

Future Evaluation Metrics

How should such runtimes and multi-agent systems be measured?

Traditional metrics focused on model performance:

• Accuracy
• Precision
• Recall
• F1 Score

However, multi-agent systems require evaluation at the system level.

Important metrics may include:

• Coordination Stability
• Runtime Safety
• Boundary Robustness
• Escalation Reliability
• Human Governance
• Decision Explainability
• Trace Integrity
• Multi-Agent Failure Recovery

These metrics do not measure:

How intelligent AI is.

Instead, they measure:

How safely and reliably an AI-enabled decision system can operate.

Evaluation is therefore evolving from:

AI Model Evaluation

to

AI Coordination System Evaluation

and ultimately toward

AI Societal System Evaluation.

Conclusion

In the era of multi-agent systems, the object of evaluation is fundamentally changing.

Traditional AI evaluation focused on:

• Model accuracy
• Model capability
• Benchmark performance

However, as multiple agents collaborate and interact with humans, organizations, business processes, and regulations, reasoning performance alone is no longer sufficient.

The key questions become:

• Can the system coordinate effectively?
• Can it stop safely?
• Can it return control to humans when necessary?
• Can its decisions be traced?
• Can it remain stable under uncertainty and exceptions?

These are questions about overall system quality.

The evaluation target is expanding from:

AI Models

to

AI Decision Systems

and ultimately to

AI Societal Systems.

The future of AI will require not only:

Model Intelligence

but also:

Coordination Intelligence

Governance Intelligence

and

Runtime Intelligence

At the center of these capabilities sits the Runtime:

the layer responsible for coordination, control, traceability, governance, and explanation across agents, humans, and organizations.

The challenge of evaluating multi-agent systems is no longer about measuring how intelligent AI is.

It is about measuring how safely, reliably, and explainably AI-enabled societal systems can operate.

---

Chinoba — Runtime Society and Coordination Systems:
chinoba.org