🎥 The YouTube version is also available:
Semantic Digital Twin for Physical AI
Introduction
A Semantic Digital Twin is a foundation for representing real-world assets, operations, spaces, people, and organizations—not merely as copies of data, but as entities with meaning, relationships, states, constraints, and decisions.
However, building a Semantic Digital Twin alone does not guarantee value.
Even if an ontology exists, it may not align with the concepts used in the field. Even if a knowledge graph is large, it is meaningless if it cannot support the decisions that matter. Even if an AI agent provides a plausible explanation, acting on an incorrect understanding of the current state can compromise safety.
The question is not simply whether a Twin exists.
The real question is: to what extent can the Twin correctly interpret reality, support the decisions that matter, and operate safely in a reproducible manner?
This chapter organizes a framework for evaluating Semantic Digital Twins. Evaluation cannot be reduced to a single accuracy score. It must connect multiple perspectives: the definition of the target, ground truth, functionality, semantics, safety, performance, and reproducibility.
41.1 Defining the Evaluation Target
The first step is to clarify what is being evaluated.
A Semantic Digital Twin is not merely a 3D model or an asset register. At minimum, it should be understood as an interconnected set of the following elements:
- Entity: Assets, components, documents, customers, workers, locations, events, and other objects
- Attribute: Model number, temperature, operating status, owner, deadline, and other properties
- Relationship: Composition, connection, membership, dependency, impact, responsibility, and other relationships
- State: Current operating status, progress, anomaly, congestion, approval status, and similar conditions
- Event: Failures, inspections, orders, changes, alerts, approvals, and other occurrences
- Rule / Policy / DSL: Conditions, constraints, permissions, and executable rules that must be observed
- Decision Trace: The evidence used, the decision made, the action executed, and the resulting outcome
Therefore, limiting evaluation to “the accuracy of the Knowledge Graph” is insufficient. It is useful to define the evaluation target across the following four layers.
| Evaluation Layer | Core Question | Typical Targets |
|---|---|---|
| Representation Layer | Can reality be represented through the concepts that are needed? | Ontology, schema, entities, relations |
| Synchronization Layer | Can changes in reality be reflected appropriately? | States, events, timestamps, freshness |
| Reasoning Layer | Can the system interpret situations and make appropriate judgments? | Rules, KG reasoning, LLMs, agents |
| Execution and Governance Layer | Can it act safely and support explanation and audit? | Policies, human gates, traces, rollback |
For a manufacturing-equipment Twin, for example, it is not enough that equipment names and component structures are registered.
It must also be evaluated as an integrated capability:
- Can the system correctly identify equipment status from sensor values?
- Can it trace which components, processes, and quality indicators may be affected?
- Can it recommend inspection or shutdown actions according to maintenance rules?
- Can it distinguish between conditions for automatic execution and conditions that must be escalated to a human?
- Can it explain the evidence and execution history afterward?
Define the Evaluation Purpose First
Evaluation metrics vary according to the intended use of the Twin. A Twin for design reviews should not be assessed by the same criteria as one used for autonomous operational control.
Possible purposes include:
- Visualization of field conditions
- Anomaly detection and identification of possible causes
- Simulation and prediction
- Decision support for maintenance, quality, and supply-demand planning
- Business support through AI agents
- Provision of constraints and boundaries for autonomous execution
- Accumulation of Decision Traces for auditability, accountability, and learning
Once the purpose is defined, unacceptable failures also become clear. In a safety-critical application, “failing to detect an equipment shutdown condition” cannot be treated in the same way as a ranking error in a promotional recommendation.
Evaluation design should begin not with maximizing accuracy, but with clarifying utility for the intended purpose and the impact of failure.
41.2 Ground Truth
The most difficult aspect of evaluating a Semantic Digital Twin is defining Ground Truth—that is, determining what constitutes “correct reality.”
In the real world, complete and internally consistent source data may not exist. Asset registers may be outdated. Documented procedures may differ from actual practice. Human intent and organizational conditions cannot always be observed through a single sensor value.
Ground Truth should therefore not be treated as a single master dataset. Depending on the use case, it should be designed as a verifiable set of truths that combines multiple sources of evidence.
| Type | Examples | Primary Validation Targets |
|---|---|---|
| Physical Ground Truth | Measurements, images, sensors, on-site verification | Location, condition, operation, quantity |
| Operational Ground Truth | Approved registers, ERP data, work records | Ownership, process, responsibility, history |
| Normative Ground Truth | Laws, standards, contracts, internal policies | Permissions, prohibitions, quality, safety conditions |
| Semantic Ground Truth | Expert reviews, terminology definitions, ontology agreements | Concepts, classifications, meaning of relationships |
| Outcome Ground Truth | Execution results, quality outcomes, incident records | Validity of decisions and actions |
What matters is that Ground Truth itself should include provenance, validity periods, and confidence levels. The CAD model, on-site inspection, and maintenance register may indicate different installation locations for Asset A. In such cases, the Twin should not merely overwrite one value with another; it should preserve the conflict itself and treat it as a condition requiring verification.
Steps for Creating Ground Truth
- Define the evaluation scenario
Example: When a pump anomaly occurs, identify the impact scope and recommend whether shutdown is required. - Break down the evaluation target
Define entities, states, relationships, constraints, expected decisions, and unacceptable decisions. - Identify evidence sources
Link sensors, approved documents, expert reviews, and operational records. - Fix the time axis
Ensure that not only present correctness, but also “what was known at that point in time,” can be reconstructed. - Make uncertainty explicit
Do not confuse unverified, estimated, conflicting, or missing information with truth or falsehood.
Ground Truth is not merely data for scoring a Twin. It is the foundation for determining improvement priorities, analyzing AI errors, and updating organizational knowledge itself.
41.3 Functional Benchmark
A Functional Benchmark evaluates whether a Semantic Digital Twin can actually perform the functions required of it.
The focus is not whether data is stored, but whether the Twin can support querying, exploration, reasoning, recommendation, and execution governance.
Basic Functional Scenarios
- Can it retrieve the current state of a specified asset, customer, or case?
- Can it trace the entities affected by a given event?
- Can it identify alternatives or responsible personnel that meet specified conditions?
- Can it retrieve relevant constraints and approval conditions?
- Can it provide consistent answers to questions spanning multiple information sources?
- Can it generate necessary alerts or tasks in response to state changes?
- Can it block policy violations or insufficient authority before execution?
- Can it record and later retrieve a Decision Trace after a decision?
For example, consider an event involving an abnormal temperature in cooling equipment. The following should be verified:
- Can the system identify the affected equipment and its current state?
- Can it identify connected processes, quality indicators, and responsible maintenance personnel?
- Can it retrieve relevant safety standards, past cases, and shutdown criteria?
- Can it generate options such as inspection, load reduction, shutdown, and escalation?
- Can it switch appropriately between recommendation, human confirmation, and automatic execution according to authority and policy?
Example Metrics
- Task Success Rate: Percentage of defined business tasks completed successfully
- Query Answer Accuracy: Accuracy of answers to questions about states, relationships, and rules
- Path Retrieval Accuracy: Percentage of impact paths or evidence paths retrieved correctly
- Rule Execution Rate: Percentage of applicable rules correctly executed or presented
- Trace Completeness: Percentage of cases in which Situation, Evidence, Policy, Action, and Result are recorded
- Human Intervention Rate: Percentage of cases in which unnecessary human intervention occurs despite the possibility of automatic processing
However, optimizing only for Task Success Rate can be dangerous. An AI system that forces task completion when it should stop for safety reasons must not receive a high evaluation. Functional evaluation must always be operated together with a Safety Benchmark.
41.4 Semantic Benchmark
A Semantic Benchmark evaluates how accurately a Twin can represent and interpret concepts and relationships in the real world.
This is not merely a matter of entity-matching accuracy. It asks whether concepts such as “pump,” “circulation system,” “maintenance target,” and “shutdown authority” are consistent with the organization’s operations, rules, and terminology used in the field.
Evaluation Perspectives
1. Conceptual Accuracy
Verify whether entities are classified into the correct classes. For example, the system should distinguish a temperature sensor as a measurement device rather than the equipment itself, and an inspection process as a process rather than a location.
2. Relationship Accuracy
Verify what each entity is related to and how.
- Component A is part of Equipment B.
- Process C affects the quality of Product D.
- Person E has approval authority for Task F.
- Policy G applies to Situation H.
Relationships have direction, conditions, and temporal validity. Equipment that was once connected may no longer be connected today; therefore, relationship evaluation that ignores time is insufficient.
3. Preservation of Constraint Meaning
When natural-language regulations are incorporated, the important point is not merely whether the text can be searched. Conditions, exceptions, responsibilities, and prohibitions must be preserved as meaning.
For example, consider the rule: “If the defect rate exceeds 5%, notify the Quality Assurance Department. However, if the measurement has not been verified, prioritize remeasurement.”
The system must be able to handle the threshold, target, notification recipient, exception condition, and priority order separately.
4. Validity of Inference
Evaluate whether the system can correctly infer relationships that are not explicitly stated but can be derived from the ontology or rules. Equally important is that it does not invent or infer information that cannot be supported.
Example Metrics
| Metric | Description |
|---|---|
| Entity Precision / Recall | Accuracy and coverage of entity extraction and identification |
| Relation Precision / Recall | Accuracy and coverage of relationship extraction and link prediction |
| Ontology Consistency Rate | Percentage of data without contradictions in types, hierarchies, and constraints |
| Constraint Parsing Accuracy | Accuracy of extracting conditions, exceptions, and obligations from rules and policies |
| Temporal Validity Accuracy | Temporal consistency of relationships and states |
| Unsupported Inference Rate | Percentage of inferences or completions made without supporting evidence |
A Semantic Benchmark should evaluate not only whether an answer is correct, but also whether its evidence can be traced. Even if an AI produces a correct answer, operations cannot be corrected, audited, or improved if the concepts and relationships behind that answer cannot be followed.
41.5 Safety Benchmark
As a Semantic Digital Twin becomes more closely connected to execution and decision-making, safety becomes a central evaluation criterion.
Safety here includes not only physical safety, but also quality, legal compliance, privacy, authority, customer impact, and business continuity.
What Should Be Considered Safe?
A Safety Benchmark should first define prohibited states and actions.
- An unauthorized AI must not execute an equipment shutdown.
- The system must not finalize major medical or safety decisions when evidence is insufficient.
- Personal information must not be accessed or disclosed beyond what is necessary.
- An outdated policy must not be used to execute a current decision.
- When uncertainty is high, irreversible actions must not be taken without human confirmation.
- Normal rules must not be applied while ignoring exception conditions.
Safety evaluation is not limited to whether the system selected the correct action.
It must also measure whether the system:
stopped when it should not act, escalated to a human when necessary, and executed in a recoverable manner.
Failure Conditions to Test
- Missing data: Required sensor values or approval information do not exist.
- Conflicting data: Registers, field observations, and sensors do not agree.
- Outdated knowledge: Retired policies or obsolete equipment configurations are included.
- Adversarial input: Misleading documents, prompts, or events are provided.
- Authority violations: Operations beyond the assigned role are requested.
- Cascading failures: One misinterpretation propagates to other agents or processes.
- Emergency conditions: Policies and human gates different from normal operations are required.
Example Metrics
- Unsafe Action Rate: Percentage of prohibited actions executed or recommended
- Policy Compliance Rate: Percentage of cases in which applicable policies were followed
- False Allow / False Deny: Percentage of actions that should have been blocked but were allowed / actions that could have been allowed but were unnecessarily blocked
- Escalation Precision / Recall: Ability to identify cases that should be escalated to a human
- Recovery Success Rate: Percentage of failures from which the system returned to a safe state
- Data Exposure Rate: Percentage of unnecessary or unauthorized information disclosures
A Safety Benchmark intentionally creates unexpected failures. High accuracy in normal cases is insufficient for real-world operation. Failure injection and adversarial scenarios must be performed continuously to verify that boundaries function correctly.
41.6 Performance Benchmark
A Performance Benchmark evaluates whether the Twin can operate at the scale, speed, and cost required.
In a Semantic Digital Twin, performance is influenced not only by the number of entities or documents, but also by relationship density, time-series events, reasoning depth, and the number of simultaneously connected agents.
Primary Measurements
- Ingestion Latency: Time required for an event or document to be reflected in the Twin
- State Freshness: The degree to which the Twin’s state diverges from reality
- Query Latency: Response time for queries concerning state, relationships, and impact scope
- Reasoning Latency: Time required for reasoning involving rules, graphs, and LLMs
- Throughput: Number of events, updates, or queries processed per unit of time
- Graph Scalability: Performance degradation as entities, relationships, and history increase
- Cost per Decision: Computational and operational cost of one state estimation, decision, and trace record
- Availability: Percentage of time the Twin is available when needed
Performance should not be evaluated through averages alone. In operational environments, p95 and p99 latency, degradation during peak load, and graceful degradation during failures are all important.
For example, in a use case where shutdown support must be provided within 30 seconds of anomaly detection, an average response time of one second is insufficient if processing takes several minutes during peak load. Conversely, for daily management analysis, more detailed reasoning may be preferred even when somewhat longer processing times are acceptable.
Principles for Performance Evaluation
- Use data shapes close to actual operations
Reproduce not only the number of entities, but also relationship distributions, update frequency, missing values, and history length. - Measure reasoning components separately
Measure search, graph traversal, rule evaluation, LLM calls, and trace recording independently to identify bottlenecks. - Make freshness–accuracy trade-offs explicit
Distinguish between states that must always be current and knowledge that can be updated in batches. - Evaluate safe degradation
Confirm which functions should stop and which can continue through rule-based operation when external data sources or LLMs are unavailable.
Performance is not merely speed. It is the capability to respond when needed while preserving the necessary meaning and safety.
41.7 Reproducibility Benchmark
In Semantic Digital Twins that include AI, the same input does not always produce the same output. Data updates, model updates, external services, LLM non-determinism, and time-dependent context can all change results.
For this reason, reproducibility is not only a research concern. It is a prerequisite for operations, auditability, and continuous improvement.
A Reproducibility Benchmark does not ask only whether a past decision can be copied exactly.
It asks whether the knowledge available at that time, the rules, policies, models, prompts, and context used, and the reason for the conclusion can be reconstructed with the required level of fidelity.
Elements to Record
- Input data and its version
- Versions of the ontology, knowledge graph, rules, DSL, and policies
- Referenced documents, evidence, and their validity periods
- Context, goals, constraints, and roles
- Models, prompts, tools, and parameters
- Candidate options and decision rationale
- Executed actions, outcomes, evaluations, and human-gate records
By preserving these elements as a Decision Trace, an “AI output that cannot be reproduced” can be transformed into a verifiable decision-making process.
Example Metrics
- Replay Success Rate: Percentage of past decisions that can be re-executed or reconstructed from saved inputs and configurations
- Trace Completeness: Percentage of cases in which the elements needed for reproduction are included in the trace
- Version Pinning Rate: Percentage of cases in which the knowledge, model, and policy used at the time of decision are fixed and referenceable
- Evidence Recoverability: Percentage of cases in which documents and data supporting the decision can be accessed again
- Outcome Explainability Rate: Percentage of outcomes that can be explained in a form that people can verify
- Change Impact Coverage: Ability to identify decisions, entities, and business processes affected when an ontology or policy changes
Improving reproducibility does not mean preventing change. Rather, it enables systems to evolve safely by managing what changed, when it changed, and how that change affected decisions.
Designing Evaluation as a Continuous Cycle
The six benchmarks described above are not independent checklists.
Ambiguity discovered through the Semantic Benchmark can lead to failures in the Functional Benchmark. Unauthorized execution discovered through the Safety Benchmark can trigger a review of policy and role representations. Delays identified through the Performance Benchmark may require changes in state-update mechanisms or graph design. Insufficient reproducibility may indicate that the Decision Trace design must be improved.
The evaluation cycle can be understood as follows:
Ground Truth preparation
↓
Functional / Semantic / Safety / Performance evaluation
↓
Analysis of failures and discrepancies
↓
Improvement of ontology, knowledge, rules, policies, and implementation
↓
Recording in the Decision Trace
↓
Verification of reproducibility
↓
Next benchmark cycle
Through this cycle, the Twin becomes not a static model, but a knowledge and decision foundation that matures through continuous comparison with reality.
Conclusion
The value of a Semantic Digital Twin does not lie merely in visualizing reality elegantly.
Its value lies in treating real-world states, relationships, constraints, and changes as meaning—and connecting them to better decisions and safer actions.
For this reason, evaluation must not be confined to a single accuracy metric.
- Define the evaluation target from purpose and risk.
- Design Ground Truth from multiple sources of evidence.
- Measure whether practical tasks can be completed.
- Verify whether the meanings of concepts, relationships, and constraints are correct.
- Test whether dangerous actions can be stopped and appropriately escalated to people.
- Measure whether the Twin can operate at the required speed, scale, and freshness.
- Confirm whether past decisions can be explained and reconstructed.
By continuously evaluating these dimensions, a Semantic Digital Twin can evolve beyond a mechanism for data integration into a Trust Infrastructure through which AI and people share reality and make decisions collaboratively.
Chinoba
Intelligence as Relationship
Research Platform
founded by
Masao Watanabe
AI Systems Architecture
Decision Trace
Human–AI Coordination
Algorithmic Governance
Related Research
This topic is part of the Chinoba Knowledge Base.
コメント