Can VLMs Become the “Eyes” of Robots? What RoboVista Reveals About Their Limits and the Role of Semantic Digital Twins

🎥 The YouTube version is also available:

Semantic Digital Twin with Omniverse and Isaac SimConnecting 3D Simulation to Operational Meaning

Books: A Practical Guide to Semantic Digital Twin for Physical Al: Building with Chinoba PF Ontology • Knowledge Graph • DSL • Decision Trace

Give a robot a camera image and ask: “Is this part fitted correctly?” “Which way should it move next?” “How should it recover from failure?”

As Vision-Language Models (VLMs) improve, such use cases are no longer limited to laboratory demonstrations. Across agriculture, manufacturing, surgery, homes, autonomous driving, and unstructured environments where people and objects constantly change, VLMs are increasingly being integrated into perception, reasoning, planning, and decision-making pipelines.

But understanding images and language is not the same as acting safely in the physical world.

Presented at RSS 2026, RoboVista is a benchmark designed to measure that gap. Through 474 expert-annotated Robot-VQA (Visual Question Answering) examples, it evaluates 39 robotics task types across six domains: agriculture, industry, domestic robotics, surgery, autonomous driving, and open robot datasets. Its questions go beyond object recognition to include spatial relationships, action feasibility, next actions, failure detection, and recovery.

RoboVista paper
RoboVista project page

These results carry an important implication for Physical AI. It is not enough to use a VLM merely as a model that “looks and answers.” A robot needs a Semantic Digital Twin that continually updates its recognized situation as objects, locations, relationships, constraints, history, and possible actions.

What Is a VLM?

A VLM is an AI model that works jointly with visual information—such as images and video—and natural language.

Traditional image-recognition models primarily aim to return labels such as “person,” “tool,” or “bolt.” A VLM, by contrast, transforms images into a representation that a language model can use and connects it to questions or instructions. This lets it answer questions such as:

• What is shown in the scene?
• Which object is above, left of, or behind another?
• At what stage is the assembly process?
• Is the object to be grasped within reach?
• Has an anomaly occurred?
• What should be checked next?

Conceptually, a VLM works through the following flow:

Images, video, and sensor observations
        ↓
Vision encoder: extracts features of shapes, objects, and relationships
        ↓
A representation that connects vision and language
        ↓
Language model: integrates questions, instructions, and context
                to generate answers or proposals

In robotics, VLMs can take at least three distinct roles.

It is more realistic to position a VLM as a perception-and-reasoning layer that handles meaning than to regard it as the robot controller itself. Executing an action still requires geometry, kinematics, collision checking, control, and safety boundaries.

What Does RoboVista Measure?

RoboVista’s defining feature is that it converts actual decision points in robot systems into the shared format of VQA. The research team calls this Robot Question Answering (RQA).

It divides robot operation into four functions:

1. Perception: Estimate objects, shapes, spatial relationships, contact states, and related conditions.
2. High-level Decision Making: Determine objectives, sequences, and next actions.
3. Motion / Action Awareness: Judge reachability, collisions, and the physical feasibility of an action.
4. Failure Recovery: Detect failures and choose corrective actions.

Each question includes a robot-visible or onboard-camera image, task context, five answer options, a correct answer, and an expert rationale. The questions are based on real robotic tasks drawn from 18 peer-reviewed studies and datasets, including bin picking, assembly, weed removal, cable routing, surgical knot tying, and navigation.

What is being measured is not general image-quiz ability. It is whether a robot can determine “what it is recognizing now, what it is permitted to do, and what it should not do.” The paper reports that RoboVista scores correlate strongly with error and success measures in physical tasks such as bimanual gripper alignment and VLM-assisted surgical knot tying.

More Thinking Is Not Always Better

One of RoboVista’s most interesting findings concerns Chain-of-Thought (CoT) prompting—the instruction to “think step by step before answering.”

CoT can help with questions involving multi-step plans and constraints. Yet for low-level scene understanding, it lowered accuracy for many models. The paper reports cases in which CoT reduced scene-understanding accuracy by up to roughly 12 percentage points.

This may seem counterintuitive, but it is natural in robotics.

For questions such as “Is the gripper securely holding the needle in the image?” or “Is the target to the right or left?” what is first required is accurate visual identification. If a model shifts too much attention toward linguistic plausibility or general knowledge, it may drift away from visual evidence such as fine position, occlusion, contact, and orientation.

By contrast, in planning tasks such as surgery and agriculture—where multiple steps, goals, and future states must be considered—explicit intermediate reasoning can be helpful.

The question in integrating VLMs into robots is therefore not, “Should we make the model think longer?”

Should this decision prioritize perception, or should it prioritize planning?

Perception, planning, execution, and recovery should not be mixed into a single prompt. Their responsibilities should be separated, and the output of each should be validated.

The Central Challenge Is Still Perception of the Physical World

RoboVista’s failure analysis shows that many failures begin with visual perception rather than logical reasoning. In an analysis of Qwen2.5-VL-7B, misidentification—incorrectly recognizing an object’s identity, state, or spatial location—accounted for 143 cases, or 30.2%. Scaling the model reduced this rate to 20.3%, but spatial reasoning involving relative position, depth, motion, and reachability, along with interpretation of task meaning, remained major sources of failure.

This reflects a fundamental gap between an image and the physical world.

An image is an observation, not the world itself. A monocular image leaves depth and scale ambiguous. Objects may be occluded, while lighting and reflections change their appearance. Contact, force, friction, and internal state cannot be fully inferred from images alone. And what a robot needs is not merely “what is visible,” but “where it is, what it relates to, what state it is in, and what is possible next.”

A Semantic Digital Twin has the role of filling this gap.

What Is a Semantic Digital Twin?

A conventional Digital Twin represents real equipment, spaces, products, robots, and other physical assets in a digital environment for monitoring, prediction, simulation, and optimization.

A Semantic Digital Twin does more than replicate geometry and numbers. It maintains, in a machine-readable structure, the meaning of what an entity is; the attributes, relationships, constraints, and roles it has; the state it occupies; and the actions it can perform.

In a robotic task, a Semantic Digital Twin may represent states such as the following.

The important point is that the Twin becomes an “executing world model” updated by observations, rather than a static registry.

Connecting VLMs and Semantic Digital Twins

A VLM is useful as an entry point for extracting meaning from images. But its natural-language answers should not be sent directly as action commands. A system should establish a path such as the following:

Cameras, depth sensing, force sensing, and encoders
              ↓
VLM / specialized perception systems: generate semantic candidates and confidence
              ↓
Semantic Digital Twin: integrate them as states, relationships, and provenance
              ↓
Rules, geometric validation, and motion planning:
assess safety and feasibility
              ↓
Act / Ask / Stop
              ↓
Record outcomes in the Twin and the Decision Trace

Suppose a VLM judges that “the cable may not be correctly connected to the connector.” This output is not a final decision; it is candidate information for updating the Twin.

1. Record the VLM observation, confidence, image, and time.
2. Cross-check it against a depth camera, electrical continuity, and the work history.
3. Reflect an “unconnected” or “uncertain” state in the Twin.
4. Verify whether reconnection can be performed safely in light of collision and task-sequence constraints.
5. If execution is feasible, propose a limited action; if the state remains ambiguous, ask a person for confirmation.
6. Record the action and outcome as a Trace and use it in subsequent decisions.

This structure prevents VLM misrecognition from being converted unconditionally into physical action. It combines the VLM’s flexible semantic understanding with the structured state, constraints, and history maintained by the Twin.

Research is already emerging that connects Semantic Digital Twins and LLMs for robotic planning, using object affordances, interaction rules, and execution-time failure feedback to update plans.

Grounding Language Models with Semantic Digital Twins for Robotic Planning

We Must Evaluate More Than Accuracy

RoboVista shows that accuracy alone is insufficient for determining whether a VLM can be used in Physical AI. In real operations, at least the following dimensions should be evaluated separately:

• Perceptual accuracy: Can the system correctly identify objects, poses, positions, contact, and anomalies?
• Spatial and geometric consistency: Can it handle distance, depth, reachability, and collision risk?
• Planning validity: Can it choose the next action in accordance with objectives, procedures, and constraints?
• Treatment of uncertainty: When a state is hidden or ambiguous, can it choose Ask or Stop rather than guess?
• Recovery after execution: Can it detect failures and transition to safe corrective action?
• Traceability: Can we later explain which observations and rationale led to an action, and who or what executed it?

RoboVista’s RQA separates perception, decision-making, motion awareness, and failure recovery for evaluation. This aligns with the architecture of a robot runtime centered on a Semantic Digital Twin. The VLM is not placed as an all-purpose decision maker; it becomes a component that provides meaningful signals. Decisions are finalized by referring to the Twin’s state, constraints, authority, and risk.

Conclusion: The Value of VLMs Is That They Can Put the World into Words

RoboVista shows that while today’s VLMs are promising for robotics, a substantial gap remains in their understanding of the physical world.

Increasing model scale can reduce misidentification. But spatial understanding, geometry, contact, affordances, and failure recovery cannot be solved by scale alone. CoT is not universal either; for perception tasks, it can weaken grounding in visual evidence.

That is why VLMs should not stop at being “the eyes of a robot.” They need to be connected to a Semantic Digital Twin that continuously transforms observations into meaning, state, relationships, and constraints.

VLMs can speak about the physical world.
A Semantic Digital Twin transforms that account into a state of the world that can be verified, acted upon, and held accountable.

Physical AI needs more than models that speak more eloquently. It needs a semantic infrastructure that connects what has been seen to safe action.

Chinoba

Chinoba

Intelligence as Relationship

Research Platform

founded by

Masao Watanabe

AI Systems Architecture
Decision Trace
Human–AI Coordination
Algorithmic Governance