Evaluating AI in the Physical World

The Story

A product team at an enterprise services company watches a demo. The AI agent navigates a web application, fills out forms, clicks through a multi-step workflow, and completes an expense report. The demo is polished. The clicks are precise. The agent handles drop-downs, date pickers, and confirmation dialogs. The VP of Operations asks: "When can we deploy this for our 3,000 field service technicians?"

The team builds a pilot. On the first day, the agent struggles. The field service application uses a dense, information-heavy interface designed for 1920x1080 monitors -- small radio buttons, tightly packed table rows, custom date selectors, and multi-tabbed forms where the relevant field is three tabs and two scrolls away from the starting view. The agent misclicks. It selects the wrong row in a table because adjacent rows are 18 pixels apart. It clicks a "Submit" button that looks visually identical to a "Save Draft" button but has permanent consequences.

ScreenSpot-Pro was built because this gap between demos and professional reality is systematic. The benchmark uses authentic high-resolution images from 23 applications across five industries and three operating systems -- the kind of screens enterprise workers actually use. The result: the best existing GUI grounding model achieved only 18.9% accuracy. Professional screens with small targets, dense layouts, and domain-specific applications broke models that looked competent on simpler benchmarks.

A second team builds a customer onboarding agent. It navigates their CRM, opens new accounts, fills in customer details, sets up billing configurations, and sends welcome emails. The team measures task completion. After a month of iteration, the agent completes 72% of onboarding tasks end-to-end. The team celebrates.

Then someone times it. The agent takes 14 minutes on a task that a trained onboarding specialist completes in 3 minutes. It uses 47 actions where the human uses 19. It opens and closes the same settings panel three times because its planning loop reconsiders the approach mid-execution. It scrolls past the right field, overshoots, scrolls back, and then clicks. The completion score says "pass." The operational reality says "no human would tolerate this."

OSWorld-Human quantified this problem. Current agents take tens of minutes on tasks humans complete in minutes, large model calls for planning and reflection dominate the latency, and even the highest-scoring agents use 1.4 to 2.7 times more actions than necessary. Task completion alone doesn't capture whether the agent is usable. Time, step count, and action efficiency are first-class evaluation dimensions, not footnotes.

A third team deploys a browser-based research agent. It searches the web, reads articles, extracts data, and compiles reports. The quality is strong. Then an internal security review discovers something the eval suite didn't test for: a webpage the agent visited during a routine research task contained hidden instructions -- text styled to be invisible to human readers but visible to the model's screenshot parser. The instructions told the agent to navigate to an external URL and paste the contents of the current session into a form field. The agent complied. It followed the injected instructions because, from its perspective, the text was part of the page content it was processing.

Three teams, three different failures, one shared lesson: once an agent stops generating text and starts interacting with an environment -- clicking, scrolling, navigating, taking actions that change state -- the evaluation model from the first thirty topics of this series is necessary but not sufficient. The agent needs to perceive, decide, act, and recover. That's a different kind of measurement.

The Core Idea

Every topic from T1 to T32 has evaluated AI by looking at what it says. Does the text answer the question correctly? Does the rubric score pass? Does the judge agree with human raters? Even agent evaluation in T16 -- with its trajectory metrics, tool correctness, and step efficiency -- still evaluates through the lens of API calls and structured outputs. The agent calls a function. The function returns a result. The evaluator checks whether the right function was called with the right arguments.

T33 is where that model breaks. Computer-use agents don't call functions. They look at screenshots, identify elements, move a cursor, and click. They navigate through real applications the way a human would -- visually, spatially, sequentially. They encounter pop-ups they didn't expect, loading states that change the layout, scroll positions that hide the target, and irreversible buttons that look identical to reversible ones. The evaluation can no longer just check the output. It has to check the entire interaction path through a visual, stateful, partially observable environment.

This is not yet robotics. Most public 2025-2026 work is computer-use evaluation: screenshots, mouse clicks, keyboard actions, browser state, desktop workflows. But the evaluation problems are the same problems that physical-world AI faces: perception under constraints, action sequencing, environment drift, state verification, latency, and partial observability. Once the evaluator must score not just text but world interaction, it has crossed an important boundary -- whether the world is made of pixels or atoms.

Think of it like forklift certification. A written test proves you know the rules -- load limits, right-of-way, stacking heights. But nobody certifies a forklift operator based on the written test alone. You watch them drive. Can they perceive the environment? Maneuver in a tight warehouse aisle? Avoid obstacles they didn't expect? Complete the job without damaging inventory? Computer-use evaluation is the same: knowing the right answer is not the same as being able to execute it inside a real environment under real constraints.

The Three-Layer Framework

The technical framework that's emerging splits evaluation into three layers, and collapsing them into a single score is exactly the kind of measurement error that hides the most important failures.

Layer 1: Grounding -- can it find the right element? Grounding asks whether the model can identify the correct on-screen element from a natural-language instruction. "Click the 'Submit Expense' button." "Select the third row in the vendor table." This sounds simple. On professional screens, it's not. ScreenSpot-Pro tests grounding on authentic high-resolution images from professional domains -- medical imaging software, financial dashboards, CAD tools, IDE environments. The result: 18.9% for the best existing model. Microsoft's UI-I2E-Bench adds another dimension: when instructions are implicit rather than explicit, performance drops further because the agent must infer the mapping between intent and interface element.

Layer 2: Task completion -- can it finish the workflow? Grounding is necessary but not sufficient. An agent can point at every element correctly and still fail the task if the sequence is wrong, if hidden state changes between steps, if a pop-up interrupts the flow, or if the agent doesn't recognize when it's stuck. OSWorld remains the flagship task-completion benchmark: a real-computer environment spanning operating systems, arbitrary applications, and multi-application workflows. The original paper reported that humans completed 72.36% of tasks while the best model achieved 12.24%. OpenAI's Operator system card later reported 38.1% on OSWorld -- real progress, but with the explicit caveat that performance is not yet highly reliable for OS automation.

Layer 3: Operational efficiency -- is the successful run actually usable? This is the layer most teams ignore. A task completed in 14 minutes that a human finishes in 3 is not a viable product feature. OSWorld-Human shows that large model calls for planning and reflection are the dominant latency bottleneck. The architecture that makes agents smarter (multi-step reasoning, self-correction, reflection loops) is also what makes them slow. This creates a fundamental tension: the features that improve task completion make operational efficiency worse. Both dimensions must be evaluated simultaneously, not traded off silently.

The Environment Is The Adversary

Topic 28 established that agents with tool access face an expanded adversarial surface -- attacks arrive through retrieved content, not just user prompts. T33 makes that surface dramatically larger. A computer-use agent perceives every pixel on screen. Every webpage, every email body rendered in a browser, every document opened in an application is a potential injection vector.

Anthropic's computer-use documentation is explicit: the model may follow instructions found in webpages or images even when they conflict with the user's request. OpenAI's computer-use guide says page content should be treated as untrusted input. Both recommend isolated environments -- VMs, containers, sandboxed browsers -- with explicit scoping of which sites and actions are allowed.

This means the security boundary is part of the evaluation setup, not separate from it. You cannot meaningfully evaluate a browser agent in a fully open environment and then add security constraints after the fact. The constraints -- domain allowlists, action scoping, privilege minimization, human confirmation for irreversible actions -- must be present during evaluation because they affect the agent's behavior. An agent evaluated without constraints will score differently than one evaluated with them.

Legacy Software: The Enterprise Opportunity

Anthropic's agent eval guide makes a strategically important observation: computer-use agents can operate any GUI, including legacy enterprise software, because they interact through screenshots, mouse, keyboard, and scrolling rather than through APIs. This is exactly why enterprises care -- they have decades of software with no API, no automation hooks, and manual workflows that consume thousands of person-hours per year.

But legacy software also makes evaluation harder. Old interfaces have brittle workflows -- a click in the wrong order triggers an unrecoverable state. UI patterns are inconsistent across modules built in different decades. Business consequences attach to every action: an expense submitted is an expense committed, a ticket escalated is a ticket that pages an engineer at 3 AM, an approval granted is a compliance decision on record. In these settings, "physical world" doesn't mean pixels on a screen. It means invoices, tickets, user accounts, approvals, and compliance states changing in response to agent actions. The evaluation must capture not just task completion but consequence awareness.

Where This Hits In Practice

The team deploying a computer-use agent on professional-grade software. If your target application has dense layouts, small interactive elements, multi-tab workflows, or domain-specific UI patterns, your agent's grounding accuracy will likely be much lower than what demos suggest. ScreenSpot-Pro's 18.9% baseline on professional screens is the reality check. The fix: evaluate grounding separately on screenshots from your actual target application before measuring task completion. If the agent can't reliably find the right element, everything downstream is noise.

The team measuring success rate without measuring efficiency. Your agent completes 70% of tasks. That sounds good until you measure time and step count. OSWorld-Human's finding -- agents taking tens of minutes and 2x+ the necessary actions -- is not an edge case. It's the current state of the art. The fix: add time-to-completion, action count, and actions-per-task-relative-to-human as first-class metrics alongside completion rate. A successful run that takes 5x human time is not a shippable feature.

The team running browser agents without an adversarial evaluation. If your agent navigates web pages, every page it visits is a potential injection surface. Anthropic and OpenAI both recommend isolated environments and treating page content as untrusted. The fix: include adversarial web content in your eval suite -- pages with hidden instructions, pop-ups that try to redirect the agent, elements that mimic legitimate UI but link to unauthorized actions.

The enterprise evaluating computer-use agents for legacy system automation. Legacy applications with no API are the highest-value target and the hardest evaluation challenge. Actions have business consequences. UI patterns are inconsistent. State is fragile. The fix: build evaluation scenarios that include the irreversible actions, the brittle state transitions, and the consequence chain. A sandboxed evaluation that only tests the happy path will produce an agent that fails exactly when the stakes are highest.

Connecting The Dots

From T16 (Evaluating Agents): T16 introduced trajectory evaluation -- measuring not just the destination but the path. T33 extends trajectory evaluation into visual, spatial, stateful environments. The metrics are cousins: T16's tool correctness becomes grounding accuracy, T16's step efficiency becomes OSWorld-Human's action ratio. The evaluation logic carries forward. The observation space gets vastly richer.

From T17 (Trace Debugging): Traces become even more important here because the failure is often not in the final output but in the interaction path: the wrong click, the wasted scroll, the unsafe action, the missed pop-up. T17 taught trace debugging for API-call agents. T33 makes traces spatial and environmental -- each span isn't just "called function X with arguments Y" but "perceived screenshot, identified element at coordinates (x, y), clicked, observed new state."

From T28 (Red Teaming): GUI and browser agents dramatically expand the adversarial surface from T28. Anthropic warns that webpage or image content can override the model's intended behavior. OpenAI says page content should be treated as untrusted. Evaluating physical-world interaction without adversarial testing is like certifying an agent as safe after testing it only in friendly environments.

From T30 (Limits of Evaluation): T33 is a concrete example of T30's limits in action. Benchmarks that looked good turned out to be too easy. Success rates hid unusable latency. Sandbox performance didn't predict open-world competence. T30's lesson that evals are maps, not territory, applies with special force when the territory is a real computer screen.

Remember This

For thirty-two topics, evaluation has been about what the system says. T33 is about what the system does -- in a visual, stateful, partially observable environment where actions have consequences and mistakes can be irreversible.

The three-layer framework matters: grounding (can it find the right element?), task completion (can it finish the workflow?), and operational efficiency (is the successful run actually usable?). Each layer can look good while the others fail. An agent with perfect grounding can fail task completion because of sequencing errors. An agent with strong task completion can be operationally useless because it takes five times longer than a human. A single score that collapses all three hides the diagnostic signal that tells you what to fix.

The security boundary is part of the evaluation, not separate from it. Computer-use agents perceive every pixel on screen. Every webpage, every document, every image is a potential injection vector. Evaluating without adversarial content in the environment is like certifying a car as safe after testing it only on empty roads.

The demo gap is real and systematic. Professional screens break models that look impressive on clean interfaces. The fix is unglamorous: evaluate on screenshots from your actual target application, measure time and action count alongside completion, and include the messy, dense, domain-specific screens that your users actually work with. Reality doesn't look like a demo. Your evaluation shouldn't either.