Vercel Labs ships agent-desktop: AI agents read desktop accessibility trees and act on refs instead of screenshots
Synopsis
Vercel Labs has released agent-desktop, which lets AI agents operate desktop apps on macOS, Windows and Linux without screenshots or pixel matching. It reads an app's accessibility tree as JSON, assigns a ref to every element the agent can act on, and the agent clicks or types on a ref, then reads the UI again to verify. The tool ships as a CLI and a C library, contains no model, and leaves the observe-act-verify loop in the agent.
Interpretation
AI agents gain a non-visual route to desktop apps: agent-desktop reads the operating system's accessibility tree and returns interface elements with their available actions as JSON, which the agent uses to click or type. agent-desktop explicitly offers computer use without screenshots or pixel matching, and actions land on element refs supplied by the accessibility tree. The tool works on macOS, Windows and Linux through each platform's accessibility API, and a snapshot returns an element tree with refs, such as a Finder Downloads window snapshot listing 59 refs.
The agent receives a snapshot of the interface with refs, and actions target refs rather than screen coordinates. A ref belongs to the snapshot that produced it; after the UI changes, an old ref returns STALE_REF or AMBIGUOUS_TARGET and a new snapshot is needed. It runs headless by default, and ref actions use the accessibility API without moving the pointer or taking focus; only the global --headed flag enables physical input.
The observe-act-observe loop is implemented by the caller, and agent-desktop itself contains no model. The CLI and the C library share one core that reaches the OS only through PlatformAdapter, and each binary targets one platform's accessibility API. Snapshots, clipboard reads, screenshots, command output and debug artifacts can contain private desktop content and should be kept out of public logs and uploads.
Perspective
The tool targets agent developers who need to operate desktop apps on macOS, Windows and Linux, and is offered as both a CLI and a C library with skill guides matching the binary version. The material covers installation, snapshots, ref actions and error handling, but gives no task completion rate or comparison with other computer-use approaches.
Refs are valid only within their snapshot, and actions return STALE_REF or AMBIGUOUS_TARGET after the UI changes; in multi-window cases the window id must be selected from the error details and passed explicitly. Snapshots, clipboard reads, screenshots and debug artifacts can contain private desktop content, and the material asks that they be kept out of public logs and uploads.