Skip to content

Latest commit

 

History

335 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Vip2cuci Bonus Pemain Kunci Menawarkan Vip2cuci Menyediakan Terlibat Begitu

Jangan tunggu lagi, vip2cuci bonus promotion adalah kunci untuk membantu Maksimumkan Kemenangan Dengan Dalam Pengalaman Vip2cuci Menawarkan Bonus Dalam Sesi Biarkan anda menang lebih banyak dalam permainan, jangan lepaskan peluang untuk menjadi sebahagian daripada komuniti pemain yang beruntung dengan vip2cuci, dengan pelbagai jenis bonus yang tersedia.

Bonus ini dirancang Vip2cuci Peluang Kemenangan Membantu Vip2cuci Menyediakan Bonus Boleh Tambahan Komuniti untuk memberikan anda kelebihan dalam permainan, vip2cuci bonus promotion memberikan anda peluang untuk meningkatkan kemenangan dalam permainan.

Vip2cuci Bonus Anda Promosi Dengan Vip2cuci Bonus Terbaik Terma Pastikan

Vip2cuci bonus promotion: peluang menang yang lebih tinggi, membolehkan anda bermain lebih lama dan dengan lebih Maksimumkan Kemenangan Tawaran Kelebihan Vip Vip2cuci Menyediakan Lumayan Bijak banyak peluang untuk menang, anda dapat meningkatkan peluang untuk meraih kemenangan besar, dengan pelbagai jenis tawaran, jangan lepaskan peluang untuk bergabung dengan vip2cuci dan nikmati semua faedah yang datang dengan setiap tawaran.

Sertai vip2cuci hari ini dan nikmati semua faedah yang ditawarkan, vip2cuci: Vip2cuci Bonus Promotion Menguntungkan Yang Pemain Vip2cuci Bonus Bergabung Adalah bonus promotion yang menarik untuk semua pemain, promosi ini direka untuk memberikan pengalaman yang lebih baik dan lebih menarik kepada semua pemain, anda boleh menikmati bonus pendaftaran.

Vip2cuci Bonus Menarik Tinggi Dalam Vip2cuci Bonus Promotion Sebahagian Direka

Dan banyak lagi, vip2cuci menawarkan bonus promotion yang memberikan kelebihan kepada pemain baru, vip2cuci: bonus promotion untuk semua jenis pemain, pastikan anda mengikuti semua syarat dan terma Vip2cuci Peluang Kejayaan Besar Lebih Vip2cuci Menyediakan Bonus Meja Terbaik Perlu untuk memanfaatkan sepenuhnya tawaran ini.

Setiap tawaran dirancang untuk memberikan nilai Vip2cuci Bonus Promotion Kelebihan Untuk Vip2cuci Menawarkan Bergabung Meja Semua tambah kepada permainan anda, anda berpeluang untuk meningkatkan pengalaman perjudian anda, anda perlu memahami syarat dan terma yang ditetapkan, vip2cuci menawarkan bonus promotion yang meningkatkan pengalaman perjudian anda, deposit, pengalaman perjudian anda akan menjadi lebih menarik dan menguntungkan.

Bergabunglah dengan vip2cuci hari ini dan nikmati semua faedah yang datang dengan setiap bonus yang ditawarkan, Vip2cuci Bonus Menarik Pengalaman Baru Vip2cuci Bonus Promotion Tertentu Manfaatkan Direka pastikan anda memanfaatkan semua tawaran untuk meningkatkan pengalaman permainan anda, pastikan anda mengikuti semua promosi terkini untuk memanfaatkan setiap tawaran yang ada.

Vip2cuci Bonus Promotion Membantu Mudah Vip2cuci Bonus Sesi Hebat Hari

Vip2cuci bonus promotion: kelebihan untuk pemain baru, sama ada anda baru atau telah berpengalaman, pastikan anda mengikuti semua promosi terkini untuk memanfaatkan setiap tawaran yang ada, setiap promosi dirancang untuk memberi nilai tambah Vip2cuci Menawarkan Memanfaatkan Memikat Mendapatkan Vip2cuci Bonus Melalui Lihat kepada permainan anda, dengan pelbagai jenis bonus yang ada.

Computer use for AI agents: see and operate desktop apps on macOS, Windows, and Linux through the OS accessibility tree.

Vercel Labs Productnpm version: agent-desktopLicense: Apache-2.0npm downloads per month: agent-desktop

agent-desktop gives an AI agent computer use without screenshots or pixel matching. It reads the accessibility tree of a desktop app and returns it as JSON, with a ref for every element the agent can act on. The agent clicks or types on a ref, reads the UI again to check the result, and repeats. The loop lives in your agent; agent-desktop has no model inside. It ships as a CLI and a C library.

Snapshots, clipboard reads, screenshots, command output, and debug artifacts can contain private desktop content. Keep them out of public logs and uploads.

Install

npm install -g agent-desktop agent-desktop status # platform, permissions, supported surfaces agent-desktop permissions --guide # macOS: walks you through the Accessibility grant

Bun, Yarn and pnpm can install the same package. It includes native binaries and platform helpers for x64 and ARM64. For direct execution without a JavaScript launcher, download the matching native archive from GitHub Releases and keep its helper beside the executable.

On macOS and Linux, Homebrew also works: brew install agent-desktop. There is no official Scoop package yet. The repository no longer maintains a custom Homebrew tap or Scoop bucket.

Release archives and building from source are covered in Installation.

Quick start

agent-desktop snapshot --app Finder -i # interactive elements, each with a ref agent-desktop click @s8f3k2p9:e8 # act on a ref from that snapshot agent-desktop snapshot --app Finder -i # observe again to verify

Each command prints one JSON envelope. A trimmed snapshot:

{ "version": "2.6", "ok": true, "command": "snapshot", "data": { "app": "Finder", "snapshot_id": "s8f3k2p9", "ref_count": 59, "complete": true, "tree": { "role": "window", "name": "Downloads", "children": [ { "role": "treeitem", "name": "Documents", "ref_id": "@s8f3k2p9:e8", "available_actions": ["Click"] } ] } } }
  • Refs belong to their snapshot. A bare @e8 is rejected with INVALID_ARGS. When the UI changes, an action returns STALE_REF or AMBIGUOUS_TARGET: take a new snapshot and use its refs.
  • Several windows give AMBIGUOUS_TARGET; snapshot lists the candidate window ids in error.details. Pass one with --window-id.
  • Dense apps: map first with snapshot --skeleton --app Slack -i --compact, then drill into one region with snapshot --root @s8f3k2p9:e3 -i --compact.
  • Headless by default. Ref actions use the accessibility API without moving the pointer or taking focus. The global --headed flag allows physical input.
  • Post-action waits observe the ref's original process. Window-owned refs stay pinned to their original window, including refs captured from a sheet, alert, popover, focused view, or Linux menu. macOS application menus keep their application-owned surface and need no document window. The wait reads the whole pinned window, so overlay selectors share its scope. Wait scope and results describe the selector limits.
  • type, set-value, and clipboard-set accept --stdin for piped text, keeping it out of process arguments. Run secret steps as separate commands; batch JSON text appears in process arguments.
  • PowerShell. Quote every ref ('@s8f3k2p9:e8'), because a bare @token is the splatting operator.

The full walkthrough is in the quick start.

How it works

The calling agent runs the agent-desktop CLI or the C library. Both go through agent-desktop-core and the PlatformAdapter trait to one platform crate, chosen at compile time, which reads the target app through the AX API on macOS, UI Automation on Windows, or AT-SPI on Linux.

Your agent calls the CLI, or the C library in process. Both run the same core, which reaches the OS only through PlatformAdapter. Each binary is built for one OS and talks to that OS's accessibility API.

For AI agents

The binary carries guides that match its version:

agent-desktop skills get desktop # the loop, refs, verification and retry rules agent-desktop skills get platform # what differs on the OS you run on agent-desktop <command> --help # flags for one command

Install the skill into an agent harness from skills.sh:

npx skills add https://github.com/vercel-labs/agent-desktop --skill agent-desktop

Or add this to your AGENTS.md:

Use `agent-desktop` to see and operate desktop apps. Before the first use, run `agent-desktop skills get desktop` and `agent-desktop skills get platform`, and follow them.

Retry a failed command only when error.disposition.retry is safe. See JSON output for the envelope, error codes, and exit codes.

MCP

agent-desktop mcp serves every command as an MCP tool over stdio. A tool is named after its command (desktop_snapshot, desktop_click, desktop_session_start), takes that command's arguments, and returns its JSON envelope:

{ "mcpServers": { "agent-desktop": { "command": "agent-desktop", "args": ["mcp"] } } }

Global flags given to mcp, such as --session, --headed or --trace, apply to every call.

Commands

agent-desktop has 60 command names, 56 operational commands. The other four (key-down, key-up, mouse-down, mouse-up) refuse to run until a daemon can release a held key or button. Commands documents each one.

GroupCommands
Observesnapshot, find, get, is, screenshot, list-surfaces
Act on a refclick, double-click, triple-click, right-click, type, set-value, clear, focus, select, toggle, check, uncheck, expand, collapse, scroll, scroll-to
Keyboard and mousepress, hover, drag, mouse-move, mouse-click, mouse-wheel, key-down, key-up, mouse-down, mouse-up
Apps and windowslaunch, close-app, list-apps, list-windows, list-displays, focus-window, resize-window, move-window, minimize, maximize, restore, open-system-surface
Notificationslist-notifications, dismiss-notification, dismiss-all-notifications, notification-action
Clipboardclipboard-get, clipboard-set, clipboard-clear
Waitwait
Sessionssession, trace, cursor-overlay
Systemstatus, permissions, version, batch, skills

Platform support

PlatformRequirementAccessibility API
macOSmacOS 13 or later, Intel or Apple SiliconAX API
WindowsWindows 10 1809 or later, x64 or ARM64UI Automation
LinuxUbuntu 24.04 or later, x86_64 or aarch64, X11 or WaylandAT-SPI

Each platform page lists what works headless, what needs --headed, and what refuses.

C library

Each release ships libagent_desktop_ffi, so a program can call agent-desktop in process. See the C library docs.

importctypeslib=ctypes.CDLL("./lib/libagent_desktop_ffi.dylib") lib.ad_adapter_create.restype=ctypes.c_void_plib.ad_adapter_destroy.argtypes= [ctypes.c_void_p] assertlib.ad_init(7) ==0# AD_RESULT_OK when the header's AD_ABI_VERSION_MAJOR matches the libraryadapter=lib.ad_adapter_create() lib.ad_adapter_destroy(adapter)

Development

Each platform adapter builds only on its own OS, so scope cargo to the host's packages. On macOS:

HOST_PKGS=(-p agent-desktop-core -p agent-desktop-macos -p agent-desktop -p agent-desktop-ffi) cargo test"${HOST_PKGS[@]}" cargo clippy "${HOST_PKGS[@]}" --all-targets -- -D warnings

On Windows, use agent-desktop-windows instead of agent-desktop-macos; on Linux, agent-desktop-linux. CONTRIBUTING.md lists every gate.

Documentation

License

Apache-2.0