Computer use

Computer use lets an Arc-managed agent see the screen and click, type, scroll and drag in any app on the Mac or Windows PC where Arc runs. Arc offers it only in Full Access rooms, pauses the agent whenever the operator touches the mouse or keyboard, and never operates security prompts or password fields. Read this page before you use the computer tool, when an action is refused, or when you are choosing between the desktop, the room browser and Arc Desktop's own controls.

Check that computer use is available

• A supported host. The daemon runs on macOS or Windows with Arc's computer helper installed. GET /v1/computer/status reports platform_supported, helper_installed and stop_hotkey.
• A Full Access room. A Safe room offers no computer tool, and switching a room to Safe ends a running session. Full Access is the only grant; there is nothing else to enable.
• A managed seat whose model sees images, on an API connection or a self-hosted model server. External harness agents, Arc-managed GGUF models and models marked text only in the operator's settings get no computer tool.
• macOS permission. The operator allows Megastructure Arc under Accessibility and Screen Recording; Screen control in the right rail opens those panes. Windows asks for no permissions.

Find the computer tool

Claude on an Anthropic connection gets Anthropic's native computer tool; send its actions in a response of their own, not mixed with Arc tool calls. Other API models see Arc's computer tool listed directly. A self-hosted model finds it by calling arc_search_tools with the query computer, then calls it directly.

Send actions

The computer tool takes an actions array. A seat whose model passed Arc's pointing test may send up to 8 actions per call; every other seat sends exactly one. The tool description states your limit, and a larger batch returns batch_too_large without running. Actions run in order and stop at the first failure; later ones report not_executed. Arc attaches a fresh screenshot after each call unless it ended with screenshot or zoom.

computer {
  "actions": [
    {"action": "left_click", "coordinate": [412, 230]},
    {"action": "type", "text": "quarterly report"},
    {"action": "key", "text": "Return"}
  ]
}

With a limit of one, send each of these actions in its own call.

Modifiers are shift, ctrl, alt and super; super is Command on macOS and the Windows key on Windows.

Use the right coordinates

Coordinates refer to your latest screenshot of the main display, scaled to your model's image limits, so take one first. Most models use screenshot pixels. Gemini models use a 0–999 grid over the screenshot (x divided by width, times 1000), which Arc converts to pixels. Arc picks the convention from the model family and states it in the tool description and on every screenshot.

The pointing test asks a model for six clicks on three synthetic app screens in that convention. Five hits allow batches of up to 8; a model that misses, or answers in another convention, sends one action per call. Arc runs the test when the operator tests a connection or refreshes a self-hosted server. The native Anthropic tool always batches.

Share the screen with the operator

One agent drives the desktop at a time. Your session starts with your first action and ends when your turn ends. While you act, a light in Arc's status bar glows and pulses; a hollow ring means paused. It opens Screen control, where the operator watches the live screen and can Pause, Resume or Stop. The room's event feed records each session as computer.session_started, computer.session_paused, computer.session_resumed and computer.session_stopped.

When the operator touches the mouse or keyboard, Arc pauses you at once. Your next action waits up to 30 seconds for the operator to stay idle for 5 seconds with no modifier held. Then a screenshot runs as asked; any other action does not run and returns the current screen instead. The stop key (⌃⌥⌘. on macOS, Ctrl+Alt+Shift+. on Windows), the Stop button, or a switch to Safe ends the session for the rest of your turn. Arc keeps session screenshots on this machine for about a week.

Know what Arc refuses

Arc never operates security and permission surfaces: authentication and privacy prompts, Gatekeeper and the login window on macOS; User Account Control, credential and Windows Hello prompts, Windows Security, SmartScreen and the lock screen on Windows. It never types into a password field, and never presses the stop key or a shortcut that logs out, locks the computer, or opens Force Quit, the Ctrl+Alt+Delete screen or Task Manager (or Alt+F4 on the Windows desktop). Hand those steps to the operator.

On Windows, Arc refuses windows that run as administrator. While one is in front, Arc does nothing in any window and returns elevated_foreground; wait, or ask the operator to switch windows. When Arc cannot check whether the focused field is a password field, it stops typing and returns focus_unverified; take a screenshot, click the field and type the rest. Neither counts as a refusal.

Read or control a session over HTTP

Choose the computer, the room browser or Arc Desktop

Prefer the room browser or arc_ui_command when they reach the target: they never move the operator's pointer.

Related

• Managed runtime tools
• Computer HTTP routes
• Browser, previews and visual evidence
• Managed agents, runs and completion
• Vocabulary