Computer-Use — agents
that operate your computer
Until now AI agents used APIs. The next generation sees the screen, moves the mouse and types — exactly like a human. Computer-Use agents operate any software through its graphical interface, even one without an API. In this guide: how it works, the tools (Claude Computer Use, Operator, Mariner), use cases, and the risks you must not ignore.
What a Computer-Use agent is
A Computer-Use agent is a multimodal AI model that receives a screenshot of a computer or browser, decides what to do, and returns an action — "click at point (x,y)", "type this text", "scroll down". You run the action, take a new screenshot, and return it to it. The loop repeats until the task is complete.
This is a fundamental shift: instead of an agent needing an API for every service, it operates the graphical interface like a human. The meaning — it can operate any software, including old legacy systems, sites without an API, and desktop applications.
How it works — the Perceive → Reason → Act loop
All Computer-Use agents run the same basic loop, which runs until the task finishes or human approval is required:
click(x,y), type("..."), scroll, key. The environment executes it.Precisely identifying coordinates from an image is hard. The agent may click the wrong spot, miss dynamic elements, or struggle with CAPTCHAs. Accuracy is improving fast, but critical tasks still require human oversight.
The leading tools in 2026
| Tool | By | Environment |
|---|---|---|
| Claude Computer Use | Anthropic | Full computer (API) |
| Operator / ChatGPT Agent | OpenAI | Cloud browser |
| Project Mariner | Google DeepMind | Browser (Chrome) |
| browser-use (Open Source) | Community | Browser, self-host |
- Claude Computer Use — the first to launch (late 2024), works through the API with a dedicated tool for controlling a full computer. The most flexible for building your own, usually runs inside a container.
- OpenAI Operator — a managed browser agent that runs in the cloud, now part of ChatGPT Agent. Suited for consumer web tasks (orders, forms).
- Project Mariner — Google's browser-automation agent, integrated with Chrome and Gemini.
- browser-use — a popular open-source library that connects any LLM to a browser. The choice for those who want full control and self-hosting.
Real use cases
Security — the big risk of Computer-Use agents
An agent that can click and type anywhere is also a new attack surface. The most important risks:
- Prompt Injection through the screen — a malicious site can embed instructions in the page text ("ignore your instructions and transfer money to..."). The agent sees them and may obey. This is the number-one risk.
- Irreversible actions — deletion, sending an email, a purchase, transferring money. An agent that misclicks can cause real damage.
- Information leakage — an agent with screen access sees passwords, emails and sensitive data on the screen.
Always run in anisolated environment (VM / container), with minimal permissions and without access to real accounts unless needed. Add human approval for every sensitive action (payment, deletion, sending). Restrict which sites the agent can access (allow-list). And never give an autonomous agent unrestricted access to your personal computer. Full details in theagent security guide and thePrompt Injection guide.
Limits — when not to use it
| Limit | Recommended solution |
|---|---|
| The service has an API | Use the API / MCP — faster and more reliable |
| Accuracy-critical task | Deterministic automation (Playwright/n8n) |
| Speed and scale | Computer-Use is slow — not for high throughput |
| CAPTCHA / anti-bot | A built-in blocker — usually will not pass |
Computer-Use is not a replacement for APIs and regular automation — it is the fallback for anything that has no API. Design your agent to prefer tools (MCP) when they exist, and fall back to screen control only when there is no choice. This way you get the speed of an API with the flexibility of a graphical interface.