Table of Contents
Anthropic just rolled out an update to Claude called Claude 3.5 Sonnet. And the headline feature isn’t another chat trick—it’s something more hands-on: computer use.
In plain terms, Claude can control your computer by watching what’s on your screen and then doing the same kind of actions a human would. Cursor movement, clicking buttons, filling fields, typing text… the whole “I’ll just handle this” vibe.
I tested the idea conceptually (and watched a bunch of demos), and what stood out to me is how different it feels from normal AI. Instead of “tell me what to do,” it’s more like “watch what I’m doing and follow through.” That’s a big shift.
What “Computer Use” Actually Means (And What It Can Do)
Most AI models live in the text world. You type a prompt, it responds. Claude 3.5 Sonnet’s “computer use” feature pushes past that boundary by letting the model interact with your screen.
So what does that look like in day-to-day tasks?
- Moving the cursor to the right spot, not just describing where to click
- Clicking buttons and navigating menus
- Typing into forms and updating fields
- Following multi-step workflows across different screens
Here’s the part I like: it’s not limited to one website or one tool. If the workflow is visible on-screen, Claude can attempt to follow it.
That also means it’s useful for the kind of work that’s repetitive and annoying—stuff you already know how to do, you just don’t want to do it 20 times.
Why This Feature Matters for Real Work
When people talk about AI “automation,” they often mean scripts or one-off integrations. This feels closer to work getting done—because the AI can interact with the actual interface you’re using.
Claude 3.5 Sonnet’s computer use is already being explored by companies like Canva and DoorDash. The common thread isn’t magic—it’s reducing the number of steps humans have to manually perform.
For example, you can imagine Claude helping with:
- Design workflows (drafting layout ideas, finding elements, iterating faster)
- Office tasks like filling out forms, organizing information, and moving through menus without you babysitting every click
- Customer support operations where the “work” is mostly navigation + responding with the right info
In my experience, the time sink in these tasks isn’t the thinking. It’s the clicking, switching tabs, copying/pasting, and double-checking every step. Computer use targets that directly.
How Good Is It, Really? (Let’s Talk Scores)
Even with the hype, this is still experimental. It’s impressive, but it’s not flawless—and I don’t think anyone should treat it like a fully reliable employee yet.
In tests measuring how well an AI can use a computer, Claude scored about 14.9%, while humans scored around 75%. That gap is huge.
But here’s the context that matters: Claude is still doing far better than other models in the same category. For example, GPT-4 scored about 7.7% in those tests.
So yes, the numbers are low. But they’re also trending in the right direction. When you’re comparing “early stage but improving” vs “early stage and struggling,” Claude’s position looks a lot more promising.
Big Benefits: Where Computer Use Can Save Time
If you’re going to use this kind of AI in your workflow, you’ll want to focus on tasks that are:
- repeatable (similar steps every time)
- screen-based (the whole process is visible)
- low-to-medium risk (so mistakes don’t cost you money)
In practice, that often means form filling, document handling, and troubleshooting software steps. The more “click-through” the job is, the more Claude can shine.
And because it can follow sequences, it’s not just “type this.” It can attempt the whole chain: open something, navigate to the right field, enter details, submit, and move on.
Limitations and Risks (This Is Not Fully Human Yet)
I’m optimistic, but I’m also realistic. Computer use introduces a new class of risk: if the AI can click and type, it can also do the wrong thing.
That’s why Anthropic has built in strict controls. For example, Claude isn’t permitted to make purchases or access sensitive data without additional security measures.
Even with those guardrails, you still have to treat it like a junior assistant—not a robot you can walk away from. If the task involves money, credentials, or anything you can’t afford to mess up, you’ll want monitoring and clear boundaries.
Also, interfaces change. Buttons move. Pop-ups appear. Pages load slowly. Those small real-world annoyances are exactly where these systems can stumble, because they don’t “know” your intent—they infer it from what they see.
Early Use Cases: Coding, Design, and Support
Early users are already finding practical applications, especially in areas where workflows are structured:
- Coding help: navigating environments, running through steps, and assisting with repetitive setup
- Design assistance: iterating faster by handling the “mechanics” of edits and navigation
- Customer service management: moving through systems and preparing responses more efficiently
What I noticed in demos is that the AI performs best when the task instructions are clear and the target screens are predictable. Give it a messy, unclear workflow and it’ll spend more time correcting itself.
What Comes Next for AI That Controls Your Computer
The future here is pretty obvious: more capable agents, more autonomy, and fewer handoffs. As Claude’s computer use improves, it could handle larger chunks of work—things like scheduling, coordinating tasks across apps, and executing more complex multi-step processes.
And it’s not just Anthropic. Major tech companies are exploring similar ideas for their own AI systems. That suggests we’ll keep seeing “agent” features expand across products.
So will AI-driven assistants become commonplace? I think they will—but it won’t happen all at once. The adoption curve will depend on reliability, safety controls, and how well these tools handle the messy edge cases of real life.
Watch It in Action
If you want to see what “computer use” looks like when it’s actually running, this video is a solid way to get a feel for it:
Bottom line: Claude 3.5 Sonnet’s computer use is one of the more interesting steps forward in AI I’ve seen lately. It’s not “human-level” yet, but it’s moving in the direction that actually matters—getting work done, not just talking about it.
