A day in AI now feels like a month in real life—the pace of progress is simply staggering. While everyone is still celebrating the release of numerous new LLM models and ChatGPT’s new Canva canvas feature, Claude AI has just unveiled another incredible breakthrough: AI can now autonomously control computers.
Give it a task, and the AI will browse relevant information on the screen and perform corresponding actions to complete it, just like a human. It looks truly amazing, and the API has already been released.
AI can already autonomously operate computers.
ChatGPT and Microsoft have both demonstrated AI that can read what’s currently happening on the screen and then provide relevant suggestions and solutions, but the user still has to perform the actions themselves. This time, Claude’s demo is different — the AI operates the mouse and keyboard on its own to complete tasks.
Anthropic stated: “We designed an API that allows Claude to interact with computer interfaces and understand their functions. By integrating this API, developers can enable Claude to turn instructions (such as ‘use my computer and online data to fill out forms’) into actual computer operations (such as viewing spreadsheets, moving the cursor to open a web browser, navigating to relevant websites, and filling out forms using web data).”
To help everyone better understand the whole process, Anthropic has also released a video. The task is to “fill out the supplier application form,” and the required information is scattered across different places, so the AI needs to search for it on its own and fill in the correct form.
This is the prompt he gave: “Please fill out the supplier application form for Ant Equipment Co., using the supplier spreadsheet in Window One, or search the data in the portal tab. List and verify each field item by item, then complete the form in Window Two.”
Then the AI will start executing, and display in real time what it is currently doing. Most importantly, the AI also has thinking capabilities, such as when it discovered that Ant Equipment Co.’s data is not in Google Sheets:
It automatically switches to the CRM and searches for Ant Equipment Co. to obtain the data needed for the form:
You even know to scroll down the page to check:
Once all the data is available, the AI will start filling out the application form on the right, moving the mouse to click on fields, and then entering the information:
Once you’ve finished entering, submit to successfully complete the task:
Although this is a very simple task, for humans it takes just a few minutes to complete, while AI takes longer. But don’t forget, this is only the beginning, just like what we saw with Claude and ChatGPT last year.
However, Claude’s video has been edited, so it’s unclear whether the AI completed the task successfully in one take or if there were failures.
Full video:
Anthropic simultaneously released the new Claude 3.5 Sonnet and Claude 3.5 Haiku.
In addition to AI operating computers, Anthropic also simultaneously released new Claude 3.5 Sonnet and Claude 3.5 HaikuPerformance has improved significantly across all areas. Below are the official test data.
Source: KOCPC Chinese