Anthropic has introduced a new set of APIs called "computer use" in its Claude 3.5 Sonnet AI model, which allows the AI to interact with computers similarly to a human user. This feature is currently in public beta and enables the AI to perform tasks such as moving a cursor, typing text, and clicking buttons by perceiving and interacting with computer interfaces. Companies like Asana, Canva, and DoorDash are already exploring its potential in their workflows.
Introducing an upgraded Claude 3.5 Sonnet, and a new model, Claude 3.5 Haiku. We’re also introducing a new capability in beta: computer use.
— Anthropic (@AnthropicAI) October 22, 2024
Developers can now direct Claude to use computers the way people do—by looking at a screen, moving a cursor, clicking, and typing text. pic.twitter.com/ZlywNPVIJP
The computer use feature marks a shift from task-specific AI applications to more general-purpose capabilities. It allows developers to automate repetitive processes, build and test software, and conduct open-ended tasks like research. This capability is seen as experimental and can be error-prone, with challenges in performing actions like scrolling or dragging. Despite these limitations, it represents a significant step forward in AI-human interaction.
Even while recording these demos, we encountered some amusing moments. In one, Claude accidentally stopped a long-running screen recording, causing all footage to be lost.
— Anthropic (@AnthropicAI) October 22, 2024
Later, Claude took a break from our coding demo and began to peruse photos of Yellowstone National Park. pic.twitter.com/r6Lrx6XPxZ
Anthropic's approach to this feature involves using an API that lets Claude translate instructions into computer commands, such as filling out forms using data from spreadsheets or web pages. The company emphasizes safety, developing classifiers to detect misuse and mitigate risks like spam or misinformation.
The release of this feature is part of Anthropic's broader strategy to enhance its AI models' capabilities, with the upgraded Claude 3.5 Sonnet showing significant improvements in coding and tool use tasks. The introduction of computer use is expected to evolve rapidly with feedback from developers.
