Released today in public beta, the feature allows Claude 3.5 Sonnet to perform tasks by viewing screens, moving cursors, and typing text – making it the first frontier AI model to offer such capabilities.
As detailed in Anthropic’s blog post, the technology enables Claude to navigate computers independently, performing complex sequences of actions across different applications and websites. Early adopters include major tech companies like Asana, Canva, and DoorDash, who are already implementing the feature for various automation tasks.
“We’re trying something fundamentally new,’ Anthropic stated in their announcement. “Instead of making specific tools to help Claude complete individual tasks, we’re teaching it general computer skills”
Initial testing shows promising results, with Claude 3.5 Sonnet scoring 14.9% on OSWorld’s screenshot-only category, nearly doubling the next-best AI system’s score of 7.8%. When given more time to complete tasks, the score improved to 22%.
However, Anthropic acknowledges that the technology is still experimental and can be error-prone. Basic actions like scrolling and dragging remain challenging for the AI.
The company says it has implemented safety measures, including new classifiers to identify potential misuse, and partnered with US and UK AI Safety Institutes for pre-deployment testing.
The feature is now available through Anthropic’s API, Amazon Bedrock, and Google Cloud’s Vertex AI, marking a significant step toward more versatile AI systems that can work alongside humans in increasingly natural ways.
Additionally, Anthropic announced Claude 3.5 Haiku, the next generation of its fastest model. The company says this latest low-latency model is particularly strong on coding tasks. Claude 3.5 Haiku will be available later this month through Anthropic’s API, Amazon Bedrock, and Google Cloud’s Vertex AI.
