The interface between the set and the computer is an AI rather than a human… Interacting with the computer and its applications using a mouse and keyboard is something that can be done with the new version of Anthropic’s Claude Sonnet 3.5. An evolution for better and maybe worse.
Driving our cars, answering mail, taking photos and videos for us… soon, Come onCome on In major areas of our lives, it can replace us, for better and perhaps for worse. For the best, it is better to trust the ethical and virtuous side of the Anthropic firm that produces Cloud Chatbot. Its founder is a former member of OpenAI who left due to the company’s directives. They are banking on AI’s public interest rather than immediate profits, with disastrous consequences. And, paradoxically, for worse, Anthropic has just published an improved version of the Claude 3.5 Sonnet. Its specialty is that it has the ability to understand and communicate with anything. ApplicationApplication Office In other words, AI can replace the “set screen keyboard” interface, ie A wise manA wise man.
Thus, thanks to an API ” Computer use ”, the new model available in open beta can mimic keystrokes, button clicks and mouse gestures. To achieve this result, Anthropic trained its model to see and understand what was happening on the screen. through Then, it uses on-screen software tools to complete tasks. Counts the number of clouds, to move the mouse pointer to the right place and click. PixelsPixels vertical and horizontal. Ultimately, it gives the illusion of a real person sitting in front of them. ComputerComputer.
We can very quickly imagine the benefit of such a system in “teleworking” mode. Practical for pretending you’re in front of the computer while taking a nap. We can also imagine a form of disappearance. With this ability, in front of the computer, Claude is potentially usable Chat GPTChat GPTGemini and maybe even yourself…
More seriously, this autonomous interface is a concern if it is misused. Applied to computers by hackers, it can make it possible to directly exploit application vulnerabilities and this is just one example. Without denying that the new Cloud Sonnet 3.5 can be used for nefarious purposes, Anthropic highlights the benefits that can be gained from using it.
Keyboard and mouse AI
For them, it is the usage that will also determine what can be done with the tool. Using it will increase the level of security accordingly. And in all cases, humans are in control by giving specific cues. They are the ones who direct Claude’s actions. This can be through examples such as ” I use my computer and online data to fill out this form. “, according to an interview with an Anthropic manager conducted by the media Tech Crunch. For example, Claude was tested for his ability to book a flight or modify his reservation. They succeeded in only half of the cases. That said, it is not uncommon for a person to encounter difficulties with the online reservation interface. But this is just the beginning…
If this is the first job for Claude, to surf the commands on the computer as a user InternetInternet For example, I am not a noveltyUniverseUniverse Some specific processes on AIs computer can already be run through a GPT-4o based interface, for example. But Anthropic’s model will be more powerful and robust. He can also correct himself when he encounters obstacles by trying to complete a task in a different way. He also knows how to achieve goals that require hundreds of steps.
However, for the moment, Anthropic admits that the new version of Claude has some issues with basic operations such as scrolling and zooming, and may also miss certain actions and notifications due to its analysis system. ScreenshotsScreenshots. In short, if the principle exists, it is unlikely to be exploited in a beneficial way, and even less so when it comes to malicious behavior. For security reasons, for the moment, restrictions prevent Claude from publishing on Social networksSocial networksCreate online accounts and interact with official websites. Despite their desire to minimize risks, these restrictions show a certain enthusiasm on the part of Anthropic regarding the use of this interface.