Google opens its Gemini tool that lets AI click and type on a computer to all developers
The AI looks at screenshots and sends back mouse clicks and keystrokes.
Google made its Gemini Computer Use tool generally available on Oct. 30, meaning it is out of testing and open to developers who build apps on its Gemini AI models, according to Google's developer notes.
Most AI chatbots, like the Gemini and ChatGPT apps, answer in text. Computer Use lets an AI operate a web browser, phone or desktop computer. The model looks at a screenshot, then sends back actions such as a mouse click or typed keys, Google says. The developer's own software carries out each step and sends a new screenshot.
Software usually talks to other software through APIs, behind-the-scenes connections built for programs. Computer Use instead works through the same buttons and text boxes a person sees on screen.
Google suggests uses such as filling out web forms, testing websites and gathering prices and reviews from shopping sites. Each action comes with a note explaining the model's reasoning. Developers can also turn on screening for hidden instructions planted in web pages to trick the AI, Google says.
Source: Google, Gemini API changelog and Computer Use documentation.