GitHub has introduced a public preview of computer use capabilities for GitHub Copilot, enabling the AI tool to interact directly with desktop applications across macOS and Windows. Available via the GitHub Copilot CLI and the standalone GitHub Copilot desktop application, the feature expands task automation beyond standard code editors and terminal environments.
GUI Automation Without Custom APIs
The new computer use capability allows Copilot to perform user-interface actions on behalf of the developer. It can read accessible visual context and screen content, enter text, trigger controls, click UI elements, drag components, and manage keystrokes across multi-step software workflows. This capability is targeted at automating tasks within legacy programs and GUI-only software that lack native command-line interfaces, APIs, or Model Context Protocol integrations.
Activation and Administrative Controls
Developers can activate the feature in the GitHub Copilot CLI using the command /computer on, inspect its operational status with /computer show, or disable it using /computer off. Within the GitHub Copilot desktop app, users can turn on computer use by navigating to Settings and enabling the feature.
Safety mechanisms require Copilot to ask for explicit approval before taking control of an application, while users maintain the ability to review or reset permissions for apps set to always allow access. Organizations can disable the computer use functionality at an enterprise settings level. On macOS platforms, the feature requires manual configuration of Screen Recording and Accessibility system permissions.
Supported Workflows and Operating Limitations
According to GitHub, computer use functions most effectively when prompts clearly specify the target outcome, relevant applications, and operational constraints. Suggested use cases include pulling notification summaries from web browsers, updating slides inside presentation software, or transferring structured information across multiple desktop applications.
AI-generated editorial illustration; not a documentary photograph.
Source: GitHub Copilot can now interact with desktop apps with computer use
SIGNAL TITANS
Leave a Reply