Desktop Control is a comprehensive desktop automation skill for OpenClaw that provides programmatic control over mouse, keyboard, screen capture, window management, and clipboard operations. Built on PyAutoGUI, it turns your AI agent into a full desktop automation engine capable of interacting with any application through simulated input and visual recognition.
Key Features
- Complete mouse control including absolute and relative positioning, clicking, dragging, scrolling, and multi-button support
- Keyboard automation with text typing at configurable speed, hotkey combinations, special key presses, and manual key state control
- Screen capture and image recognition using OpenCV template matching to locate UI elements, with full-screen and region-specific screenshots
- Window management for listing, activating, minimizing, and maximizing windows by title, plus position and size queries
- Multi-monitor support with pixel color detection for precise visual state verification
- Built-in safety failsafe — moving the mouse to any corner of the screen immediately aborts all running automation
Use Cases
- Automating repetitive desktop workflows that span multiple applications not accessible through APIs
- Testing desktop applications by simulating realistic user interactions with configurable timing
- Building end-to-end automation pipelines that bridge web-based and desktop-based tools
- Creating assistive automation for accessibility scenarios where programmatic input control is needed
How It Works
The skill exposes a
DesktopController
class that accepts method calls for mouse, keyboard, screen, and window operations. Commands execute sequentially with optional timing parameters that simulate realistic human interaction speeds. Image recognition uses OpenCV template matching to find UI elements on screen, enabling automation that adapts to dynamic layouts rather than relying on fixed coordinates.
Getting Started
Install the required Python packages with
pip install pyautogui pillow opencv-python pygetwindow
, then import the controller with
from skills.desktop_control import DesktopController
. The skill includes an approval-mode with confirmation prompts for sensitive operations and logs all actions for auditability.
