vibe-coded-mcp-dekstop-control/README.md

MCP Desktop Control

An open-source Model Context Protocol (MCP) server that provides AI agents with native, low-level control over your desktop environment.

Designed specifically with modern Linux (Wayland) in mind, this server bypasses high-level display server restrictions by communicating directly with the Linux kernel via evdev and /dev/uinput to simulate real physical hardware inputs.

Features

• Keyboard Control: Type text with full layout awareness, or trigger common shortcuts (e.g., copy, paste, select_all).
• Layout Support: Currently supports standard US (us) and Slovenian (si) layouts out of the box, with intelligent fallback mechanisms (reading localectl, environment variables, or setxkbmap).
• Mouse Control: Absolute and relative cursor movement, clicking (left, right, middle), and scrolling.
• Configurable Delays: Adjust typing speeds and pre-action delays for smooth interactions.
• Multiple Transports: Run seamlessly over standard stdio or streamable-http.
• (WIP) Screenshots: Stubs are in place for future full-screen and area screenshot support.

---

Requirements

• OS: Linux (Wayland or X11)
• Python: 3.11+
• System: uinput kernel module loaded (standard on almost all distributions).

Installation

This project is managed by Poetry.

1. Clone the repository and navigate into it:

   cd vibe-coded-mcp-desktop-control
   

2. Install the dependencies:

   poetry install
   

This will automatically create the mcp-desktop-control executable in your Poetry environment.

---

The Permission Problem

Because this server simulates real hardware at the kernel level, it requires write access to the /dev/uinput device node. By default, regular users do not have this permission.

You have two choices on how to run this server depending on your workflow.

Option 1: The Native Route (Recommended for background agents like Goose)

Permanently grant your standard user account access to simulate input. This allows agents to seamlessly start the server in the background without getting stuck waiting for a sudo password.

1. Add your user to the input group:

   sudo usermod -aG input $USER
   

2. Create a udev rule to ensure the input group can write to /dev/uinput:

   echo 'KERNEL=="uinput", GROUP="input", MODE="0660"' | sudo tee /etc/udev/rules.d/99-uinput.rules
   sudo udevadm control --reload-rules && sudo udevadm trigger
   

3. Reboot your computer (or log out and back in) for the group changes to take effect.

You can now run the server normally:

poetry run mcp-desktop-control

Option 2: The Sudo Route (Using Streamable HTTP)

If you don't want to change your user groups, you can run the server as root in a separate terminal using sudo. To prevent your AI agent from hanging on a password prompt, you can run the server over Streamable HTTP and have your agent connect to it over the local network.

1. Install dependencies for the root user (only needed once):

   sudo poetry install
   

2. Start the server in a separate terminal:

   sudo poetry run mcp-desktop-control --transport streamable-http --host 127.0.0.1 --port 8000
   

(You can also pass XKB_DEFAULT_LAYOUT=si before the command if you want to force a specific layout).

---

Integrating with Goose

Goose stores extensions in ~/.config/goose/config.yaml (Linux/macOS) or %APPDATA%\Block\goose\config\config.yaml (Windows).

Option A: STDIO (command) extension

Use this when Goose should start the server as a subprocess. The most reliable form is to give the absolute path to the poetry binary and the absolute path to the project.

extensions:
  desktop_control:
    enabled: true
    name: desktop_control
    type: stdio
    cmd: /usr/bin/poetry
    args:
      - run
      - mcp-desktop-control
    envs:
      XKB_DEFAULT_LAYOUT: "si"   # Optional

If you have activated the Poetry virtual environment, you can also use just cmd: mcp-desktop-control with no args.

Option B: Streamable HTTP extension

Start the server in a terminal first (this needs /dev/uinput access, so run as root if you have not set up the udev rule):

poetry run mcp-desktop-control --transport streamable-http --host 127.0.0.1 --port 8000

Then point Goose at it with type: streamable_http and uri: (Goose uses uri, not url):

extensions:
  desktop_control:
    enabled: true
    name: desktop_control
    type: streamable_http
    uri: http://localhost:8000/mcp
    timeout: 300
    headers: {}
    envs: {}

The server listens on /mcp for streamable HTTP traffic, so use http://localhost:8000/mcp.

---

Available Tools

Once connected, your AI agent will automatically have access to the following tools:

• keyboard_type: Types a string of text dynamically calculating Shift and AltGr modifiers based on the active layout.
• keyboard_action: Triggers common keyboard shortcuts (copy, paste, select_all, cut, undo).
• mouse_action: Presses, releases, or clicks the mouse (left, right, middle).
• mouse_move_absolute: Moves the cursor to precise X/Y screen coordinates.
• mouse_move_rel: Moves the cursor relative to its current position.
• mouse_scroll: Scrolls the mouse wheel up or down.
• set_pre_action_delay: Adjusts the sleep timer before actions are executed.
• screenshot_full / screenshot_area: (Coming soon)

Development

Run the test suite with pytest:

poetry run pytest

The Linux backend tests mock evdev.UInput so they can run without /dev/uinput access.

Future Roadmap

1. Windows Support: Abstract the InputBackend to implement PyAutoGUI or ctypes for native Windows support.
2. macOS Support: CoreGraphics backend implementation.
3. Screenshots: Implement screenshot capture returning base64 image data for visual agents.