vibe-coded-mcp-dekstop-control/README.md

# MCP Desktop Control

An open-source Model Context Protocol (MCP) server that provides AI agents with native, low-level control over your desktop environment.

Designed specifically with modern Linux (Wayland) in mind, this server bypasses high-level display server restrictions by communicating directly with the Linux kernel via `evdev` and `/dev/uinput` to simulate real physical hardware inputs.

## Features

- **Keyboard Control**: Type text with full layout awareness, or trigger common shortcuts (e.g., `copy`, `paste`, `select_all`).
- **Layout Support**: Currently supports standard US (`us`) and Slovenian (`si`) layouts out of the box, with intelligent fallback mechanisms (reading `localectl`, environment variables, or `setxkbmap`).
- **Mouse Control**: Absolute and relative cursor movement, clicking (left, right, middle), and scrolling.
- **Configurable Delays**: Adjust typing speeds and pre-action delays for smooth interactions.
- **Multiple Transports**: Run seamlessly over standard `stdio` or `streamable-http`.
- **(WIP) Screenshots**: Stubs are in place for future full-screen and area screenshot support.

---

## Requirements

- **OS:** Linux (Wayland or X11)
- **Python:** 3.11+
- **System:** `uinput` kernel module loaded (standard on almost all distributions).

## Installation

This project is managed by [Poetry](https://python-poetry.org/).

1. Clone the repository and navigate into it:
   ```bash
   cd vibe-coded-mcp-desktop-control
   ```

2. Install the dependencies:
   ```bash
   poetry install
   ```

This will automatically create the `mcp-desktop-control` executable in your Poetry environment.

---

## The Permission Problem

Because this server simulates real hardware at the kernel level, it requires write access to the `/dev/uinput` device node. By default, regular users do not have this permission.

You have two choices on how to run this server depending on your workflow.

### Option 1: The Native Route (Recommended for background agents like Goose)
Permanently grant your standard user account access to simulate input. This allows agents to seamlessly start the server in the background without getting stuck waiting for a `sudo` password.

1. Add your user to the `input` group:
   ```bash
   sudo usermod -aG input $USER
   ```
2. Create a `udev` rule to ensure the input group can write to `/dev/uinput`:
   ```bash
   echo 'KERNEL=="uinput", GROUP="input", MODE="0660"' | sudo tee /etc/udev/rules.d/99-uinput.rules
   sudo udevadm control --reload-rules && sudo udevadm trigger
   ```
3. **Reboot your computer** (or log out and back in) for the group changes to take effect.

You can now run the server normally:
```bash
poetry run mcp-desktop-control
```

### Option 2: The Sudo Route (Using Streamable HTTP)
If you don't want to change your user groups, you can run the server as `root` in a separate terminal using `sudo`. To prevent your AI agent from hanging on a password prompt, you can run the server over **Streamable HTTP** and have your agent connect to it over the local network.

1. Install dependencies for the root user (only needed once):
   ```bash
   sudo poetry install
   ```
2. Start the server in a separate terminal:
   ```bash
   sudo poetry run mcp-desktop-control --transport streamable-http --host 127.0.0.1 --port 8000
   ```

*(You can also pass `XKB_DEFAULT_LAYOUT=si` before the command if you want to force a specific layout).*

---

## Integrating with Goose

Goose stores extensions in `~/.config/goose/config.yaml` (Linux/macOS) or `%APPDATA%\Block\goose\config\config.yaml` (Windows).

### Option A: STDIO (command) extension

Use this when Goose should start the server as a subprocess. The most reliable form is to give the absolute path to the `poetry` binary and the absolute path to the project.

```yaml
extensions:
  desktop_control:
    enabled: true
    name: desktop_control
    type: stdio
    cmd: /usr/bin/poetry
    args:
      - run
      - mcp-desktop-control
    envs:
      XKB_DEFAULT_LAYOUT: "si"   # Optional
```

If you have activated the Poetry virtual environment, you can also use just `cmd: mcp-desktop-control` with no `args`.

### Option B: Streamable HTTP extension

Start the server in a terminal first (this needs `/dev/uinput` access, so run as root if you have not set up the udev rule):

```bash
poetry run mcp-desktop-control --transport streamable-http --host 127.0.0.1 --port 8000
```

Then point Goose at it with `type: streamable_http` and `uri:` (Goose uses `uri`, not `url`):

```yaml
extensions:
  desktop_control:
    enabled: true
    name: desktop_control
    type: streamable_http
    uri: http://localhost:8000/mcp
    timeout: 300
    headers: {}
    envs: {}
```

The server listens on `/mcp` for streamable HTTP traffic, so use `http://localhost:8000/mcp`.

---

## Available Tools

Once connected, your AI agent will automatically have access to the following tools:

- `keyboard_type`: Types a string of text dynamically calculating `Shift` and `AltGr` modifiers based on the active layout.
- `keyboard_action`: Triggers common keyboard shortcuts (`copy`, `paste`, `select_all`, `cut`, `undo`).
- `mouse_action`: Presses, releases, or clicks the mouse (`left`, `right`, `middle`).
- `mouse_move_absolute`: Moves the cursor to precise X/Y screen coordinates.
- `mouse_move_rel`: Moves the cursor relative to its current position.
- `mouse_scroll`: Scrolls the mouse wheel up or down.
- `set_pre_action_delay`: Adjusts the sleep timer before actions are executed.
- `screenshot_full` / `screenshot_area`: *(Coming soon)*

## Development

Run the test suite with pytest:

```bash
poetry run pytest
```

The Linux backend tests mock `evdev.UInput` so they can run without `/dev/uinput` access.

## Future Roadmap

1. **Windows Support**: Abstract the `InputBackend` to implement `PyAutoGUI` or `ctypes` for native Windows support.
2. **macOS Support**: CoreGraphics backend implementation.
3. **Screenshots**: Implement screenshot capture returning base64 image data for visual agents.