|
| 1 | +# SimulateInput |
| 2 | + |
| 3 | +Cross-platform desktop and browser automation platform for testing your own websites, desktop applications, installers, and system-level UI flows. |
| 4 | + |
| 5 | +## Features |
| 6 | + |
| 7 | +- Window attach, focus, click, drag, type, hotkey, clear text, and screenshot actions |
| 8 | +- Multiple locator strategies: UIA/AX/AT-SPI style lookup, visible text, OCR, image matching, and coordinate fallback |
| 9 | +- CLI, MCP, and YAML case runner interfaces |
| 10 | +- Windows implementation with real smoke-tested execution |
| 11 | +- macOS MVP, Linux X11 MVP, and Linux Wayland compatibility layer |
| 12 | +- Skill docs for AI-driven automation workflows |
| 13 | + |
| 14 | +## Project Layout |
| 15 | + |
| 16 | +- `src/simulateinput/` - core engine, drivers, CLI, MCP server, runner, and locators |
| 17 | +- `docs/automation-platform-design.md` - architecture and implementation plan |
| 18 | +- `docs/cross-platform-installation.md` - platform-specific setup and permissions |
| 19 | +- `skills/simulateinput/` - skill definition and MCP/CLI references |
| 20 | +- `tests/` - unit tests and smoke case YAML files |
| 21 | + |
| 22 | +## Quick Start |
| 23 | + |
| 24 | +```powershell |
| 25 | +$env:PYTHONPATH='src' |
| 26 | +python -m simulateinput.cli.main doctor |
| 27 | +python -m simulateinput.cli.main session start |
| 28 | +python -m simulateinput.cli.main mcp tools |
| 29 | +``` |
| 30 | + |
| 31 | +## Common CLI Flow |
| 32 | + |
| 33 | +```powershell |
| 34 | +$env:PYTHONPATH='src' |
| 35 | +
|
| 36 | +python -m simulateinput.cli.main session start |
| 37 | +python -m simulateinput.cli.main window list --session-id <session_id> |
| 38 | +python -m simulateinput.cli.main window attach --session-id <session_id> --window-id <window_id> |
| 39 | +
|
| 40 | +python -m simulateinput.cli.main locate uia --session-id <session_id> --name "Submit" |
| 41 | +python -m simulateinput.cli.main action click-uia --session-id <session_id> --name "Submit" |
| 42 | +python -m simulateinput.cli.main action screenshot --session-id <session_id> --output artifacts/shot.png |
| 43 | +``` |
| 44 | + |
| 45 | +## YAML Case Runner |
| 46 | + |
| 47 | +```powershell |
| 48 | +$env:PYTHONPATH='src' |
| 49 | +python -m simulateinput.cli.main case run tests/e2e/cases/windows-smoke.yaml |
| 50 | +``` |
| 51 | + |
| 52 | +Example step types: |
| 53 | + |
| 54 | +- `attach_window` |
| 55 | +- `locate_text` |
| 56 | +- `locate_uia` |
| 57 | +- `locate_ocr` |
| 58 | +- `locate_image` |
| 59 | +- `click_text` |
| 60 | +- `click_uia` |
| 61 | +- `click_ocr` |
| 62 | +- `click_image` |
| 63 | +- `click` |
| 64 | +- `drag` |
| 65 | +- `type_text` |
| 66 | +- `press_key` |
| 67 | +- `hotkey` |
| 68 | +- `clear_text` |
| 69 | +- `screenshot` |
| 70 | + |
| 71 | +## MCP |
| 72 | + |
| 73 | +Run the local MCP server: |
| 74 | + |
| 75 | +```powershell |
| 76 | +$env:PYTHONPATH='src' |
| 77 | +python -m simulateinput.cli.main mcp serve |
| 78 | +``` |
| 79 | + |
| 80 | +Current MCP tools include session management, window attach, text/UIA/OCR/image lookup, click actions, keyboard actions, drag, and screenshot capture. |
| 81 | + |
| 82 | +## Platform Status |
| 83 | + |
| 84 | +- `Windows` - primary implementation, real execution and smoke tested |
| 85 | +- `macOS` - MVP driver implemented, requires Accessibility / Automation / Screen Recording permissions |
| 86 | +- `Linux X11` - MVP driver implemented, depends on `wmctrl`, `xdotool`, and a screenshot helper |
| 87 | +- `Linux Wayland` - compatibility layer, helper-tool dependent and not full parity |
| 88 | + |
| 89 | +## Installation Notes |
| 90 | + |
| 91 | +See `docs/cross-platform-installation.md` for: |
| 92 | + |
| 93 | +- Python dependencies |
| 94 | +- Tesseract OCR setup |
| 95 | +- macOS permissions |
| 96 | +- Linux X11 and Wayland helper packages |
| 97 | + |
| 98 | +## Safety Boundary |
| 99 | + |
| 100 | +This project is intended for automation of your own software, test environments, and explicitly authorized systems. It is not intended for bypassing third-party anti-bot controls or CAPTCHAs. |
0 commit comments