harness
AI-driven user testing for iOS Simulator, macOS apps, and web apps. Write a goal in plain language; an LLM agent drives the UI and reports friction. macOS 14+, Swift 6.
What is it?
What it is
A native macOS developer tool that drives an iOS Simulator, a macOS app, or a web app with an AI agent to run user tests (real-user simulation) and identify UX friction.
Why it exists
To enable developers to run realistic user tests (not scripted) using AI agents that simulate real user behavior, uncovering UX issues like dead ends, ambiguous labels, and unresponsive controls.
Who should use it
iOS, macOS, and web developers, QA engineers, product managers, and AI/LLM developers who want to run realistic user‑simulation tests driven by an LLM without writing scripted UI tests.
Who should avoid it
Users who do not have macOS 14+ or are unfamiliar with Swift/macOS development, or who require a fully cross‑platform solution beyond iOS/macOS/web, or who cannot grant the necessary accessibility and screen‑recording permissions.
How it works
A quick walkthrough in plain English
How harness works
Step 1 of 3
You interact with it
Open harness, send a request, or connect it to your stack.
Features
Advantages
- Eliminates scripted UI tests in favor of realistic user behavior
- Multi-platform compatibility with unified friction reporting
- Cost-effective local inference option with Ollama
- Improved accuracy via Set-of-Mark element targeting
- Agent runs integrated into GUI history for easy tracking
- Auto-update ensures minimal maintenance overhead
- Enhanced reliability with Web-driver timeouts and stability gates
- Secure credential management with Keychain integration
- Flexible token budget controls for cost management
- Replayable artifacts for debugging and analysis
Disadvantages
- macOS requires Screen Recording and Accessibility permissions
- Local models may have slower performance compared to cloud models
- Web support limited to WebKit (Chrome via CDP in roadmap)
- Alpha status may imply potential stability issues
- Initial setup requires simulator installation and configuration
- Local inference speed trade-offs (5-10× slower per step)
- Friction detection quality may vary between local and cloud models
- Web driver hardening may impact certain edge cases
- Credential storage relies on Keychain, which may have platform-specific limitations
Installation
native
git clone https://github.com/awizemann/harness.git cd harness git submodule update --init --recursive brew install xcodegen xcodegen generate open Harness.xcodeproj
FAQ
Which platforms does Harness support and how do I run a target on each?
Harness can drive iOS Simulator, macOS apps, and web apps. For iOS you build and launch a scheme with `xcodebuild`, boot the simulator with `simctl`, and use WebDriverAgent for input. For macOS you launch a pre‑built .app or build a macOS scheme with `xcodebuild`, then use CGEvent for input and CGWindowListCreateImage for screenshots. For web you embed a WKWebView at a chosen viewport (default 1280×1600 desktop or 375×812 mobile) and drive it with JavaScript events and WKWebView.takeSnapshot for capture.
How can I run Harness entirely locally without sending screenshots to the cloud?
Harness includes a Local Mac inference provider that runs a vision LLM on your machine via Ollama at http://127.0.0.1:11434. In Settings you can select the Local Mac card, choose a model such as Qwen3‑VL 8B, and the screenshots never leave your Mac. This mode is free and works offline, though it is slower and may have slightly lower friction‑event quality compared to cloud models.
What credentials do I need to provide for a target app and how are they stored?
Per‑Application credential storage lets you pre‑stage username/password pairs against an Application. In the Compose Run UI you can pick a credential label, and the agent receives a `fill_credential(field: "username"|"password")` tool call. Password bytes never enter the model’s context or logs; they are stored securely in the Keychain and only sent to the agent as a field name.
How do I start an agent‑driven run using the MCP server?
Install the development‑time MCP server (`harness-mcp`) from the same source as the GUI. The MCP server exposes JSON‑RPC endpoints such as `start_run`, `get_run_status`, `get_run_result`, `cancel_run`, and `list_runs`. An MCP client (e.g., Claude) can create Applications, Personas, and Actions, stage credentials, and then start a run. The run appears in the GUI’s history with a badge indicating it was driven by an agent.
What is the Set‑of‑Mark feature and why is it useful?
Set‑of‑Mark overlays numbered badges over focusable elements in screenshots. The agent calls `tap_mark(id)` instead of raw coordinates, eliminating pixel‑guessing errors. This feature is available for iOS (via WebDriverAgent’s accessibility tree), macOS (via AXUIElement), and web (via WKWebView’s DOM). It keeps the on‑disk screenshots clean while giving the agent precise element references.
Featured in Videos
YouTube tutorials and walkthroughs for harness
Alternatives
Similar projects ranked by category, topics, and text overlap.
