Implement OmniMCP for Claude computer control #946
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…md, .env.example, .dockerignore
This new strategy: 1. Uses OmniParser for parsing visual state and Gemini 2.0 for evaluation 2. Takes natural language task descriptions instead of recording IDs 3. Processes coalesced actions from events.py 4. Builds and maintains a process graph where states are nodes and actions are edges The graph is constructed before replay based on recording + task description, and is updated during replay based on observed states. Uses Pydantic for structured data handling and robust JSON parsing. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
This commit adds OmniMCP, a system that enables Claude to control the computer using the Model Control Protocol. Key components: - OmniParser adapter for UI element detection - MCP server implementation - CLI interface for commands and debugging - Comprehensive documentation OmniMCP combines OmniParser's visual understanding with Claude's natural language capabilities to automate UI interactions.
|
Closing in favor of PR #947, which contains a clean version of this implementation based on the current main branch. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
OmniMCP Implementation
This PR adds OmniMCP, a system that enables Claude to control the computer through the Model Control Protocol (MCP). OmniMCP combines OmniParser's visual understanding with Claude's natural language capabilities to automate UI interactions.
Key Components
OmniParser Adapter (
openadapt/adapters/omniparser.py)OmniMCP Core (
openadapt/omnimcp.py)MCP Server (
openadapt/mcp/server.py)CLI Runner (
openadapt/run_omnimcp.py)fireUsage
Implementation Details
pynputfor keyboard and mouse controlconfig.py🤖 Generated with Claude Code
Co-Authored-By: Claude [email protected]