Metadata-Version: 2.4
Name: screen2prompt
Version: 0.0.2
Summary: Give any LLM computer control (VLA) in three lines — Set-of-Mark screen tagging, Pydantic actions, zero-latency OS control.
Home-page: https://github.com/AshgrayIM/screen2prompt
Author: AshgrayIM
Author-email: imchangkyu@gmail.com
License: MIT
Project-URL: Bug Tracker, https://github.com/AshgrayIM/screen2prompt/issues
Project-URL: Source Code, https://github.com/AshgrayIM/screen2prompt
Project-URL: Documentation, https://github.com/AshgrayIM/screen2prompt#readme
Keywords: vla,vision-language-action,llm,computer-use,gui-automation,set-of-mark,screen-agent,gpt-4o,structured-output
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Desktop Environment
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: mss
Requires-Dist: opencv-python
Requires-Dist: pyautogui
Requires-Dist: pydantic
Requires-Dist: rich
Requires-Dist: fastapi
Requires-Dist: uvicorn
Requires-Dist: easyocr
Requires-Dist: openai
Requires-Dist: python-dotenv
Provides-Extra: demo
Requires-Dist: openai; extra == "demo"
Requires-Dist: python-dotenv; extra == "demo"
Dynamic: author
Dynamic: author-email
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: keywords
Dynamic: license
Dynamic: license-file
Dynamic: project-url
Dynamic: provides-extra
Dynamic: requires-dist
Dynamic: requires-python
Dynamic: summary

# Screen2Prompt 🎯

> Grant any LLM computer control (VLA) capabilities with just 1 CLI command or 3 lines of code.

[![PyPI version](https://img.shields.io/pypi/v/screen2prompt.svg)](https://pypi.org/project/screen2prompt/)
[![Python 3.9+](https://img.shields.io/badge/python-3.9+-blue.svg)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

![Demo](https://via.placeholder.com/800x400?text=Drag+and+Drop+Demo+GIF)

**Screen2Prompt** is an open-source VLA (Vision-Language-Action) toolkit[cite: 1]. It captures the screen, applies a **Set-of-Mark (numerical tags)** to UI elements, and executes OS actions autonomously based on the Pydantic structured output returned by any LLM (OpenAI, Ollama, LM Studio, etc.)[cite: 1].

---



## ✨ Key Features



### 🏷️ Ultimate Set-of-Mark (OpenCV + EasyOCR)

Extracts UI blocks assigning a unique numerical ID to each element[cite: 1]. 

- **OCR Immunity:** Text detected by EasyOCR is guaranteed to be tagged, preventing critical web elements (like search bars) from being filtered out.
- **DPI Aware:** Automatically calibrates coordinates for multi-monitor setups and OS display scaling (e.g., 4K monitors).



### ⚡ Universal LLM Support (Plug-and-Play)

Not locked into OpenAI! Use **Ollama, vLLM, or LM Studio** seamlessly. Simply override the `base_url` to run your local AI agent entirely for free.

### 🔄 Autonomous Closed-Loop & Token Optimization

- **ReAct Feedback Loop:** `agent.act()` automatically captures the new screen state and returns it to the LLM, creating a true autonomous agent.
- **Sliding Window:** Prevents API cost explosions and context limit errors by keeping only the most recent visual context and converting older screens to text stubs.



### 🛡️ Human-in-the-Loop & Advanced OS Control

- **Safety First:** CLI prompts a `[Y/N]` confirmation before executing physical clicks or typing.
- **Rich Action Schema:** Supports `click`, `drag`, `type`, `wait`, `press_key` (e.g., Enter, Esc), and `hotkey` (e.g., Win+D). LLMs can actively navigate, wait for loading screens, or recover from errors.



### 🚀 Hardware Acceleration (GPU / MPS)

By default, Screen2Prompt checks your system environment. If an NVIDIA GPU (CUDA) or Apple Silicon (MPS) is available, it automatically boosts OCR processing speed. 

You can also explicitly control hardware acceleration via CLI or Python code:

```bash
# Force enable GPU/MPS via CLI
screen2prompt run "Open browser" --gpu
```

```python
# Enable GPU acceleration in Python
agent = ScreenAgent(gpu=True)
```

---



## 🚀 Quick Start



### Installation

```bash
pip install screen2prompt
# Note: For NVIDIA GPU acceleration, ensure CUDA is installed for EasyOCR.
```



### 1. One-Click CLI Agent (The Magic "Click")

You don't need to write Python code. Control your PC straight from the terminal.

```bash
# Using OpenAI (requires OPENAI_API_KEY in .env)
screen2prompt run "Empty the recycle bin"

# Using Local LLM (Ollama) - 100% Free & Private
screen2prompt run "Search the weather on Google" --model ollama/llama3-vision --base-url http://localhost:11434/v1
```



### 2. Python API (Custom Integration)

```python
from screen2prompt import ScreenAgent, ActionSchema

agent = ScreenAgent()

# 1. Observe the screen
prompt_data, image_path = agent.observe() 

# 2. Execute Action & Get Feedback (Closed-Loop)
success, new_prompt, new_image = agent.act(
    ActionSchema(action_type="double_click", target_id=12)
)
```

---



## 📦 API Overview



### ActionSchema (Pydantic)

Forces all actions into a strict schema, fully compatible with GPT-4o's Structured Outputs[cite: 1].

```python
ActionSchema(
    action_type="click",       # click | double_click | drag | scroll | type | wait | press_key | hotkey | done
    target_id=3,               # ID from the mapping JSON
    destination_id=None,       # Used for drag
    text=None,                 # Used for type
    scroll_clicks=None,        # Used for scroll (positive=up, negative=down)
    wait_seconds=2.5,          # Used for wait (let the LLM decide loading times)
    key_name="enter",          # Used for press_key
    hotkey_names=["win", "d"]  # Used for hotkey combinations
)
```

---



## 🌍 REST API Server

Turn your machine into an accessible AI node.

```bash
# Start REST API server (default port 8000)
screen2prompt serve --port 8080 --host 0.0.0.0
```

**Execute Agent via POST:**

```bash
curl -X POST [http://127.0.0.1:8080/run_agent](http://127.0.0.1:8080/run_agent) \
  -H "Content-Type: application/json" \
  -d '{
        "goal": "Open YouTube and play jazz music",
        "model": "gpt-4o"
      }'
```

---



## ⚙️ Platform Permissions


| OS          | Required Permissions                                                                          |
| ----------- | --------------------------------------------------------------------------------------------- |
| **macOS**   | System Settings → Privacy & Security → **Accessibility** (Allow Terminal/Python/IDE)[cite: 1] |
| **Windows** | Allow mouse/keyboard control (Run as Administrator if needed)[cite: 1]                        |


---



## 📁 Project Structure

```text
screen2prompt/
├── screen2prompt/
│   ├── __init__.py
│   ├── llm_client.py  # Universal LLM Client (OpenAI API spec)
│   ├── vision.py      # Set-of-Mark & EasyOCR analysis
│   ├── actions.py     # OS control (PyAutoGUI)
│   ├── agent.py       # ScreenAgent + ActionSchema (Closed-Loop)
│   ├── server.py      # FastAPI REST Server
│   └── cli.py         # Terminal CLI
├── setup.py
├── requirements.txt
└── README.md
```

---



## 📄 License

MIT License — Free to use, modify, and distribute[cite: 1].

---

**⭐ Star this repo if Screen2Prompt saves your tokens and your sanity!**
