Metadata-Version: 2.5
Name: streamembed
Version: 0.1.0
Summary: Search your own screen history by describing what you saw, from screenshots, on your machine
License-Expression: Apache-2.0
License-File: LICENSE
License-File: NOTICE
Keywords: embeddings,on-device,retrieval,screen-history,screenshots
Classifier: Operating System :: MacOS
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Requires-Python: >=3.12
Requires-Dist: huggingface-hub>=0.30
Requires-Dist: mss>=9
Requires-Dist: numpy>=2.2
Requires-Dist: pillow>=11
Requires-Dist: pyobjc-framework-quartz>=10; sys_platform == 'darwin'
Requires-Dist: qwen-vl-utils==0.0.14
Requires-Dist: safetensors>=0.4
Requires-Dist: torch==2.8.0
Requires-Dist: torchvision==0.23.0
Requires-Dist: transformers==5.17.0
Provides-Extra: dev
Requires-Dist: pytest>=8; extra == 'dev'
Description-Content-Type: text/markdown

# StreamEmbed

Search your own screen history by describing what you saw. StreamEmbed indexes screenshots, from a folder or from your screen as you work, and finds a past screen from a sentence. Everything runs on your machine.

It embeds the pixels with Qwen3-VL-Embedding-2B, and encodes screenshots as a stream so that this keeps up on a laptop: blocks that repeat a neighbor are skipped, and the keys and values of blocks that did not change are reused from the previous frame.

## Use

```python
import streamembed as se

se.configure(index_type="flat")                      # optional; see below
se.build_index_from_directory("screenshots")         # a folder of .png / .jpg / .webp, searched recursively
capture = se.build_index_from_now(interval=10)       # capture the screen every 10 s, in the background

results = se.query_index("the flight search page with fares to Denver", top_k=10)
se.visualize_results(results)                        # writes an HTML page and opens it

capture.stop()
```

`examples/main.py` is a complete script.

| Function | What it does |
|---|---|
| `available_index_types()` | `["flat", "binary"]` |
| `available_models()` | Model names: those in the model repo, then the built-in base models |
| `configure(index_dir, index_type, model, models_repo, device, width)` | Where the index lives (default `./streamembed_index`), the index type and the model. Optional |
| `build_index_from_directory(directory, order="mtime")` | Index a folder as one stream. `order="mtime"` uses each file's time as its capture time; `order="name"` sorts by path |
| `build_index_from_now(interval, duration=None, block=False)` | Capture the primary display every `interval` seconds, in the background. Returns a `Capture` with `stop()`, `pause()`, `resume()`, `wait()`, `stats` |
| `query_index(query, top_k)` | `Result`s with score, time, app, window title and the stored image path. A match within 30 s of a better one is dropped |
| `visualize_results(results)` | A self-contained HTML page. Each result opens full size with the frames before and after it; arrow keys step through time |

Screens that did not change are not stored again: a capture is encoded when more than 10 % of its kept blocks changed since the last stored one, or 30 s later if anything changed (the paper's change gate).

### The index folder

| File | Content |
|---|---|
| `frames.jsonl` | One line per stored screen: time, image path, app, window title |
| `vectors.f16` | One 2048-d fp16 vector per line of `frames.jsonl` |
| `images/` | The stored screenshots, 1280 px wide |
| `seen.jsonl` | When an unchanged screen was still on screen |
| `codes.bin`, `mu.npy` | `binary` only |
| `thumbs/`, `results.html` | Made by `visualize_results` |

### Index types

| Type | Search |
|---|---|
| `flat` | Exact inner product over every stored vector (fp16, 4 KB per frame) |
| `binary` | Exact up to 1,000 frames. Beyond that, Hamming top 1,000 over 1-bit codes of the first 512 dimensions (64 B per frame, kept in memory), then exact re-rank. About 0.13 GB of memory per year of 5-s captures |

An index folder can be reopened with the other type. It also records the encoder, and refuses vectors from a different one.

### Models

A model is a base model from its own Hugging Face repository, pinned to a commit, plus two LoRAs from the StreamEmbed model repository: one for screenshots (a consistency adapter that keeps pruned, streamed frames close to full frames) and one for questions. Only the LoRAs live in the StreamEmbed repository; its `models.json` names each model's base repository, commit and files.

| Model | What it is |
|---|---|
| `qwen3-vl-2b` (default) | Qwen3-VL-Embedding-2B with the StreamEmbed LoRAs |
| `qwen3-vl-8b` | Qwen3-VL-Embedding-8B with the StreamEmbed LoRAs; needs a GPU with about 20 GB |
| `qwen3-vl-2b-base` | Qwen3-VL-Embedding-2B without LoRAs; built in, needs no model repo |

The model repository also holds LoRAs for SigLIP2 So400m, SigLIP2 Base and VultronRetrieverFlash 0.8B. `available_models()` lists them, but the stream encoder supports Qwen3-VL models only, so selecting one raises `NotImplementedError` when StreamEmbed first encodes.

`configure(models_repo=...)` or `$STREAMEMBED_MODELS` points to another Hugging Face repository or a local folder with the same layout. An index records its model and refuses vectors from another one.

The first run of `qwen3-vl-2b` downloads the base model (about 4 GB) and the LoRAs (70 MB each); after that nothing is downloaded. Encoding needs about 8 GB of memory and takes 0.7 to 1.3 s per frame on an M2 Pro GPU; the change gate skips screens that did not change.

## Requirements

Python 3.12 or later on macOS (Apple silicon, tested) or Linux under X11 (capture not yet tested). On macOS, capturing the screen needs Screen Recording permission for the app that runs Python; `build_index_from_now` raises `PermissionError` until it is granted.

Dependencies are pinned to torch 2.8.0 and transformers 5.17.0: the stream encoder re-implements parts of the Qwen3-VL forward pass against transformers internals, and its outputs were checked for equality with these versions. Install StreamEmbed in its own environment (for example `uv tool`, `pipx`, or a venv) if your project needs other versions.

## Privacy

- Screenshots, vectors and metadata stay in the index folder (owner-only). Screenshots are stored unencrypted.
- Capture is skipped when the screen is locked, when the frontmost app is a password manager, and when the frontmost window is a private browsing window. Other windows on screen are not checked; use `capture.pause()` for sensitive work.
- Nothing is sent over the network after the model download.

## Development

```bash
uv venv && uv pip install -e ".[dev]"
.venv/bin/python -m pytest          # API tests with a stand-in encoder; no model needed
```

`src/streamembed/engine/` is the stream encoder from the paper's code and keeps its names: Token Pruning is spatial pruning, KV Reuse is temporal reuse.

## License

Apache-2.0. See `LICENSE` and `NOTICE`; the model weights are downloaded under their own license.
