Metadata-Version: 2.5
Name: codex-can-see
Version: 0.1.0
Summary: Local-first, frame-backed video analysis with verifiable evidence
License-Expression: MIT
License-File: LICENSE
Requires-Python: >=3.12
Requires-Dist: pyyaml<7,>=6
Requires-Dist: tomli-w<2,>=1.2
Requires-Dist: yt-dlp<2027,>=2026.8.19
Provides-Extra: full-cpu
Requires-Dist: torch==2.8.0; extra == 'full-cpu'
Requires-Dist: torchaudio==2.8.0; extra == 'full-cpu'
Requires-Dist: whisperx==3.8.6; extra == 'full-cpu'
Provides-Extra: full-cuda
Requires-Dist: torch==2.8.0; extra == 'full-cuda'
Requires-Dist: torchaudio==2.8.0; extra == 'full-cuda'
Requires-Dist: whisperx==3.8.6; extra == 'full-cuda'
Provides-Extra: full-mlx
Requires-Dist: mlx-whisper==0.4.3; extra == 'full-mlx'
Description-Content-Type: text/markdown

# Codex Can See

Codex Can See is a local-first, agent-friendly video evidence tool. It extracts verifiable frames, optionally adds local transcription, asks a configured vision-language model for analysis, and emits one JSON object on stdout for agents to parse. People can inspect the same frames, transcript, report, and work directory.

## How it works

```text
plugin instructions -> non-mutating doctor -> local evidence extraction -> configured provider -> JSON + report
```

The plugin never contains Python source code, credentials, state, or an installer. The pinned bootstrap command installs a persistent uv-managed tool environment and records readiness under `~/.config/codex-can-see/`.

## Requirements

- macOS Apple Silicon, macOS x86_64, or Linux x86_64
- [uv](https://docs.astral.sh/uv/)
- `ffmpeg`
- `ffprobe`
- A provider key for OpenAI or Anthropic

Windows is unsupported in v0.1.0. Linux CUDA is implemented but remains runtime-untested until validated on real NVIDIA hardware.

## Direct installation

Use `uvx` only for first-run bootstrap:

```bash
uvx codex-can-see@0.1.0 bootstrap \
  --provider openai \
  --transcription yes
```

Bootstrap then performs the persistent installation itself. After setup, invoke the installed CLI directly:

```bash
codex-can-see doctor --mode invocation --json
codex-can-see analyze "$HOME/Movies/demo.mp4" \
  --between 00:10:00 00:13:00 \
  --question "What does the person do with the package?"
```

For a smaller frame-provider installation without local transcription:

```bash
uvx codex-can-see@0.1.0 bootstrap \
  --provider anthropic \
  --transcription no
```

A provider is required even when transcription is disabled because Codex Can See always uses frame-backed provider analysis.

### Non-interactive key input

Export the selected provider key before bootstrap:

```bash
read -r -s OPENAI_API_KEY
export OPENAI_API_KEY
uvx codex-can-see@0.1.0 bootstrap --provider openai --transcription yes
unset OPENAI_API_KEY
```

Alternatively, pipe exactly one key:

```bash
printf '%s\n' "$OPENAI_API_KEY" | \
  uvx codex-can-see@0.1.0 bootstrap \
    --provider openai \
    --transcription yes \
    --key-stdin
```

The key is stored only in `~/.config/codex-can-see/.env`. It is not placed in command arguments, state, runtime configuration, logs, stdout, or the plugin.

## Install as a Codex skill plugin

This is a three-step process: register the catalog, install the plugin payload, then bootstrap the Python runtime.

```bash
# 1. Register the marketplace/catalog
codex plugin marketplace add monodeepdas1215/codex-can-see --ref v0.1.0

# 2. Install the actual plugin
codex plugin add codex-can-see@codex-can-see-marketplace

# 3. Bootstrap the persistent runtime
uvx codex-can-see@0.1.0 bootstrap \
  --provider openai \
  --transcription yes
```

## Install as a Claude Code skill plugin

```bash
# 1. Register the marketplace/catalog
claude plugin marketplace add monodeepdas1215/codex-can-see

# 2. Install the actual plugin
claude plugin install codex-can-see@codex-can-see-marketplace

# 3. Bootstrap the persistent runtime
uvx codex-can-see@0.1.0 bootstrap \
  --provider anthropic \
  --transcription yes
```

The installed skill begins with `codex-can-see doctor --mode invocation --json`. It does not install dependencies, download models, mutate state, or ask for credentials during analysis.

## Runtime profiles

| Platform/profile | Local runtime | Fixed model | Package |
|---|---|---|---|
| Darwin/arm64 | MLX Whisper | `mlx-community/whisper-small-mlx` | `codex-can-see[full-mlx]` |
| Darwin/x86_64 | WhisperX CPU | `small` | `codex-can-see[full-cpu]` |
| Linux/x86_64 CPU | WhisperX CPU | `small` | `codex-can-see[full-cpu]` |
| Linux/x86_64 CUDA 12.6+ | WhisperX CUDA | `small` | `codex-can-see[full-cuda]` |
| Transcription disabled | none | none | `codex-can-see` |

Whisper model selection is fixed and intentionally not a public setting. A disabled transcription profile performs no model warm-up and checks no model cache, but frame extraction and provider analysis continue.

URL sources use the yt-dlp executable installed inside the persistent Codex Can See tool environment. Local files do not make network requests.

## Command interface

```bash
codex-can-see analyze SOURCE \
  (--start TIME --end TIME | --at TIME | --after TIME | --before TIME | --between TIME TIME) \
  [--question TEXT] \
  [--out-dir DIR]
```

Only one time-filter form is allowed:

| Form | Window |
|---|---|
| `--start 10:00 --end 13:00` | exactly `[10:00, 13:00]` |
| `--between 10:00 13:00` | `[10:00, 13:00]` |
| `--after 10:00` | `[10:00, end-of-media]` |
| `--before 13:00` | `[media-start, 13:00]` |
| `--at 14:32` | `[14:32, 14:37]` |

Timestamps remain on the original media timeline.

`--out-dir` is request-specific:

- missing directories, including parents, are created;
- an existing empty directory is accepted;
- a non-directory is rejected;
- a non-empty directory is rejected to prevent stale artifact mixing;
- `work_dir` is always an absolute path.

Unknown arguments use normal command-line usage errors. Provider, model, frame count, frame quality, frame dimensions, and output format are setup-time settings, not analyze arguments.

## Readiness and exit codes

Always start with:

```bash
codex-can-see doctor --mode invocation --json
```

The doctor checks state, package version, runtime configuration, provider/key presence, ffmpeg, ffprobe, yt-dlp, backend/profile consistency, and fixed model cache when transcription is enabled. It performs no installation, provider request, model download, state repair, or prompt.

```text
0  ready/success
2  usage or provider resolution error
3  state/schema/package mismatch or incomplete setup
4  missing ffmpeg, ffprobe, or yt-dlp
5  provider configuration error
6  backend/profile mismatch
7  bootstrap/runtime installation failure
8  media/evidence processing failure
```

Provider failure during analysis is different from a setup error: Codex Can See preserves local evidence, writes the report, emits valid JSON, sets `answer: null`, adds a redacted warning, and exits `0`.

## Provider and frame overrides

Bootstrap accepts one non-secret YAML override:

```yaml
schema: codex-can-see/runtime-override/v1
provider: openai

auth:
  env: EXAMPLE_API_KEY
endpoint:
  base_url: https://replacement.example/v1
model:
  default: compatible-model

frames:
  count: 16
  quality: 0.80
  max_dimension: 1280
```

Install it with:

```bash
export EXAMPLE_API_KEY="..."
uvx codex-can-see@0.1.0 bootstrap \
  --provider openai \
  --transcription yes \
  --provider-overrides ./runtime-override.yaml
```

Allowed bounds are:

```text
frames.count          1..100
frames.quality        0.0..1.0
frames.max_dimension  16..4096
```

Defaults are `12`, `0.75`, and `1024`. A custom `auth.env` is a variable name, never a secret value.

## Private configuration

Readiness and profile metadata are stored in `~/.config/codex-can-see/state.toml`.

```text
~/.config/codex-can-see/
├── state.toml
├── runtime.json
├── provider-overrides.json
└── .env
```

The directory mode is `0700`; files are mode `0600` and are replaced atomically. `state.toml` contains no key values. Switching providers writes the newly selected key and removes the previous provider's saved key.

## Output contract

stdout contains exactly one JSON object. Diagnostics go to stderr.

```json
{
  "source": {
    "url": "/absolute/path/demo.mp4",
    "duration_sec": 1200.0,
    "title": "demo.mp4"
  },
  "analysis_window": {
    "requested_form": "between",
    "start_seconds": 600.0,
    "end_seconds": 780.0
  },
  "answer": {
    "provider": "openai",
    "model": "gpt-4o",
    "text": "The person lifts the package..."
  },
  "evidence": {
    "frames": [
      {
        "path": "/absolute/path/frames/frame_000.jpg",
        "timestamp_sec": 600.0
      }
    ],
    "transcript": [],
    "transcript_source": "none",
    "transcript_status": "disabled",
    "transcript_error": null
  },
  "warnings": [],
  "work_dir": "/absolute/path"
}
```

Interpret transcript status exactly:

| Status | Meaning |
|---|---|
| `ok` | Transcript segments are available. |
| `no_speech` | Local Whisper completed and found no speech. |
| `unavailable` | Local transcription failed; inspect `transcript_error`. |
| `disabled` | The installed profile intentionally excludes transcription. |

`disabled` is not a media observation. `unavailable` does not invalidate frame evidence.

## Work directory

`work_dir` contains:

- `frames/frame_000.jpg` and sibling keyframes;
- `frames.json`;
- `transcript.txt`;
- `transcription.json` when local Whisper runs;
- downloaded media and metadata for URL sources;
- `report.md`.

Frame paths in JSON are absolute. Open them directly when visual verification is possible.

## Development

Run tests without creating a public development extra:

```bash
uv run --with pytest --with PyYAML python -m pytest -q
```

Build Python and plugin artifacts:

```bash
uv build
uv run python scripts/build_plugin.py
```

Validate both plugin surfaces:

```bash
claude plugin validate ./plugin --strict
claude plugin validate ./ --strict
```

## Uninstall

Remove the Python runtime and private configuration:

```bash
uv tool uninstall codex-can-see
rm -rf ~/.config/codex-can-see
```

Remove the agent plugin through Codex or Claude. Agent-plugin removal does not delete user media or Python state.

## License

MIT. See [LICENSE](LICENSE).
