Metadata-Version: 2.4
Name: thinqos-harvest
Version: 0.5.1
Summary: Mind-as-observer probe - ship Claude Code (and other external) conversations into your thinqOS Mind.
Project-URL: Homepage, https://thinqos.com
Project-URL: Repository, https://github.com/AI4Outcomes/thinqos
Author-email: AI4Outcomes <support@thinqos.com>
License: MIT
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.13
Requires-Dist: click>=8.3
Requires-Dist: httpx>=0.28
Description-Content-Type: text/markdown

# thinqos-harvest

Mind-as-observer probe for [thinqOS](https://thinqos.com). Ships Claude Code and Codex sessions into your thinqOS Mind as captured Episodes, so the Mind learns from where you actually do your work - without sitting in the request path.

Pattern: **observer, not proxy.** The probe reads session JSONL from disk, normalizes it to the public [external ingest v1 contract](https://thinqos.com/docs/api/external-ingest/v1/schema.json), and POSTs it to `/api/ingest/external/v1` on your thinqOS instance. Claude Code and Codex can call it from Stop hooks; Codex Desktop, VS Code Codex, and Codex CLI can also be swept from the shared `~/.codex` session store for backfill.

## Install

The hook command needs [uv](https://docs.astral.sh/uv/) on PATH. `uvx --from`
fetches and runs the tagged harvester directly from this repo, so it does not
depend on a stale local checkout or a package-index release:

```bash
uvx --from 'thinqos-harvest @ git+https://github.com/AI4Outcomes/thinqos.git@main#subdirectory=tools/thinqos-harvest-reference' thinqos-harvest --help
```

```bash
curl -LsSf https://astral.sh/uv/install.sh | sh
```

Avoid copying setup commands that point at a local worktree path. Local paths
rot as worktrees are pruned; the tagged GitHub source above is the stable install
target.

## Wire up

1. **Mint an ingest key.** Go to <https://thinqos.com/api-keys>, click *Create*, copy the `tq_...` value. (Same key kind as the MCP keys - one key per machine is fine.)

2. **Set env vars** in your shell rc (`~/.zshrc`, `~/.bashrc`, or equivalent):

   ```bash
   export THINQOS_BASE_URL=https://thinqos.com
   export INGEST_API_KEY=tq_xxxx...
   ```

   Reload your shell (`source ~/.zshrc`).

3. **Print the Claude Code hook JSON snippet:**

   ```bash
   uvx --from 'thinqos-harvest @ git+https://github.com/AI4Outcomes/thinqos.git@main#subdirectory=tools/thinqos-harvest-reference' thinqos-harvest install-hook
   ```

   Copy the JSON it prints.

4. **Paste the snippet into `~/.claude/settings.json`** under the top-level `"hooks"` object. Example after pasting:

   ```json
   {
     "hooks": {
       "Stop": [
         {
           "matcher": "",
           "hooks": [
             {
               "type": "command",
               "command": "uvx --from 'thinqos-harvest @ git+https://github.com/AI4Outcomes/thinqos.git@main#subdirectory=tools/thinqos-harvest-reference' thinqos-harvest hook capture",
               "timeout": 30
             }
           ]
         }
       ]
     }
   }
   ```

   If you already have a `Stop` hook configured, append our entry to the existing `hooks` array rather than replacing. `install-hook` will warn you when this is the case.

5. **Smoke test Claude Code:**
   - Start a Claude Code session, type one prompt, exit normally.
   - Run `uvx --from 'thinqos-harvest @ git+https://github.com/AI4Outcomes/thinqos.git@main#subdirectory=tools/thinqos-harvest-reference' thinqos-harvest list` - your session should appear within a second.

## Codex

Codex Desktop, VS Code Codex, and Codex CLI write rollout JSONL files under
`~/.codex/sessions/YYYY/MM/DD/`; older files may live under
`~/.codex/archived_sessions/`. The Codex adapter reads both locations.

After setting `THINQOS_BASE_URL` and `INGEST_API_KEY`, print the Codex hook
snippet:

```bash
uvx --from 'thinqos-harvest @ git+https://github.com/AI4Outcomes/thinqos.git@main#subdirectory=tools/thinqos-harvest-reference' thinqos-harvest install-hook --source openai.com/codex
```

Paste the snippet into `~/.codex/hooks.json`. Codex passes the
current `transcript_path` to the hook; the hook uploads the completed turn and
exits 0 so it does not block Codex.

For backfill, or if you do not want to install a Codex hook yet, run:

```bash
uvx --from 'thinqos-harvest @ git+https://github.com/AI4Outcomes/thinqos.git@main#subdirectory=tools/thinqos-harvest-reference' thinqos-harvest run --sources openai.com/codex --since 2026-05-01T00:00:00 --batch-size 5
```

Prefer a bounded backfill such as the last 7 days. `run` chunks uploads by
default so large local histories do not exceed server request limits.

For a one-shot run without installing the package globally:

```bash
THINQOS_BASE_URL=https://thinqos.com \
INGEST_API_KEY=tq_xxxx... \
uvx --from 'thinqos-harvest @ git+https://github.com/AI4Outcomes/thinqos.git@main#subdirectory=tools/thinqos-harvest-reference' thinqos-harvest run --sources openai.com/codex --since 2026-05-01T00:00:00 --batch-size 5
```

Omit `--sources` to sweep all registered adapters.

## CLI

| Command | What it does |
| --- | --- |
| `thinqos-harvest list [--limit N]` | List your captured sessions newest-first. |
| `thinqos-harvest forget <session_id>` | Delete one captured session by id (irreversible). |
| `thinqos-harvest install-hook [--source SOURCE]` | Print the JSON hook snippet for Claude Code or Codex. |
| `thinqos-harvest run [--sources SOURCE] [--since TIMESTAMP] [--batch-size N] [--no-drain-pending]` | Manual sweep - discover any sessions not yet shipped and POST them in chunks. Use for bounded Codex or Claude Code backfill after a long offline period. |
| `thinqos-harvest hook capture [--source SOURCE]` | Hook entry point - reads `transcript_path` from stdin JSON. You should not run this directly; Claude Code and Codex hooks do. |
| `thinqos-harvest doctor` | Show hook health, replay queue depth, local log location, and recent server captures. |

## Denylists

Two opt-out layers, both edited at `~/.config/thinqos-harvest/`:

- **`denylist.txt`** - newline-delimited substrings; any session whose `cwd` contains a substring is skipped entirely. Example: add `personal-taxes` to skip captures from `~/Documents/personal-taxes/`.
- **Content denylist** is hard-coded: turns containing `.env`, `api_key=…`, `sk-…`, or matching `(api_key|secret|password|token)=<16+ chars>` are dropped before POST. Oversized `tool_result` content (>32KB) is also dropped.

## Reliability

- Hook capture always exits 0; never blocks a Claude Code or Codex session even on bug or network failure.
- Hook health is written to `~/.config/thinqos-harvest/status.json`; detailed hook output goes to `~/.config/thinqos-harvest/hook.log`.
- Failed POSTs spool to `~/.config/thinqos-harvest/pending/<uuid>.json` and drain on the next `run` by default. Use `--no-drain-pending` for a tightly scoped backfill.
- Server-side dedup keyed by `(source, source_external_id)`; re-shipping the same session is idempotent. Claude Code captures include a monotonic `session_revision` so a later hook can replace an earlier partial capture.

## Forgetting a capture

```bash
thinqos-harvest forget <session_id>
```

Where `<session_id>` is the value `thinqos-harvest list` shows in the last column. This deletes the Episode on the server side and cascades to any extracted knowledge. For Codex captures, pass `--source openai.com/codex`.

For a per-turn / per-content scrub (rather than whole-session delete), contact support. It's a deliberate Phase-A non-goal but ship-able if asked for.

## Privacy posture

Opt-out, not opt-in. Adapter ships everything that isn't denied. If you'd rather have explicit opt-in per session, this probe is the wrong tool for you - the design decision is explicit in the [spec](https://github.com/AI4Outcomes/thinqos/blob/main/docs/superpowers/specs/2026-05-26-mind-observer-pattern-design.md).

## Versioning

Tagged `tools/thinqos-harvest-vX.Y.Z` in the thinqos monorepo. A push to a matching tag triggers the publish workflow.
