Metadata-Version: 2.4
Name: dbtips-curate
Version: 0.3.1
Summary: Three-way merge curation for dbtips pipeline-generated dossier caches. Ships agentskills.io-compatible skills for Claude Code, OpenAI Codex, and Google Antigravity.
Author: Aganitha
Author-email: dev@aganitha.ai
Requires-Python: >=3.11,<4.0
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Requires-Dist: click (>=8.1,<8.2)
Requires-Dist: pydantic (>=2.6.0,<3.0.0)
Requires-Dist: rich (>=13.7.0,<14.0.0)
Requires-Dist: typer (>=0.15.0,<1.0.0)
Description-Content-Type: text/markdown

# dbtips-curate

Three-way merge curation for dbtips pipeline-generated dossier JSON caches.

Scientists hand-curate cache files after pipeline generation (deleting noise,
adding records from external sources, patching status fields). When the
pipeline refreshes, those edits get wiped. `dbtips-curate` preserves scientist
intent as an append-only git-tracked operation log and replays it onto every
refresh — so the merged served file is always "fresh pipeline data + scientist
intent" without manual re-curation.

## Scope

- Sections (section-agnostic engine; add more via config — see
  [docs/adding-sections.md](docs/adding-sections.md)):
  - `/evidence/target-literature/` — `target_disease`, key `PMID`
  - `/target-assessment/antibodies/` — `target`, key `Antibody` + `Provider`
  - `/evidence/target-pathway/` — `target`, key `image_url`
  - `/target-assessment/assayability/` — `target`, key `ChEMBL ID`
- Entity-type folders: `cached_data_json/target_disease/<target>-<disease>.json`
  and `cached_data_json/target/<target>.json`
- Implemented ops: `delete`, `add`, `undo`
- Deferred to v2: `patch`, `accept` CLI commands (merge engine handles both)

**Adding a new section?** Read [docs/adding-sections.md](docs/adding-sections.md)
— it has the data constraints to check, a copy-paste validator, and the
config-vs-code decision flow.

## How it actually works (the model in one paragraph)

Every tracked entity has a `curation/<entity-type>/<id>/` folder containing:
immutable raw snapshots of the section (Base for 3-way merge), an append-only
`ops.jsonl` (scientist intent), `state.json` (mode + pointers), and per-refresh
changelogs. Each scientist action is one git commit, so blame works on every
clinical decision. On every pipeline refresh, `entity ingest` takes a fresh
snapshot (= Theirs), runs a 3-way merge (Base + Theirs + ops → Merged), and
splices the merged section back into the served file. Scientist edits survive
across refreshes; pipeline updates land on top.

## Install

The package is published to public PyPI. Any Python tool runner works.

### End-user install (recommended)

```bash
# One-shot, no persistent install (great for trying it out):
uvx dbtips-curate --version

# Persistent global install (recommended for daily use):
uv tool install dbtips-curate
# or:  pipx install dbtips-curate
# or:  pip install dbtips-curate

dbtips-curate --version              # dbtips-curate/0.2.0
```

Then bootstrap the skills into your AI coding harnesses:

```bash
dbtips-curate dev bootstrap
```

Interactive menu picks Claude / Codex / Antigravity / all. Non-interactive form:

```bash
dbtips-curate dev bootstrap --targets all
dbtips-curate dev bootstrap --targets claude,codex
dbtips-curate dev bootstrap --targets claude --project .
```

### Skill install matrix

Every supported harness reads the same [agentskills.io](https://agentskills.io)
`SKILL.md` format we ship — no conversion.

| Harness | Global path | Project path |
|---|---|---|
| Claude Code | `~/.claude/skills/<name>/` | `<project>/.claude/skills/<name>/` |
| OpenAI Codex | `~/.agents/skills/<name>/` | `<project>/.agents/skills/<name>/` |
| Google Antigravity | `~/.gemini/config/skills/<name>/` | `<project>/.agents/skills/<name>/` (shared with Codex) |

### Dev install (contributors only)

```bash
git clone https://github.com/aganitha/dbtips-curate.git
cd dbtips-curate
poetry install
poetry run dbtips-curate --version
```

For the test suite: `poetry install --with dev && poetry run pytest`.

> **Version note:** `typer` 0.12.x is incompatible with `click` 8.4+ (released
> April 2026) — options like `--path` get misparsed as positional args and
> `--version` stops exiting eagerly. The `typer = "^0.15.0"` pin in
> `pyproject.toml` sidesteps this. If you see *"Got unexpected extra argument"*
> or *"Missing command"* on a valid flag, upgrade typer.

## One-time setup (per platform repo)

From the platform repo root, create the curation folder + per-team config:

```bash
dbtips-curate dev init \
    --path backend/res-immunology-automation/res_immunology_automation/src/scripts/curation
```

This drops `.dbtips-curate.toml` next to `cached_data_json/` and registers all
ten curated section endpoints (literature, antibodies, pathway, assayability,
target-pipeline, ontology, gene-essentiality, orthologs, supplementary
materials). The file is committed to git so every scientist gets identical
config.

## Commands at a glance

```
dbtips-curate
  entity      list | status | track | snapshot | diff | ingest | records | compare | finalize
  op          delete | add | patch | undo
  history     ops | report
  dev         init | bootstrap | install-skill
```

Every command supports `--json` for machine-readable output, exits 0/1/2 for
success / runtime error / usage error, and follows the contract:
`{"ok": true, "data": {...}}` on success, `{"ok": false, "error": {"code", "message", "hint"}}` on error.

### `entity` — lifecycle

| Command | Purpose |
|---|---|
| `entity list [--type T]` | All tracked entities of one type with their mode + last refresh |
| `entity track <id>` | Bootstrap. **Walks git history** of the served file → synthesizes `delete` ops for historical removals; flags historical additions for scientist review; sets mode based on what was found (see modes below). `--skip-history` opts out. |
| `entity snapshot <id> [--reason ...]` | Capture the current served file's section as an immutable raw snapshot. First snapshot becomes Base for 3-way merge. |
| `entity status <id>` | Prioritized worklist: conflicts → historical_additions → bootstrap_candidates → resurrected → new from pipeline. Includes `suggested_filters` for greenfield noise triage. |
| `entity records <id> [--page N --page-size 20] [--full] [--fields F1,F2]` | List current records, paginated. For tabular review in chat — Claude renders as markdown table with `delete` action per row. |
| `entity diff <id>` | **Read-only preview** of the 3-way merge vs the current served file. Writes nothing. Use before `ingest` if scientist wants to see the table first. |
| `entity ingest <id>` | Take fresh snapshot → 3-way merge → splice merged section back into served file. Writes changelog + conflicts file. This is what the pipeline calls. |
| `entity finalize <id> [--no-push]` | Mark current review complete: advance `last_reviewed_raw`, clear bootstrap_candidates, prune old snapshots, git commit. Pushes to remote if `auto_push_on = "finalize"`. |

### `op` — scientist actions

| Command | Purpose |
|---|---|
| `op delete <id> --section <s> --key K=V [--key K2=V2 ...] --reason "..."` | Tombstone a record. Whole-record delete. Compound-key sections pass one `--key` flag per identity field. Per-op git commit. |
| `op add <id> --section <s> --record-file F --source "..."` | Add a record from an external source. Validates against the section's `required_fields`. Use `-` to read from stdin. |
| `op patch <id> --section <s> --key K=V [--key K2=V2 ...] --set field=value [--set ...] --reason "..."` | Change one or more fields on an existing record. Values are JSON-decoded when possible (`Phase=3` → int, `Approved=true` → bool). Identity fields cannot be `--set` (rejects with `PATCH_TOUCHES_IDENTITY`). |
| `op undo <id> --undoes <op_id> --reason "..."` | Cancel a prior op. Endpoint inferred from the target op. Rejects unknown op_ids, undo-of-undo, and double-undo. Original op stays in the log for audit. |

All support `--dry-run` to preview what would be logged without writing.

### `history` — read-only views

| Command | Purpose |
|---|---|
| `history ops <id> [--since DATE] [--actor EMAIL]` | The full operation log with optional filters. |
| `history report <id> [--output FILE] [--open]` | Generate standalone HTML report (all queues + ops table). `--open` launches it in your browser. Good for review meetings or async sharing. |

### `dev` — setup + maintenance

| Command | Purpose |
|---|---|
| `dev init --path DIR` | Create the curation folder + write a default `.dbtips-curate.toml`. |
| `dev bootstrap [--targets claude,codex,antigravity\|all] [--project DIR]` | Install every skill into the chosen AI coding harnesses. Interactive menu if `--targets` omitted. `--project .` also drops skills into the current repo's workspace-scope skill folder. Idempotent. |
| `dev install-skill [--target claude\|codex\|antigravity] [--home DIR] [--only NAME]` | Single-target install primitive used by `bootstrap`. Prefer `bootstrap` for end-user setup. |

## Mode state machine

| Mode | Set by | Meaning |
|---|---|---|
| `greenfield` | `track` with no git history changes | Fresh pipeline output, no prior scientist edits detected |
| `brownfield-from-git` | `track` when git history shows record-level adds/removes in past commits | Prior scientist edits detected via git history; synthesized delete ops + historical_additions queue surfaced |
| `brownfield-detected` | `ingest` (auto-promotion) when pipeline output drops Base records that aren't tombstoned | Records preserved as bootstrap_candidates pending scientist confirmation |
| `brownfield-bootstrap` | Legacy label; behaves identically | (Backward compatible) |
| `active` | `finalize` after first review session | Steady-state operation |

## Filesystem layout

```
backend/.../scripts/
  cached_data_json/                                  (pipeline writes here, served to clients)
  curation/                                          (this tool writes here, tracked in git)
    .dbtips-curate.toml                              (per-team config + section schema registry)
    target_disease/<entity-id>/
      raw/<ts>.json                                  (immutable section snapshots)
      ops.jsonl                                      (append-only scientist operations)
      state.json                                     (mode + last_reviewed_raw + git history summary)
      conflicts/<ts>.json                            (open conflicts after last merge)
      changelog/<ts>.md                              (human-readable per-refresh summary)
      merged.json                                    (post-merge section, spliced back into served file)
      reports/<ts>.html                              (HTML reports — only created on demand)
```

## Three-way merge semantics

Each `entity ingest` runs:

```
Base    = snapshot at state.last_reviewed_raw         (what scientist last accepted)
Theirs  = fresh snapshot of current served file       (what pipeline just produced)
Ours    = apply_ops(Base, ops.jsonl)                  (scientist intent layered on Base)

Merged  = three_way_merge(Base, Theirs, Ours)
```

Per identity-keyed record (PMID for literature):

| In Base | In Theirs | Has op | Result |
|---|---|---|---|
| ✓ | ✗ | tombstone | silent (both agree to remove) |
| ✓ | ✗ | none | `removed_by_pipeline` queue — in brownfield modes, **preserved** as bootstrap candidate |
| ✗ | ✓ | none | `new_from_pipeline` queue — added to merged |
| ✗ | ✓ | tombstone | dropped from merged, logged as `resurrected_tombstone` |
| ✓ | ✓ | patch | per-field merge; pipeline-changed + scientist-patched same field → `conflict` (scientist wins by default) |
| ✗ | ✗ | add | scientist-added record, included in merged |

## Pipeline integration (build_dossier.py)

Already wired in via a CURATION HOOK block at the top of `build_dossier.py`.
Two knobs:

```python
IS_REFRESH: bool = False        # True = clear section + redis before pipeline regen
CURATE_ENABLED: bool = True     # False = bypass the hook entirely (rollback)
```

With `IS_REFRESH = True`, the hook (in order):
1. Snapshots the current served file's section (pre-refresh Base)
2. Clears the section key from served JSON + redis
3. Lets the pipeline regenerate from scratch
4. Calls `entity ingest` → 3-way merge → splices merged back into served file

With `IS_REFRESH = False` (the safe default), only step 4 runs — keeps
curation state in sync without forcing regeneration.

## Driving from Claude (Claude Code only)

After `dev install-skill`, in any Claude Code session:

```
curate acvr2a-no-disease
```

Claude reads the SKILL.md, then:

1. Runs `entity status` and presents the worklist as a markdown table
2. Surfaces brownfield-from-git evidence (commits walked, historical adds/removes)
   prominently before anything else
3. Walks decisions one-by-one, calling `op delete` / `op add` with `--json` after explicit confirmation
4. On `show me records`, renders `entity records --page N --page-size 20` as a paginated markdown table with delete actions per row
5. On `diff` or `preview`, calls `entity diff` (read-only) and renders the four delta tables
6. On session end, calls `entity finalize` and summarizes

> **Claude Code is required.** Skills work in Claude Code (terminal, desktop
> app, IDE extensions). The Claude.ai consumer Desktop app would need an MCP
> server wrapper (the `mcp_server.py` stub exists; `mcp serve` subcommand not
> built).

## Testing

```bash
make test   # 26 tests covering merge engine + CLI end-to-end + git history walk + records pagination
```

The fixture in `tests/fixtures/acvr2a-no-disease.json` is a real 5-record slice
of the production literature section. End-to-end tests run the full lifecycle
(track → snapshot → ingest → op delete → simulate refresh → ingest → finalize)
against a temp git repo mirroring the production layout.

## Status

- **Production-ready for literature curation** on `target_disease` entities
- 26/26 tests passing
- Pipeline hook in `build_dossier.py` ready behind the `IS_REFRESH` / `CURATE_ENABLED` flags
- Scientist-facing skill installed via `dev install-skill`

## Roadmap (deferred to v2)

- `op patch` (field-level updates) — merge engine handles them, no CLI yet
- `op accept` (resolve conflicts via choice)
- MCP server runtime (`mcp serve` subcommand) for Claude.ai consumer Desktop
- Field-level brownfield detection (needed when expanding to clinical trial endpoints)
- Other section schemas: `/evidence/search-patent/`, `/market-intelligence/*`, etc.

