Metadata-Version: 2.4
Name: dbtips-curate
Version: 0.4.0
Summary: Three-way merge curation for dbtips pipeline-generated dossier caches. Ships agentskills.io-compatible skills for Claude Code, OpenAI Codex, and Google Antigravity.
Author: Aganitha
Author-email: dev@aganitha.ai
Requires-Python: >=3.11,<4.0
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Requires-Dist: click (>=8.1,<8.2)
Requires-Dist: pydantic (>=2.6.0,<3.0.0)
Requires-Dist: rich (>=13.7.0,<14.0.0)
Requires-Dist: typer (>=0.15.0,<1.0.0)
Description-Content-Type: text/markdown

# dbtips-curate

**Three-way merge curation for dbtips pipeline-generated dossier JSON caches.**

Scientists hand-curate cache files after pipeline generation (deleting noise,
adding records from external sources, patching status fields). When the
pipeline refreshes, those edits get wiped. `dbtips-curate` preserves scientist
intent as an append-only git-tracked operation log and replays it onto every
refresh — so the merged served file is always "fresh pipeline data + scientist
intent" without manual re-curation.

Published on [PyPI](https://pypi.org/project/dbtips-curate/) as `dbtips-curate`.
Latest version: **0.3.1**.

## Install (end user)

```bash
# 1. Install the CLI (persistent global tool)
uv tool install --force --refresh dbtips-curate
# or:  pipx install dbtips-curate
# or:  pip install dbtips-curate  (into a venv)

# 2. Bootstrap SKILL.md files into your AI coding harness(es)
dbtips-curate dev bootstrap
```

`dev bootstrap` shows an interactive menu (Claude / Codex / Antigravity / all
combinations). Non-interactive form:

```bash
dbtips-curate dev bootstrap --targets all
dbtips-curate dev bootstrap --targets claude,codex
dbtips-curate dev bootstrap --targets claude --project .    # also install to <repo>/.claude/skills/
```

**Update path:** `uv tool install --force --refresh dbtips-curate` picks up
the latest PyPI release. `--refresh` is important — uv caches its resolver
decisions and will happily reinstall an old version without it.

### Multi-harness skill install matrix

Every supported harness reads the same [agentskills.io](https://agentskills.io)
`SKILL.md` format we ship — zero conversion.

| Harness | Global path | Project path |
|---|---|---|
| Claude Code | `~/.claude/skills/<name>/` | `<project>/.claude/skills/<name>/` |
| OpenAI Codex | `~/.agents/skills/<name>/` | `<project>/.agents/skills/<name>/` |
| Google Antigravity | `~/.gemini/config/skills/<name>/` | `<project>/.agents/skills/<name>/` (shared with Codex) |

## Sections registered (11 out of the box)

All sections have `entity_scope`, `identity` (single or compound), and where
appropriate an `editable_fields` allowlist for `op patch`. Adding a new one
is usually a config edit only.

| Endpoint | Scope | Identity | Notes |
|---|---|---|---|
| `/evidence/target-literature/` | `target_disease` | `[PMID]` | Literature refs; noise filters + abstract modal |
| `/evidence/target-pathway/` | `target` | `[image_url]` | Pathway figure diagrams |
| `/evidence/literature-supplementary-materials-analysis/` | `target_disease` | `[pmcid]` | LLM-analyzed PMC supp materials |
| `/evidence/target-literature-images/` | `target_disease` | `[image_url]` | Disease-specific pathway figures; nested `array_path = "results.pathways.{disease}"` |
| `/target-assessment/antibodies/` | `target` | `[Antibody, Provider]` | Compound key (same catalog # from multiple vendors) |
| `/target-assessment/assayability/` | `target` | `[ChEMBL ID]` | ChEMBL assays |
| `/target-assessment/orthologs/` | `target` | `[speciesId, homologue]` | Cross-species homology; compound key handles paralogs |
| `/target-assessment/geneEssentialityMap/` | `target` | `[tissueName]` | DepMap tissue-essentiality map |
| `/target-profile/ontology/` | `target` | `[GO ID]` | GO terms attributed to target |
| `/market-intelligence/target-pipeline/` | `target` | `[nct_id, Drug, Disease]` | Compound triple; handles basket trials (same NCT+Drug, different indications) and preclinical rows (empty nct_id) |

**Adding a new section?** See [docs/adding-sections.md](docs/adding-sections.md).
It has the data constraints to check, a copy-paste identity-uniqueness validator,
and the config-vs-code decision tree.

### Two entity-scope folders

- **`target_disease/<target>-<disease>.json`** — per-dossier data for a
  target × disease pair (e.g. `wars2-obesity.json`). Entity id includes the
  disease suffix.
- **`target/<target>.json`** — per-target data (e.g. `acvr2a.json`). Entity id
  is just the target symbol.

The CLI auto-picks `--type` from the section's `entity_scope` at runtime, so
you rarely need to pass `--type` explicitly.

## How it works (the model in one paragraph)

Every tracked entity has a `curation/<entity-type>/<id>/` folder containing:
immutable raw snapshots of the section (Base for 3-way merge), an append-only
`ops.jsonl` (scientist intent), `state.json` (mode + pointers), and per-refresh
changelogs. Each scientist action is one git commit, so `git blame` works on
every clinical decision. On every pipeline refresh, `entity ingest` takes a
fresh snapshot (= Theirs), runs a 3-way merge (Base + Theirs + ops → Merged),
and splices the merged section back into the served file. Scientist edits
survive across refreshes; pipeline updates land on top.

## One-time setup (per platform repo)

From the platform repo root:

```bash
dbtips-curate dev init \
    --path backend/res-immunology-automation/res_immunology_automation/src/scripts/curation
```

Drops `.dbtips-curate.toml` next to `cached_data_json/`, pre-registered with all
11 curated sections. Commit the toml to git so every scientist gets identical
config.

## Commands at a glance

```
dbtips-curate
  entity      list | status | track | snapshot | diff | ingest | records | compare | finalize
  op          delete | add | patch | undo
  history     ops | report
  dev         init | bootstrap | install-skill
```

Every command supports `--json` for machine-readable output, exits `0` / `1` / `2`
for success / runtime error / usage error, and follows the contract:

- Success: `{"ok": true, "data": {...}}`
- Error: `{"ok": false, "error": {"code": "...", "message": "...", "hint": "..."}}`

### `entity` — lifecycle

| Command | Purpose |
|---|---|
| `entity list [--type T]` | All tracked entities of one type with mode + last refresh. |
| `entity track <id> [--section S] [--skip-history] [--max-history-commits N]` | Bootstrap. **Walks git history** of the served file with `git log --follow --full-history --reverse` → synthesizes `delete` ops for removed records; flags historical additions for review. Uses `--full-history` to survive non-linear merge topologies (fixed in 0.3.1). `--skip-history` bypasses the walk. |
| `entity snapshot <id> [--section S] [--reason ...]` | Capture the current served file's section as an immutable raw snapshot. First snapshot becomes Base for 3-way merge. |
| `entity status <id> [--section S]` | Prioritized worklist: conflicts → historical_additions → bootstrap_candidates → resurrected → new from pipeline. Includes `suggested_filters` for greenfield noise triage where the section has them defined. |
| `entity records <id> [--section S] [--page N --page-size 20] [--full] [--fields F1,F2]` | List current records, paginated. Skills render as widget artifacts. |
| `entity diff <id> [--section S] [--emit-widget FILE]` | **Read-only preview** of the 3-way merge vs the current served file. Writes nothing. With `--emit-widget`, generates a self-contained HTML artifact with checkbox selection, per-row `View` + `Edit` modal, and `Apply deletions` batch button. |
| `entity compare <id> [--section S] [--page N --page-size N] [--fields ...]` | Side-by-side rows: previous (Base snapshot) vs current (served file). |
| `entity ingest <id> [--section S]` | Take fresh snapshot → 3-way merge → splice merged section back into served file. Writes changelog + conflicts file. This is what the pipeline hook calls. |
| `entity finalize <id> [--section S] [--no-push]` | Mark review complete: advance `last_reviewed_raw`, clear bootstrap_candidates, prune old snapshots, git commit + optional push. |

### `op` — scientist actions

Every op is a per-op git commit with full audit trail (SHA, author, reason).
All support `--dry-run`.

| Command | Purpose |
|---|---|
| `op delete <id> --section S --key K=V [--key K2=V2 ...] --reason "..."` | Tombstone a record. Compound-key sections pass one `--key` flag per identity field (e.g. `--key "Antibody=ab001" --key "Provider=Abcam"`). |
| `op add <id> --section S --record-file F --source "..."` | Add a record from an external source. Validates against the section's `required_fields`. Use `-` to read the record body from stdin. |
| `op patch <id> --section S --key K=V [--key K2=V2 ...] --set field=value [--set ...] --reason "..."` | Change one or more fields on an existing record. Values are JSON-decoded when possible (`Phase=3` → int, `Approved=true` → bool, `gene_symbols=["A","B"]` → list). Identity fields are rejected (`PATCH_TOUCHES_IDENTITY`) — use delete + re-add if you truly need to re-key. |
| `op undo <id> --undoes <op_id> --reason "..."` | Cancel a prior op. Endpoint is inferred from the target op — no `--section` flag. Rejects unknown op_ids, undo-of-undo, and double-undo with structured errors. The original op stays in the log for audit. |

### `history` — read-only views

| Command | Purpose |
|---|---|
| `history ops <id> [--since DATE] [--actor EMAIL]` | The full operation log with optional filters. |
| `history report <id> [--output FILE] [--open]` | Standalone HTML report (all queues + ops table). `--open` launches it in your default browser. |

### `dev` — setup + maintenance

| Command | Purpose |
|---|---|
| `dev init --path DIR` | Create the curation folder + write a default `.dbtips-curate.toml`. |
| `dev bootstrap [--targets claude,codex,antigravity\|all] [--project DIR]` | Install every skill into the chosen AI coding harnesses. Interactive menu if `--targets` omitted. `--project .` also drops skills into the current repo's workspace-scope skill folder. Idempotent. |
| `dev install-skill [--target claude\|codex\|antigravity] [--home DIR] [--only NAME]` | Single-target install primitive used by `bootstrap`. Prefer `bootstrap` for end-user setup. |

## Engine features

### Compound identity

Sections with multiple identity fields pass one `--key` per field. Examples:

- **Antibodies** — `--key "Antibody=ab001" --key "Provider=Abcam"` (same catalog
  number can be resold by multiple vendors).
- **Orthologs** — `--key "speciesId=9606" --key "homologue=ACVR2B"` (paralogs
  share species id).
- **Target-pipeline** — `--key "nct_id=NCT00752557" --key "Drug=DIBOTERMIN ALFA"
  --key "Disease=bone disease"` (basket trials share NCT+Drug across
  indications; preclinical rows have empty nct_id).

### Dotted + templated `array_path`

The `array_path` field on each section is walked as a dotted path and supports
a `{disease}` template that's substituted from the entity_id at runtime.
Example (target-literature-images, a nested-container section):

```toml
[sections."/evidence/target-literature-images/"]
entity_scope = "target_disease"
array_path = "results.pathways.{disease}"     # walks results → pathways → obesity for wars2-obesity
identity = ["image_url"]
```

Single-key `array_path` values still work unchanged (backward compatible).

### `editable_fields` allowlist

Each section can declare which fields `op patch` and the widget's Edit modal
expose. Identity fields are always excluded. Example (target-pipeline):

```toml
editable_fields = [
    "Phase", "Status", "Sponsor", "ApprovalStatus",
    "Mechanism of Action", "Disease", "Modality",
    "WhyStopped", "OutcomeStatus", "intervention_types",
]
```

### Brownfield detection via git history

`entity track` walks the served file's git history to detect prior scientist
edits (from Strapi, hand edits, other tools) and synthesizes matching ops so
those changes survive the next refresh. As of 0.3.1 the walker uses
`git log --follow --full-history --reverse`, which disables git's default
merge-based path-history simplification — the simplification silently drops
commits at merge points and previously caused record changes to be attributed
to the wrong commit in non-linear repo histories.

## Mode state machine

| Mode | Set by | Meaning |
|---|---|---|
| `greenfield` | `track` with no git-history record changes | Fresh pipeline output, no prior scientist edits detected. |
| `brownfield-from-git` | `track` when git history shows record-level adds/removes | Prior scientist edits detected via the git-history walk; synthesized `delete` ops + `historical_additions` queue surfaced. |
| `brownfield-detected` | `ingest` auto-promotion when pipeline drops Base records that aren't tombstoned | Records preserved as `bootstrap_candidates` pending scientist confirmation. |
| `brownfield-bootstrap` | Legacy label; behaves identically to `brownfield-detected` | Backward-compatible. |
| `active` | `finalize` after first review session | Steady-state operation. |

## Filesystem layout

```
backend/.../scripts/
  cached_data_json/                             (pipeline writes here, served to clients)
    target/<target>.json                        (target-scope dossiers)
    target_disease/<target>-<disease>.json      (target × disease dossiers)
  curation/                                     (this tool writes here, tracked in git)
    .dbtips-curate.toml                         (per-team config + section schema registry)
    target/<entity-id>/                         (or target_disease/<entity-id>/)
      raw/<ts>.json                             (immutable section snapshots)
      ops.jsonl                                 (append-only scientist operations)
      state.json                                (mode + last_reviewed_raw + git-history summary)
      conflicts/<ts>.json                       (open conflicts after last merge)
      changelog/<ts>.md                         (human-readable per-refresh summary)
      merged.json                               (post-merge section, spliced back into served file)
      reports/<ts>.html                         (HTML reports — created on demand)
```

## Three-way merge semantics

Each `entity ingest` runs:

```
Base    = snapshot at state.last_reviewed_raw     (what scientist last accepted)
Theirs  = fresh snapshot of current served file   (what pipeline just produced)
Ours    = apply_ops(Base, ops.jsonl)              (scientist intent layered on Base)

Merged  = three_way_merge(Base, Theirs, Ours)
```

Per identity-keyed record:

| In Base | In Theirs | Has op | Result |
|---|---|---|---|
| ✓ | ✗ | tombstone | silent (both agree to remove) |
| ✓ | ✗ | none | `removed_by_pipeline` queue — in brownfield modes, **preserved** as bootstrap candidate |
| ✗ | ✓ | none | `new_from_pipeline` queue — added to merged |
| ✗ | ✓ | tombstone | dropped from merged, logged as `resurrected_tombstone` |
| ✓ | ✓ | patch | per-field merge; pipeline-changed + scientist-patched same field → `conflict` (scientist wins by default) |
| ✗ | ✗ | add | scientist-added record, included in merged |

Ops are cancelled by `op undo`; the two-pass state computation in the merge
engine treats cancelled ops as if they never applied, without dropping later
ops on the same record/field.

## Skills catalog (11 SKILL.md files)

The `dev bootstrap` command installs all of these into your chosen harness(es).
Each is agentskills.io standard — a folder with a SKILL.md, YAML frontmatter,
markdown body.

| Skill | Section it drives |
|---|---|
| `dbtips-curate` | Umbrella router — resolves entity id + section, hands off. Owns CLI mechanics + universal ops (undo, finalize, widget batches). |
| `dbtips-curate-target-literature` | `/evidence/target-literature/` |
| `dbtips-curate-antibodies` | `/target-assessment/antibodies/` |
| `dbtips-curate-target-pathway` | `/evidence/target-pathway/` |
| `dbtips-curate-assayability` | `/target-assessment/assayability/` |
| `dbtips-curate-target-pipeline` | `/market-intelligence/target-pipeline/` |
| `dbtips-curate-ontology` | `/target-profile/ontology/` |
| `dbtips-curate-gene-essentiality` | `/target-assessment/geneEssentialityMap/` |
| `dbtips-curate-orthologs` | `/target-assessment/orthologs/` |
| `dbtips-curate-supplementary-materials` | `/evidence/literature-supplementary-materials-analysis/` |
| `dbtips-curate-literature-images` | `/evidence/target-literature-images/` (`results.pathways.<disease>` sub-array only) |

## Pipeline integration (build_dossier.py)

Already wired via a `CURATION HOOK` block at the top of `build_dossier.py`. Two
knobs:

```python
IS_REFRESH: bool = False        # True = clear section + redis before pipeline regen
CURATE_ENABLED: bool = True     # False = bypass the hook entirely (rollback)
```

With `IS_REFRESH = True`, the hook (in order):

1. Snapshots the current served file's section (pre-refresh Base)
2. Clears the section key from served JSON + redis
3. Lets the pipeline regenerate from scratch
4. Calls `entity ingest` → 3-way merge → splices merged back into served file

With `IS_REFRESH = False` (the safe default), only step 4 runs — keeps curation
state in sync without forcing regeneration.

## Driving from an AI coding harness

After `dev bootstrap`, in a session (Claude Code Desktop, Codex, or Antigravity):

```
QC ACVR2A target for antibodies
curate WARS2 for obesity literature images
review NPY1R pipeline
```

The umbrella skill:

1. Parses the target + section from what you said.
2. Asks explicitly if the section is missing — never guesses.
3. Resolves `entity_id` (`<target>` for target-scoped sections, `<target>-<disease>`
   for target × disease) and hands off to the matching section skill.
4. The section skill runs `entity status --json`, presents the worklist as an
   inline widget artifact, walks decisions with per-row `View`/`Edit`
   modals, and executes ops via the widget's `Apply` batch button.
5. On session end, calls `entity finalize` and summarizes.

### Harness compatibility (tested)

| Harness | Widget rendering | Notes |
|---|---|---|
| **Claude Code Desktop** | Inline, native artifact — first try | **Recommended.** Full support. |
| **OpenAI Codex** (tested with local Qwen on DGX) | Widget HTML redirected to browser (not inline). Occasionally requires up to 3 iterations before Codex opens the artifact correctly. | Works with friction. |
| **Google Antigravity** | Falls back to the Markdown renderer — no widget component. Data integrity is inconsistent. | Usable in limited cases. |

Requires a **Claude Pro subscription** for Claude Code Desktop. This release
ships SKILL.md files only — no MCP server — so users can't converse with it
in the browser at claude.ai.

## Testing

```bash
poetry install --with dev && poetry run pytest      # 76 tests
```

Test coverage:

- Three-way merge engine (base, brownfield, resurrected, conflicts)
- CLI end-to-end (`track` → `snapshot` → `ingest` → `op delete` → refresh →
  `ingest` → `finalize`)
- `op patch` compound-identity + JSON-typed values + basket-trial isolation
- `op undo` (delete/add/patch, undo-of-undo rejection, already-undone
  rejection)
- Git-history walk (linear + merge-with-side-branch topologies)
- Dotted + templated `array_path` for nested containers
- `--help` smoke tests on every subgroup (guards against typer/click ABI
  breaks)

## Version pinning notes

- `typer` `>=0.15.0,<1.0.0` — 0.12.x misparses options against click 8.4+.
- `click` `>=8.1,<8.2` — 8.2 changed `Parameter.make_metavar()` to require
  `ctx`, which breaks help rendering in every typer version we've tested.
- Both pins are baked into the published wheel; `uv tool install dbtips-curate`
  gets the working combo automatically.

## Status

- **Production-ready** for 11 sections across target and target × disease
  dossiers.
- 76/76 tests passing.
- Pipeline hook in `build_dossier.py` gated behind `IS_REFRESH` /
  `CURATE_ENABLED` flags.
- Skills published, Claude Code Desktop verified end-to-end.

## Roadmap

- **Field-level brownfield detection** — extend the git-history walker to also
  detect same-key field-level changes (currently identity-set only).
- **MCP server** — a `mcp serve` subcommand so the same skills can drive
  browser-based Claude conversations at claude.ai.
- **Singleton-dict sections** — `/target-assessment/targetability/`,
  `/target-assessment/tractability/`, and similar shape-mismatched endpoints.
  Design note lives at `src/dbtips_curate/skill/_deferred/targetability-design-note.md`.
- **`/genomics/evidence-heatmap/`** — wildcard `array_path` support for
  sections keyed by target symbol.
- **`op accept`** — CLI form for resolving conflicts by choice. Merge engine
  already handles the semantic; only the CLI wrapper is missing.

