Metadata-Version: 2.4
Name: dbtips-curate
Version: 0.5.0
Summary: Three-way merge curation for dbtips pipeline-generated dossier caches. Ships agentskills.io-compatible skills for Claude Code, OpenAI Codex, and Google Antigravity.
Author: Aganitha
Author-email: dev@aganitha.ai
Requires-Python: >=3.11,<4.0
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Requires-Dist: click (>=8.1,<8.2)
Requires-Dist: pydantic (>=2.6.0,<3.0.0)
Requires-Dist: rich (>=13.7.0,<14.0.0)
Requires-Dist: typer (>=0.15.0,<1.0.0)
Description-Content-Type: text/markdown

# dbtips-curate

**Three-way merge curation for dbtips pipeline-generated dossier JSON caches.**

Scientists hand-curate cache files after pipeline generation (deleting noise,
adding records from external sources, patching status fields). When the
pipeline refreshes, those edits get wiped. `dbtips-curate` preserves scientist
intent as an append-only git-tracked operation log and replays it onto every
refresh — so the merged served file is always "fresh pipeline data + scientist
intent" without manual re-curation.

Published on [PyPI](https://pypi.org/project/dbtips-curate/) as `dbtips-curate`.
Latest version: **0.4.0**.

## What's new in 0.4.0

- **Multi-section per entity** (breaking change to on-disk layout). One target
  or target × disease can now be tracked across many sections simultaneously.
  `state.json` v2 shape holds a `{sections: {endpoint: {...}}}` dict rather
  than a single section. Snapshots and conflicts are per-section.
- **New commit strategy**: `op delete` / `op add` / `op patch` / `op undo` no
  longer create per-op git commits. `entity ingest` commits everything from
  the cycle as one atomic commit with a summary message. Keeps git history
  proportional to review cycles.
- **New command `entity progress <target>`**: aggregates per-section state
  across scopes and returns `reviewed / in_progress / pending` buckets. Drives
  guided multi-section QC flows from AI coding harnesses.
- **New command `dev migrate`**: one-shot v1→v2 on-disk migration for
  existing curation folders. `--dry-run` previews. Backups originals to
  `.backup-v1/`.
- **`merged.json` deleted** — was write-only, redundant with the served file
  after splice.
- **`changelog/` folder collapsed to single `changelog.md`** at entity root,
  appended per ingest with `## <endpoint> · <ts>` headers.
- **New section registered**: `/evidence/search-patent/` (target × disease
  scope, identity=`[publication_number]`, editable fields for `assignee` /
  grant/expiry/priority dates). Includes two noise filters (`expired_patent`,
  `no_grant_date`).
- **New skill**: `dbtips-curate-patents` for the search-patent section, and
  a "Multi-section session flow" paragraph added to the umbrella skill.

Migration for existing users:
```bash
uv tool install --force --refresh dbtips-curate
dbtips-curate dev migrate --dry-run              # preview
dbtips-curate dev migrate                        # apply
```

## Install (end user)

```bash
# 1. Install the CLI (persistent global tool)
uv tool install --force --refresh dbtips-curate
# or:  pipx install dbtips-curate
# or:  pip install dbtips-curate  (into a venv)

# 2. Bootstrap SKILL.md files into your AI coding harness(es)
dbtips-curate dev bootstrap
```

`dev bootstrap` shows an interactive menu (Claude / Codex / Antigravity / all
combinations). Non-interactive form:

```bash
dbtips-curate dev bootstrap --targets all
dbtips-curate dev bootstrap --targets claude,codex
dbtips-curate dev bootstrap --targets claude --project .    # also install to <repo>/.claude/skills/
```

**Update path:** `uv tool install --force --refresh dbtips-curate` picks up
the latest PyPI release. `--refresh` is important — uv caches its resolver
decisions and will happily reinstall an old version without it.

### Multi-harness skill install matrix

Every supported harness reads the same [agentskills.io](https://agentskills.io)
`SKILL.md` format we ship — zero conversion.

| Harness | Global path | Project path |
|---|---|---|
| Claude Code | `~/.claude/skills/<name>/` | `<project>/.claude/skills/<name>/` |
| OpenAI Codex | `~/.agents/skills/<name>/` | `<project>/.agents/skills/<name>/` |
| Google Antigravity | `~/.gemini/config/skills/<name>/` | `<project>/.agents/skills/<name>/` (shared with Codex) |

## Sections registered (12 out of the box)

All sections have `entity_scope`, `identity` (single or compound), and where
appropriate an `editable_fields` allowlist for `op patch`.

| Endpoint | Scope | Identity | Notes |
|---|---|---|---|
| `/evidence/target-literature/` | `target_disease` | `[PMID]` | Literature refs; noise filters + abstract modal |
| `/evidence/target-pathway/` | `target` | `[image_url]` | Pathway figure diagrams |
| `/evidence/literature-supplementary-materials-analysis/` | `target_disease` | `[pmcid]` | LLM-analyzed PMC supp materials |
| `/evidence/target-literature-images/` | `target_disease` | `[image_url]` | Disease-specific pathway figures; nested `array_path = "results.pathways.{disease}"` |
| `/evidence/search-patent/` | `target_disease` | `[publication_number]` | Patent landscape; noise filters for expired and no-grant patents |
| `/target-assessment/antibodies/` | `target` | `[Antibody, Provider]` | Compound key (same catalog # from multiple vendors) |
| `/target-assessment/assayability/` | `target` | `[ChEMBL ID]` | ChEMBL assays |
| `/target-assessment/orthologs/` | `target` | `[speciesId, homologue]` | Cross-species homology; compound key handles paralogs |
| `/target-assessment/geneEssentialityMap/` | `target` | `[tissueName]` | DepMap tissue-essentiality map |
| `/target-profile/ontology/` | `target` | `[GO ID]` | GO terms attributed to target |
| `/market-intelligence/target-pipeline/` | `target` | `[nct_id, Drug, Disease]` | Compound triple; handles basket trials and preclinical rows |

Two entity-scope folders on disk:

- **`target_disease/<target>-<disease>.json`** — target × disease dossiers
- **`target/<target>.json`** — target-only dossiers

The CLI auto-picks `--type` from the section's `entity_scope` at runtime, so
you rarely need to pass `--type` yourself.

**Adding a new section?** See [docs/adding-sections.md](docs/adding-sections.md).
It has the data constraints to check, a copy-paste identity-uniqueness validator,
and the config-vs-code decision tree.

## How it works (the model in one paragraph)

Every tracked entity has a `curation/<entity-type>/<id>/` folder. Inside it,
`state.json` holds a per-section cursor (mode + Base pointer + queues awaiting
decision). `ops.jsonl` is an append-only log of scientist decisions across all
sections. `raw/` holds section-slugged snapshots (Base + latest per section).
`conflicts/<section-slug>.json` holds the latest conflicts per section.
`changelog.md` is a single human-readable log appended per ingest with
`## <endpoint> · <ts>` headers. Each `entity ingest` cycle takes a fresh
snapshot (= Theirs), runs a 3-way merge (Base + Theirs + ops → Merged),
splices the merged section back into the served file, then commits everything
in one atomic git commit summarizing the ops applied.

## One-time setup (per platform repo)

From the platform repo root:

```bash
dbtips-curate dev init \
    --path backend/res-immunology-automation/res_immunology_automation/src/scripts/curation
```

Drops `.dbtips-curate.toml` next to `cached_data_json/`, pre-registered with all
12 curated sections. Commit the toml to git so every scientist gets identical
config.

## Commands at a glance

```
dbtips-curate
  entity      list | progress | status | track | snapshot | diff | ingest | records | compare | finalize
  op          delete | add | patch | undo
  history     ops | report
  dev         init | migrate | bootstrap | install-skill
```

Every command supports `--json` for machine-readable output, exits `0` / `1` / `2`
for success / runtime error / usage error, and follows the contract:

- Success: `{"ok": true, "data": {...}}`
- Error: `{"ok": false, "error": {"code": "...", "message": "...", "hint": "..."}}`

### `entity` — lifecycle

| Command | Purpose |
|---|---|
| `entity list [--type T]` | All tracked entities of one type. Now shows every section tracked per entity (multi-section-aware). |
| `entity progress <target>` | **NEW.** Per-target QC progress across scopes: `reviewed` (mode=active), `in_progress` (has ops but not finalized), `pending` (registered but never tracked). Drives guided multi-section flows from AI harnesses. |
| `entity track <id> [--section S] [--skip-history] [--max-history-commits N]` | Bootstrap. Walks git history of the served file with `git log --follow --full-history --reverse`. Synthesizes `delete` ops for removed records; flags historical additions for review. `--skip-history` bypasses the walk. Multi-section-aware: tracking a new section on an already-tracked entity is now supported. |
| `entity snapshot <id> [--section S] [--reason ...]` | Capture the current served file's section as an immutable raw snapshot. |
| `entity status <id> [--section S]` | Prioritized worklist: conflicts → historical_additions → bootstrap_candidates → resurrected → new from pipeline. |
| `entity records <id> [--section S] [--page N --page-size 20] [--full] [--fields F1,F2]` | List current records, paginated. Widget-friendly. |
| `entity diff <id> [--section S] [--emit-widget FILE]` | Read-only preview of the 3-way merge. Writes nothing. With `--emit-widget`, generates a self-contained HTML artifact. |
| `entity compare <id> [--section S] [--page N --page-size N] [--fields ...]` | Side-by-side rows: previous (Base snapshot) vs current (served file). |
| `entity ingest <id> [--section S]` | Take fresh snapshot → 3-way merge → splice merged section back into served file. **In 0.4.0, this is the ONLY command that creates a git commit** — one atomic commit per ingest cycle summarizing all ops applied. |
| `entity finalize <id> [--section S] [--no-push]` | Mark review complete: advance `last_reviewed_raw`, clear bootstrap_candidates, prune old snapshots, git commit + optional push. |

### `op` — scientist actions

Every op is an append-only line in `ops.jsonl`. **In 0.4.0, ops no longer commit
per op** — they accumulate as staged filesystem changes and are committed
together by the next `entity ingest`. All support `--dry-run`.

| Command | Purpose |
|---|---|
| `op delete <id> --section S --key K=V [--key K2=V2 ...] --reason "..."` | Tombstone a record. Compound-key sections pass one `--key` flag per identity field. |
| `op add <id> --section S --record-file F --source "..."` | Add a record from an external source. Validates against the section's `required_fields`. |
| `op patch <id> --section S --key K=V [--key K2=V2 ...] --set field=value [--set ...] --reason "..."` | Change one or more fields on an existing record. Values are JSON-decoded when possible. Identity fields are rejected (`PATCH_TOUCHES_IDENTITY`). |
| `op undo <id> --undoes <op_id> --reason "..."` | Cancel a prior op. Endpoint is inferred from the target op — no `--section` flag. |

### `history` — read-only views

| Command | Purpose |
|---|---|
| `history ops <id> [--since DATE] [--actor EMAIL]` | The full operation log with optional filters. |
| `history report <id> [--output FILE] [--open]` | Standalone HTML report (all queues + ops table). `--open` launches it in your browser. |

### `dev` — setup + maintenance

| Command | Purpose |
|---|---|
| `dev init --path DIR` | Create the curation folder + write a default `.dbtips-curate.toml`. |
| `dev migrate [--path DIR] [--dry-run]` | **NEW.** One-shot v1→v2 on-disk migration. Rewrites `state.json` to nested shape, renames snapshots/conflicts with section slugs, concatenates changelogs into `changelog.md`, deletes `merged.json`. Backs up originals to `.backup-v1/`. Idempotent. |
| `dev bootstrap [--targets claude,codex,antigravity\|all] [--project DIR]` | Install every skill into the chosen AI coding harnesses. Interactive menu if `--targets` omitted. |
| `dev install-skill [--target claude\|codex\|antigravity] [--home DIR] [--only NAME]` | Single-target install primitive used by `bootstrap`. |

## Engine features

### Compound identity

Sections with multiple identity fields pass one `--key` per field. Examples:

- **Antibodies** — `--key "Antibody=ab001" --key "Provider=Abcam"` (same catalog
  number can be resold by multiple vendors).
- **Orthologs** — `--key "speciesId=9606" --key "homologue=ACVR2B"` (paralogs
  share species id).
- **Target-pipeline** — `--key "nct_id=NCT00752557" --key "Drug=DIBOTERMIN ALFA"
  --key "Disease=bone disease"` (basket trials share NCT+Drug across
  indications; preclinical rows have empty nct_id).

### Dotted + templated `array_path`

The `array_path` field on each section is walked as a dotted path and supports
a `{disease}` template that's substituted from the entity_id at runtime.
Example (target-literature-images, a nested-container section):

```toml
[sections."/evidence/target-literature-images/"]
entity_scope = "target_disease"
array_path = "results.pathways.{disease}"     # walks results → pathways → obesity for wars2-obesity
identity = ["image_url"]
```

Single-key `array_path` values still work unchanged (backward compatible).

### `editable_fields` allowlist

Each section can declare which fields `op patch` and the widget's Edit modal
expose. Identity fields are always excluded. Example (target-pipeline):

```toml
editable_fields = [
    "Phase", "Status", "Sponsor", "ApprovalStatus",
    "Mechanism of Action", "Disease", "Modality",
    "WhyStopped", "OutcomeStatus", "intervention_types",
]
```

### Brownfield detection via git history

`entity track` walks the served file's git history to detect prior scientist
edits (from Strapi, hand edits, other tools) and synthesizes matching ops so
those changes survive the next refresh. As of 0.3.1 the walker uses
`git log --follow --full-history --reverse`, which disables git's default
merge-based path-history simplification — the simplification silently drops
commits at merge points and previously caused record changes to be attributed
to the wrong commit in non-linear repo histories.

### Per-ingest commit strategy (v0.4.0)

Prior to 0.4.0, every scientist op (delete / add / patch / undo) created its
own git commit. In a real review session with dozens of decisions, that
polluted history quickly. As of 0.4.0:

- **Ops append to `ops.jsonl` but do NOT commit.** The line is staged as an
  uncommitted change.
- **`entity ingest` commits everything as one atomic commit** — `ops.jsonl` +
  section-slugged snapshots + `changelog.md` + `conflicts/<slug>.json` + updated
  `state.json` — with a summary message: `curate(<entity>): ingest <endpoint>
  @ <ts> · delete=X, patch=Y, add=Z, undo=W`.
- Git history stays proportional to review cycles, not to individual clicks.
- Each op still has its own line in `ops.jsonl` for granular audit; only the
  git commit is consolidated.
- Undo semantics are unchanged — you undo by `op_id` from `ops.jsonl`, regardless
  of when the underlying op was committed.

## Mode state machine

| Mode | Set by | Meaning |
|---|---|---|
| `greenfield` | `track` with no git-history record changes | Fresh pipeline output, no prior scientist edits detected. |
| `brownfield-from-git` | `track` when git history shows record-level adds/removes | Prior scientist edits detected via the git-history walk. |
| `brownfield-detected` | `ingest` auto-promotion when pipeline drops Base records that aren't tombstoned | Records preserved as `bootstrap_candidates` pending scientist confirmation. |
| `brownfield-bootstrap` | Legacy label; behaves identically to `brownfield-detected` | Backward-compatible. |
| `active` | `finalize` after first review session | Steady-state operation. |

## Filesystem layout (v0.4.0)

```
backend/.../scripts/
  cached_data_json/                             (pipeline writes here)
    target/<target>.json                        (target-only dossiers)
    target_disease/<target>-<disease>.json      (target × disease dossiers)

  curation/                                     (this tool writes here, tracked in git)
    .dbtips-curate.toml                         (per-team config + section schema registry)
    target/<entity-id>/                         (or target_disease/<entity-id>/)
      state.json                                (nested per-section cursor state — v2 shape)
      ops.jsonl                                 (mixed sections, endpoint field per line)
      changelog.md                              (mixed sections, appended per ingest)
      raw/
        <section-slug>__<ts>.json               (Base + latest per section)
      conflicts/
        <section-slug>.json                     (latest only per section, overwritten)
```

### state.json v2 shape

```json
{
  "schema_version": 2,
  "entity_id": "acvr2a",
  "entity_type": "target",
  "sections": {
    "/target-profile/ontology/": {
      "mode": "active",
      "tracked_at": "...",
      "last_reviewed_raw": "...",
      "bootstrap_candidates": [],
      "historical_additions": [],
      "git_history_summary": {...},
      "last_committed_op_id": "..."
    },
    "/market-intelligence/target-pipeline/": {...}
  }
}
```

## Three-way merge semantics

Each `entity ingest` runs:

```
Base    = snapshot at state.sections[endpoint].last_reviewed_raw   (what scientist last accepted)
Theirs  = fresh snapshot of current served file                    (what pipeline just produced)
Ours    = apply_ops(Base, ops.jsonl)                               (scientist intent layered on Base)

Merged  = three_way_merge(Base, Theirs, Ours)
```

Per identity-keyed record:

| In Base | In Theirs | Has op | Result |
|---|---|---|---|
| ✓ | ✗ | tombstone | silent (both agree to remove) |
| ✓ | ✗ | none | `removed_by_pipeline` queue — in brownfield modes, **preserved** as bootstrap candidate |
| ✗ | ✓ | none | `new_from_pipeline` queue — added to merged |
| ✗ | ✓ | tombstone | dropped from merged, logged as `resurrected_tombstone` |
| ✓ | ✓ | patch | per-field merge; pipeline-changed + scientist-patched same field → `conflict` (scientist wins by default) |
| ✗ | ✗ | add | scientist-added record, included in merged |

Ops are cancelled by `op undo`; the two-pass state computation in the merge
engine treats cancelled ops as if they never applied, without dropping later
ops on the same record/field.

## Skills catalog (12 SKILL.md files)

The `dev bootstrap` command installs all of these into your chosen harness(es).
Each is agentskills.io standard — a folder with a SKILL.md, YAML frontmatter,
markdown body.

| Skill | Section it drives |
|---|---|
| `dbtips-curate` | Umbrella router — resolves entity id + section, hands off. Owns CLI mechanics + universal ops (undo, finalize, widget batches). Contains the multi-section session flow paragraph that calls `entity progress`. |
| `dbtips-curate-target-literature` | `/evidence/target-literature/` |
| `dbtips-curate-antibodies` | `/target-assessment/antibodies/` |
| `dbtips-curate-target-pathway` | `/evidence/target-pathway/` |
| `dbtips-curate-assayability` | `/target-assessment/assayability/` |
| `dbtips-curate-target-pipeline` | `/market-intelligence/target-pipeline/` |
| `dbtips-curate-ontology` | `/target-profile/ontology/` |
| `dbtips-curate-gene-essentiality` | `/target-assessment/geneEssentialityMap/` |
| `dbtips-curate-orthologs` | `/target-assessment/orthologs/` |
| `dbtips-curate-supplementary-materials` | `/evidence/literature-supplementary-materials-analysis/` |
| `dbtips-curate-literature-images` | `/evidence/target-literature-images/` (`results.pathways.<disease>` sub-array only) |
| `dbtips-curate-patents` | `/evidence/search-patent/` |

## Pipeline integration (build_dossier.py)

Already wired via a `CURATION HOOK` block at the top of `build_dossier.py`. Two
knobs:

```python
IS_REFRESH: bool = False        # True = clear section + redis before pipeline regen
CURATE_ENABLED: bool = True     # False = bypass the hook entirely (rollback)
```

With `IS_REFRESH = True`, the hook (in order):

1. Snapshots the current served file's section (pre-refresh Base)
2. Clears the section key from served JSON + redis
3. Lets the pipeline regenerate from scratch
4. Calls `entity ingest` → 3-way merge → splices merged back into served file

With `IS_REFRESH = False` (the safe default), only step 4 runs.

## Driving from an AI coding harness

After `dev bootstrap`, in a session (Claude Code Desktop, Codex, or Antigravity):

```
QC ACVR2A target for antibodies
curate WARS2 for obesity literature images
review NPY1R pipeline
QC ACVR2A   # (multi-section session — asks entity progress, walks pending sections)
```

The umbrella skill:

1. Parses the target + section from what you said.
2. Asks explicitly if the section is missing — never guesses.
3. For a full-dossier ask, runs `entity progress <target>` and presents three
   grouped tables (reviewed / in progress / pending), then asks which section
   to do next.
4. Resolves `entity_id` and hands off to the matching section skill.
5. The section skill runs `entity status --json`, presents an inline widget,
   walks decisions with per-row `View`/`Edit` modals, and executes ops via
   the widget's `Apply` batch button.
6. When the section skill returns control, the umbrella re-runs
   `entity progress` and repeats step 3.

### Harness compatibility (tested)

| Harness | Widget rendering | Notes |
|---|---|---|
| **Claude Code Desktop** | Inline, native artifact — first try | **Recommended.** Full support. |
| **OpenAI Codex** (tested with local Qwen on DGX) | Widget HTML redirected to browser (not inline). Occasionally requires up to 3 iterations before Codex opens the artifact correctly. | Works with friction. |
| **Google Antigravity** | Falls back to the Markdown renderer — no widget component. Data integrity is inconsistent. | Usable in limited cases. |

Requires a **Claude Pro subscription** for Claude Code Desktop. This release
ships SKILL.md files only — no MCP server — so users can't converse with it
in the browser at claude.ai.

## Testing

```bash
poetry install --with dev && poetry run pytest      # 87 tests
```

Test coverage:

- Three-way merge engine (base, brownfield, resurrected, conflicts)
- CLI end-to-end (`track` → `snapshot` → `ingest` → `op delete` → refresh →
  `ingest` → `finalize`)
- `op patch` compound-identity + JSON-typed values + basket-trial isolation
- `op undo` (delete/add/patch, undo-of-undo rejection, already-undone rejection)
- Git-history walk (linear + merge-with-side-branch topologies)
- Dotted + templated `array_path` for nested containers
- v0.4.0 features: multi-section per entity, per-ingest commit strategy,
  v1→v2 migration + startup guard, `entity progress` bucketing
- `--help` smoke tests on every subgroup (guards against typer/click ABI breaks)

## Version pinning notes

- `typer` `>=0.15.0,<1.0.0` — 0.12.x misparses options against click 8.4+.
- `click` `>=8.1,<8.2` — 8.2 changed `Parameter.make_metavar()` to require
  `ctx`, which breaks help rendering in every typer version we've tested.
- Both pins are baked into the published wheel; `uv tool install dbtips-curate`
  gets the working combo automatically.

## Status

- **Production-ready** for 12 sections across target and target × disease
  dossiers.
- 87/87 tests passing.
- Multi-section tracking + per-ingest commit strategy validated end-to-end.
- Skills published, Claude Code Desktop verified end-to-end.

## Roadmap

- **Field-level brownfield detection** — extend the git-history walker to also
  detect same-key field-level changes (currently identity-set only).
- **MCP server** — a `mcp serve` subcommand so the same skills can drive
  browser-based Claude conversations at claude.ai.
- **Singleton-dict sections** — `/target-assessment/targetability/`,
  `/target-assessment/tractability/`, and similar shape-mismatched endpoints.
  Design note lives at `src/dbtips_curate/skill/_deferred/targetability-design-note.md`.
- **`/genomics/evidence-heatmap/`** — wildcard `array_path` support for
  sections keyed by target symbol.
- **`op accept`** — CLI form for resolving conflicts by choice. Merge engine
  already handles the semantic; only the CLI wrapper is missing.

