Metadata-Version: 2.4
Name: iatro-clintrace
Version: 0.0.1.dev2026070802
Summary: ClinTrace clinical-course compression, grounding, and longitudinal decision modeling
License: CC-BY-4.0
Project-URL: Homepage, https://github.com/iatrode/iatro-clintrace
Project-URL: Repository, https://github.com/iatrode/iatro-clintrace
Project-URL: Issues, https://github.com/iatrode/iatro-clintrace/issues
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: brotli>=1.2.0
Requires-Dist: iatro-base-iac>=0.0.3
Requires-Dist: imagecodecs>=2024.1.1
Requires-Dist: numpy>=2.0
Requires-Dist: openai>=1.40
Requires-Dist: peft>=0.17
Requires-Dist: pyarrow>=24.0.0
Requires-Dist: scipy>=1.11
Requires-Dist: torch>=2.6
Requires-Dist: transformers>=4.57
Provides-Extra: train
Requires-Dist: accelerate>=1.10; extra == "train"
Requires-Dist: modelscope; extra == "train"
Requires-Dist: optuna>=4.0; extra == "train"
Requires-Dist: PyYAML>=6.0; extra == "train"
Requires-Dist: tensorboard>=2.16; extra == "train"
Requires-Dist: tqdm>=4.66; extra == "train"
Provides-Extra: serve
Requires-Dist: vllm>=0.6.0; extra == "serve"
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Dynamic: license-file

# Iatro ClinTrace

ClinTrace is a longitudinal clinical-course model for HCC notes. Its job is to
turn free-text clinical notes into a reusable course representation that can
support treatment-pathway reasoning, evidence-grounded decision completion, and
downstream model integration.

**Research use only.** ClinTrace is not a medical device and must not be used
for clinical diagnosis, treatment selection, triage, or patient management.

The authoritative design record is
[`docs/clintrace_total_design.md`](docs/clintrace_total_design.md). This README
is the open-source entrypoint: it describes the package boundary, public command
surface, and controlled-asset contract.

## Table of Contents

- [Scientific Contract](#scientific-contract)
- [Components](#components)
- [Release Boundary](#release-boundary)
- [Installation](#installation)
- [Training Workflow](#training-workflow)
  - [1. Build Source IAC Assets](#1-build-source-iac-assets)
  - [2. Build or Import Compressed Text Assets](#2-build-or-import-compressed-text-assets)
  - [3. Build and Check Paired IAC Assets](#3-build-and-check-paired-iac-assets)
  - [4. Train, Select, and Export the Compressor](#4-train-select-and-export-the-compressor)
  - [5. Build Frozen Note Features](#5-build-frozen-note-features)
  - [6. Build Decision Supervision](#6-build-decision-supervision)
  - [7. Build Direct-Token and Decoder Planes](#7-build-direct-token-and-decoder-planes)
  - [8. Build History-State and Decision-Contrast Assets](#8-build-history-state-and-decision-contrast-assets)
  - [9. Train the ClinTrace Decision Expert](#9-train-the-clintrace-decision-expert)
  - [10. Evaluate and Freeze Release Candidates](#10-evaluate-and-freeze-release-candidates)
- [Configuration](#configuration)
- [Package Layout](#package-layout)
- [Open-Source Hygiene](#open-source-hygiene)
- [Optional LLM Client Utilities](#optional-llm-client-utilities)

## Scientific Contract

ClinTrace learns from clinical notes, not from manually supplied decision-chain
labels at inference time.

```text
input
  Stage-A/tokenized note representations
  observable note title/type text
  natural longitudinal order

supervision and audit
  LLM-extracted E/P/X/O chains
  timeline-derived labels
  history-state summaries

output
  course expert records
  grounded evidence spans
  ranked open-world decision candidates
```

E/P/X/O chains are supervision and evaluation assets. They are not deployment
inputs. Timeline-derived fields such as document role, label type, pathway
labels, and decision-bearing flags are also not model inputs in the formal
expert, because they are not guaranteed to exist when a user sends a new note.

The central modeling target is:

```text
history state + observed clinical evidence -> plausible P/X/O decision candidates
```

This is intentionally open-world. Missing labels in the record are treated as
unobserved, not as clinical negatives. The model is evaluated primarily by
whether documented decisions are ranked highly and whether generated evidence
and decision spans are clinically inspectable.

## Components

```text
iatro.clintrace.compressor
  Single-note compression into fixed latent note tokens.

iatro.clintrace.routing
  Compactness router used before tokenization to choose compression vs direct
  token representation.

iatro.clintrace.grounding
  Full-note EXPO grounding: evidence, plan, executed action, and outcome span
  supervision from extracted decision chains.

iatro.clintrace.completion
  Observed-evidence decision completion within a note.

iatro.clintrace.longitudinal
  History-state-conditioned decision completion and decision-contrast diagnostics.

iatro.clintrace.timeline
  Patient-level chronological note indexing.

iatro.clintrace.inspect
  Local IAC asset browsers for development and audit.
```

## Release Boundary

This GitHub repository contains source code, configuration, prompt contracts,
tests, and documentation only. Clinical data, extracted chains, generated
caches, model weights, evaluation predictions, API credentials, and training
outputs are not part of the GitHub repository.

Model and auxiliary binary assets are produced by the controlled full
training/validation pipeline for a separate gated Hugging Face release. They
must not be committed to GitHub, including through Git LFS.

Tracked GitHub content:

```text
src/                             Python package
configs/                         reproducible training/evaluation configs
prompts/                         versioned LLM prompt contracts
docs/                            design and method notes
tests/                           unit tests
README.md, LICENSE, pyproject.toml
```

Expected local asset classes:

```text
data/00_source/                  raw clinical_text IAC packs
data/01_compressed/              compressed note tables keyed by doc_id
data/02_paired/                  source/compressed paired IAC packs
data/03_timeline_index/          longitudinal note indexes
data/04_clintrace_features/      frozen compressor feature caches
data/05_decision_chains/         extracted decision-chain JSONL
data/06_decision_supervision/    compiled EXPO supervision assets
data/07_grounding_features/
                                 tokenizer/direct-note feature caches
data/08_patient_state_history/   history-state text summaries
runs/                            ignored training/evaluation run outputs
artifacts/compressor/            selected local compressor artifacts
artifacts/clintrace/             selected local ClinTrace planes and auxiliaries
artifacts/hf/clintrace/          gated Hugging Face release tree
dist/                            ignored release/build output directory
```

Legacy local caches may still use older non-contiguous directory numbers. The
public workflow uses the contiguous layout above.

`data/`, `artifacts/`, `runs/`, `result/` legacy outputs, and `dist/` are
ignored by git.
Public examples should use de-identified toy data or externally releasable
fixtures only.

Gated Hugging Face artifacts are generated only as part of the complete
controlled training and validation chain. Do not create or commit ad hoc local
asset bundles from intermediate results.

## Installation

Use the shared project environment.

```bash
pip install -e .
pip install -e ".[train]"
pip install -e ".[serve]"
pip install -e ".[dev]"
```

Dependency groups:

```text
.[train]  training, cache generation, and compression-data preparation
.[serve]  optional vLLM-compatible local serving
.[dev]    tests and packaging checks
```

## Training Workflow

The public package exposes the same component order used by the full training
run. All commands below read and write ignored local assets under `data/`,
`runs/`, and `artifacts/`; those outputs are never committed to GitHub.

### 1. Build Source IAC Assets

Start from a de-identified document table in JSONL, CSV, or TSV. Required
columns are:

```text
patient_id      stable de-identified patient identifier
doc_id          stable de-identified document identifier, unique within dataset
text            clinical note text
```

Optional columns are:

```text
doc_type        observable note title/type
doc_timestamp   sortable clinical timestamp string
episode_index   integer visit/admission index, -1 if unknown
encounter       optional encounter identifier
doc_category    optional source category, defaults to 病案文书
source          optional source-system label
```

Pack the table into a clinical_text IAC:

```bash
clintrace build source-iac \
  --input data/source_documents/site_a.jsonl \
  --output data/00_source/site_a.iac \
  --institution site_a \
  --overwrite
```

The source pack builder sorts documents within each patient by
`episode_index`, `doc_timestamp`, and `doc_id`. It skips rows with missing
required fields, duplicate `doc_id`, or empty text. It reports:

```text
input_rows=<input table rows>
patients=<packed patient count>
docs=<packed document count>
duplicate_doc_ids=<skipped duplicate document ids>
empty_text=<skipped empty documents>
missing_required=<missing required-field counts>
text_raw_mb=<UTF-8 text payload size>
output=<source IAC path> size_mb=<written pack size>
```

Verify the source pack:

```bash
clintrace build source-iac \
  --input data/source_documents/site_a.jsonl \
  --output data/00_source/site_a.iac \
  --institution site_a \
  --verify
```

Verification must report:

```text
verify_docs == verify_expected_docs
verify_unique_doc_ids == verify_expected_docs
verify_mismatches == 0
```

The resulting asset is:

```text
data/00_source/{site}.iac
  clinical_text pack containing original de-identified notes.
```

### 2. Build or Import Compressed Text Assets

Create or import compressed note text before compressor training. This asset is
a normal JSONL, CSV, or TSV table, not an IAC pack. Required columns are:

```text
doc_id             same de-identified document identifier as source IAC
compressed_text    compressed clinical note text
```

Accepted aliases for `compressed_text` are `text` and `output`. Optional
columns are `doc_type`, `doc_timestamp`, and `episode_index`; when omitted, the
source IAC metadata is used.

```text
data/01_compressed/{site}.jsonl
  compressed-note table keyed by doc_id.
```

The compression method is outside this public contract. It may be produced by a
human-reviewed pipeline, a local model, or an external system; ClinTrace only
requires aligned `doc_id` values and clinically readable compressed text. If a
dataset requires de-identification or identifier alignment, complete that data
preparation before building ClinTrace pairs. The open-source package does not
define or distribute private identifier-mapping protocols. For the optional
LLM client wrapper that uses `prompts/compressor_zh.md` and writes a
`paired-iac`-ready JSONL, see
[Optional LLM Client Utilities](#optional-llm-client-utilities).

### 3. Build and Check Paired IAC Assets

Once the source IAC and compressed table share de-identified document
identifiers, build the source/compressed training pairs:

```text
data/02_paired/{site}.iac
  source/compressed clinical_text_pair pack used by the compressor and router.
```

Build a paired source/compressed pack:

```bash
clintrace build paired-iac \
  --source-iac data/00_source/site_a.iac \
  --compressed-input data/01_compressed/site_a.jsonl \
  --output data/02_paired/site_a.iac \
  --institution site_a \
  --overwrite
```

`paired-iac` is the only command that writes compressed text into an IAC asset.
There is no standalone compressed IAC in the public workflow. If a dataset
needs de-identification or identifier mapping to make source and compressed IDs
align, complete that preparation before `paired-iac`. The paired IAC itself is
source text paired with compressed text; it is not a
de-identified/non-de-identified pair.

Record the following IAC-level acceptance metrics before model training:

```text
patients=<paired patient count>
paired_docs=<paired document count>
source_docs=<source document count>
compressed_rows=<compressed input table rows>
compressed_docs=<unique usable compressed document count>
duplicate_doc_ids=<skipped duplicate compressed document ids>
missing_compressed=<source docs without compressed counterpart>
empty_source=<skipped empty source docs>
empty_compressed=<skipped empty compressed docs>
missing_required=<missing required-field counts>
source_raw_mb=<UTF-8 source text payload size>
compressed_raw_mb=<UTF-8 compressed text payload size>
output=<paired IAC path> size_mb=<written pack size>
```

Verify the paired pack:

```bash
clintrace build paired-iac \
  --source-iac data/00_source/site_a.iac \
  --compressed-input data/01_compressed/site_a.jsonl \
  --output data/02_paired/site_a.iac \
  --institution site_a \
  --verify
```

Verification must report:

```text
verify_docs == verify_expected_docs
verify_unique_doc_ids == verify_expected_docs
verify_mismatches == 0
```

Then inspect both the raw pack and paired pack before starting a long run:

```bash
clintrace inspect iac data/00_source/site_a.iac --head 5
clintrace inspect pairs -i site_a --head 5
```

The IAC acceptance gate is: all expected documents are paired exactly once,
source/compressed text round-trips without mismatch, document timestamps and
episode indexes are populated enough for timeline construction, and sampled
records are clinically readable after de-identification.

### 4. Train, Select, and Export the Compressor

The compressor maps a clinical note to fixed note tokens. Train it from the
paired source/compressed IAC:

```bash
clintrace train compressor --config configs/compressor/train.yaml
```

Training writes checkpoints and metrics under `runs/` or the configured
compressor output directory. Select the checkpoint by the predeclared validation
rule, then export the release-sized compressor artifact into the ignored
`artifacts/compressor/` tree:

```bash
clintrace build compressor-artifact \
  --checkpoint runs/compressor/checkpoints/course_encoder_step5500.pt \
  --output artifacts/compressor/clintrace_compressor_qwen35_2b_n32_d768_step5500.pt
```

Run a single-note sanity check on the exported artifact:

```bash
clintrace compress \
  --checkpoint artifacts/compressor/clintrace_compressor_qwen35_2b_n32_d768_step5500.pt \
  --input tmp/source_note.txt \
  --output tmp/compressed_note.txt
```

If a validated compressor artifact already exists, start from this sanity check
and continue with feature generation.

### 5. Build Frozen Note Features

Create longitudinal indexes and frozen Stage-A note-token caches. These caches
are the model input substrate for downstream decision training.

```bash
clintrace build timeline
clintrace build compressor-features --checkpoint artifacts/compressor/clintrace_compressor_qwen35_2b_n32_d768_step5500.pt
```

### 6. Build Decision Supervision

Compile the extracted EXPO chains into supervision assets. The chain labels and
spans are used only as training/evaluation targets, not as inference inputs.

```bash
clintrace build grounding-supervision --overwrite
```

### 7. Build Direct-Token and Decoder Planes

Build the direct-token feature plane used for compact or already-compressed
text, then cache per-note grounding features.

```bash
clintrace build decoder-plane
clintrace build grounding-features --overwrite
```

### 8. Build History-State and Decision-Contrast Assets

History-state summaries provide longitudinal patient context for admission-level
training. Decision-contrast pairs are conservative EXPO bundle comparisons used
to reduce clinically implausible near-neighbor confusion.

```bash
clintrace llmc history --help
clintrace build decision-contrast
```

### 9. Train the ClinTrace Decision Expert

The decision expert is trained in three ordered components. The component names
describe the capability being learned, not separate deployable experts.

```bash
clintrace train router
clintrace train grounding --overwrite
clintrace train completion --initialization grounding_init --overwrite
clintrace train longitudinal --initialization completion_init --eval-splits validation test external --overwrite
```

`grounding` learns full-note EXPO span grounding. `completion` learns
observed-evidence to P/X/O decision completion. `longitudinal` adds H1 history
state conditioning while keeping EXPO grounding and completion active.

### 10. Evaluate and Freeze Release Candidates

Evaluate every selected checkpoint on validation, internal test, and external
splits. The gated release candidate is chosen from the complete validation chain,
not from an ad hoc artifact bundle.

```bash
clintrace evaluate grounding --split validation
clintrace evaluate completion --checkpoint runs/clintrace_completion_grounding_init/clintrace_decision_expert.pt --split test
clintrace evaluate longitudinal --checkpoint runs/clintrace_longitudinal_completion_init/clintrace_longitudinal.pt --split test
clintrace evaluate decision-contrast --help
```

After the complete validation chain is frozen, release assets are staged under
`artifacts/hf/clintrace/`. That directory is the local Hugging Face gated-release
root and remains ignored by git.

## Configuration

Configs are grouped by component:

```text
configs/compressor/              compressor training and local smoke configs
configs/decision/default.yaml     grounding/completion/longitudinal defaults
```

## Package Layout

```text
src/iatro/clintrace/
  compressor/
  routing/
  grounding/
  completion/
  longitudinal/
  timeline/
  inspect/
```

Top-level `scripts/` is intentionally empty for the public package. Stable
functionality belongs in the package and is exposed through `clintrace`.

## Open-Source Hygiene

Do not commit:

```text
clinical data
LLM extraction outputs containing protected text
model checkpoints or generated feature caches
runs folders
local credentials or API keys
Hugging Face gated release assets
throwaway intermediate files
```

Institution-specific names are avoided in public code and documentation. Local
asset aliases should use neutral identifiers such as `site_a` and `site_b`.

## Optional LLM Client Utilities

`clintrace llmc` commands are wrappers around OpenAI-compatible chat endpoints.
They are client-side data-preparation utilities, not ClinTrace build steps.

```text
clintrace llmc compress    prompts/compressor_zh.md -> compressed-note JSONL
clintrace llmc expo        prompts/expo_chain_extraction_zh.md -> EXPO chain JSONL
clintrace llmc history     prompts/history_state_h1_zh.md -> H1 history-state JSONL
```

`llmc compress` reads a source IAC, applies the versioned compression prompt,
and appends compressed-note rows to a JSONL file that can be consumed directly
by `clintrace build paired-iac --compressed-input`.

The output JSONL contains at least:

```text
doc_id
patient_id
institution
doc_type
doc_timestamp
episode_index
compressed_text
usage
```

Default execution targets a local OpenAI-compatible server so clinical text
stays on the host.

```bash
export OPENAI_API_BASE=http://127.0.0.1:8000/v1
export OPENAI_API_KEY=local
export OPENAI_MODEL=qwen36

clintrace llmc compress \
  --institution site_a \
  --source-iac data/00_source/site_a.iac \
  --prompt prompts/compressor_zh.md \
  --out data/01_compressed/site_a.jsonl \
  --model qwen36 \
  --workers 16 \
  --thinking disabled
```

Useful preparation checks:

```bash
clintrace llmc compress \
  --institution site_a \
  --source-iac data/00_source/site_a.iac \
  --out data/01_compressed/site_a.jsonl \
  --dry-run

clintrace build paired-iac \
  --source-iac data/00_source/site_a.iac \
  --compressed-input data/01_compressed/site_a.jsonl \
  --output data/02_paired/site_a.iac \
  --institution site_a \
  --overwrite
```

Rows are resumable by `doc_id`: rerunning the wrapper skips documents that
already have a non-empty `compressed_text` in the output file. Non-local API
endpoints require `--allow-external-endpoint`; only use that with an approved
clinical-data processing endpoint.

EXPO and history-state preparation use the same endpoint policy:

```bash
clintrace llmc expo \
  --input data/01_compressed/site_a.jsonl \
  --out data/05_decision_chains/results.jsonl \
  --label-vocab data/06_decision_supervision/label_vocab.json \
  --model qwen36 \
  --workers 16

clintrace llmc history \
  --institution site_a \
  --compressed-input data/01_compressed/site_a.jsonl \
  --timeline-dir data/03_timeline_index \
  --output data/08_patient_state_history/site_a.jsonl \
  --model qwen36 \
  --workers 16
```
