Metadata-Version: 2.4
Name: corpusscope
Version: 0.1.1
Summary: Profile a clinical EMR database into a portable, schema-validated corpus profile: exact tiktoken token counts (full-record vs clinical-content), per-stream scope metrics, one worked patient record, and a schema-validated YAML/JSON deliverable.
Author: Meridian Intelligence
License: Proprietary — Meridian Intelligence, engagement-scoped (see LICENSE)
Project-URL: Homepage, https://github.com/Meridian-Int/meridian-corpus-scope
Project-URL: Issues, https://github.com/Meridian-Int/meridian-corpus-scope/issues
Keywords: clinical,emr,corpus,profiling,tiktoken,tokens,healthcare,data-profiling
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: License :: Other/Proprietary License
Classifier: Operating System :: OS Independent
Classifier: Topic :: Scientific/Engineering :: Medical Science Apps.
Classifier: Intended Audience :: Healthcare Industry
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: LICENSE.pdf
Requires-Dist: SQLAlchemy>=1.4
Requires-Dist: pyodbc>=4.0
Requires-Dist: tiktoken>=0.7
Requires-Dist: PyYAML>=5.4
Requires-Dist: jsonschema>=4.0
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Dynamic: license-file

# corpusscope

**Profile a clinical EMR database into a portable, schema-validated corpus profile.**

Point `corpusscope` at your SQL database. It reads through the whole thing,
counts tokens exactly with [tiktoken](https://github.com/openai/tiktoken),
measures the scope of every clinical data stream, captures one representative
patient in full, and hands you two files:

- **`profile.yaml`** — the profile in a human-readable form you can open and read.
- **`profile.json`** — the same content, machine-readable and validated against
  the corpus schema bundled with the tool.

That pair is the whole deliverable. It describes *what your corpus contains and
how big it is* — enough to size, plan, and reason about the dataset.
`corpusscope` reads and measures only; it leaves the database untouched.

---

## Table of contents

- [What problem it solves](#what-problem-it-solves)
- [What it produces](#what-it-produces)
- [How it works — the four steps](#how-it-works--the-four-steps)
- [Install](#install)
- [Using corpusscope on the command line](#using-corpusscope-on-the-command-line)
  - [The three commands](#the-three-commands)
  - [Step-by-step walkthrough](#step-by-step-walkthrough)
  - [Command & option reference](#command--option-reference)
  - [Exit codes](#exit-codes)
- [Using corpusscope from Python](#using-corpusscope-from-python)
- [The mapping file — full reference](#the-mapping-file--full-reference)
- [The token model (full vs clinical)](#the-token-model-full-vs-clinical)
- [Quality gates](#quality-gates)
- [Read-only](#read-only)
- [Requirements & supported databases](#requirements--supported-databases)
- [License](#license)

---

## What problem it solves

Every clinical EMR database is laid out differently — different table names,
different column names, labs stored one way here and another way there, some
data streams present and some absent. Before you can plan anything with such a
dataset, you need an honest, precise description of it: how many patients and
encounters, how many years it spans, which clinical streams exist, how the
diagnoses are coded, and — crucially — **how large it is in tokens**, the unit
that actually governs the cost and feasibility of any downstream language-model
work.

`corpusscope` produces exactly that description, in a single standard shape, from
any SQL schema. You teach it your schema **once** (a small, hand-editable
[mapping file](#the-mapping-file--full-reference)); after that it does the
counting, the measuring, and the self-checking on its own.

---

## What it produces

The output has three parts. All three appear in both `profile.yaml` and
`profile.json`.

### 1. `corpus` — identity block

The dataset's identity: `name`, `provider`, `country`, `source_system`,
`source_database`, and a `contact`. **You fill these in** at the top of the
mapping file; `corpusscope` copies them through verbatim. They describe *whose*
data this is and *where it came from* — the tool cannot infer them.

### 2. Scope — the whole dataset measured

Twelve scope sections, `A1`–`A12`, each computed as SQL aggregates across the
**entire** database (not a sample):

| # | Section | What it measures |
| --- | --- | --- |
| A1 | `scale` | Patients, encounters, source rows, linked tables — **plus the exact token counts** (see [token model](#the-token-model-full-vs-clinical)). |
| A2 | `stream_inventory` | For each of the 17 canonical clinical streams: whether it's present, and how many source rows it holds. Streams you don't hold are recorded as absent. |
| A3 | `record_depth` | Fields populated per encounter, visits per patient (mean & median), and the split of documentation across *consultation / treatment / investigations*. |
| A4 | `longitudinal` | First and last encounter date, number of years covered, and encounters (and new patients) per year. |
| A5 | `geography` | Distinct facilities, regions covered, and the per-region share of activity. |
| A6 | `demographics_scope` | Gender split, mean age, age-parse rate, and the age-band distribution. |
| A7 | `diagnoses_scope` | Coding system, coded-record count, distinct codes, ICD-10 shape-match rate, paired free-text rate, a breakdown by ICD-10 chapter, and the top conditions. |
| A8 | `laboratory_scope` | Distinct analytes, result and order counts, how often units and reference ranges travel with a result, and the top analytes. |
| A9 | `vitals_scope` | Triage row count and per-vital coverage (temperature, blood pressure, pulse, weight, height, BMI, …). |
| A10 | `examination_scope` | Regions in the exam grid, total region cells, and the normal / abnormal / not-examined split. |
| A11 | `medications_scope` | Prescription lines, distinct items, and how completely frequency / route / duration are recorded. |
| A12 | `specialties_scope` | Number of distinct clinical specialties. |

Sections for streams your dataset doesn't hold come back empty or `present:
false`, so a dataset that lacks (say) radiology or vitals profiles just as
cleanly as one that has them.

### 3. One worked patient

A single real patient assembled into the full nested record shape —
demographics, then encounters, each with its notes, diagnoses, labs,
prescriptions, and so on. This shows the *shape* of a record end-to-end, so the
scope numbers above have a concrete example to stand next to. `corpusscope` picks
the first patient that has a real encounter, so the example is representative
rather than a stub.

---

## How it works — the four steps

```
   ┌─────────────┐   1 autodetect    ┌──────────────┐   2 review / edit
   │  your SQL   │ ────────────────▶ │ mapping.yaml │ ◀───────────────  you
   │  database   │                   └──────┬───────┘
   └──────┬──────┘                          │
          │            3 profile            │
          └───────────────┬─────────────────┘
                          ▼
                 ┌──────────────────┐   4 QA + schema-validate
                 │   corpusscope    │ ─────────────────────────▶  profile.yaml
                 │  (exact tokens + │                             profile.json
                 │  scope + patient)│                             (ready to send)
                 └──────────────────┘
```

1. **autodetect** — `corpusscope` reflects your live schema and writes a *proposed*
   mapping: its best guess at which physical table and columns feed each canonical
   stream.
2. **review** — you open that one file and confirm it. Fix any column it guessed
   wrong, and mark streams you don't have as absent. This is the only manual step,
   and it's hand-editable YAML.
3. **profile** — `corpusscope` reads the whole database. It makes **two exact
   passes**: a token pass (one patient at a time, so even a billion-token corpus
   never loads more than one record into memory) and a pass of SQL aggregates for
   the scope sections.
4. **QA + validate** — every run checks the finished profile against the bundled
   schema and a set of [quality gates](#quality-gates), then writes the two files.
   A run that fails a gate stops with a non-zero exit instead of writing.

---

## Install

```bash
pip install corpusscope
```

Then confirm it's on your `PATH`:

```bash
corpusscope --version
```

Everything it needs installs with it — the database drivers (SQLAlchemy + pyodbc),
the exact tokeniser (tiktoken), the YAML writer, and the schema validator. There
is nothing else to install and no companion tool to run. **Python 3.9+.**

To install from a source checkout instead of the published package:

```bash
pip install .            # or:  pip install -e ".[dev]"   (editable + test deps)
```

---

## Using corpusscope on the command line

Installing the package puts one command on your `PATH`: **`corpusscope`**. It has
three sub-commands, run in order. You only ever touch two things: a **connection
URL** (which database to read) and a **mapping file** (how your schema maps to the
canonical streams).

### The three commands

| Command | What it does | You run it |
| --- | --- | --- |
| `corpusscope autodetect` | Inspects your live schema and writes a **proposed** `mapping.yaml`. | Once, to get a starting point. |
| `corpusscope profile` | Reads the whole database and writes `profile.yaml` + `profile.json`. | Every time you want a profile. |
| `corpusscope validate` | Checks an existing `profile.json` against the schema. | Any time, e.g. before sending. |

Run `corpusscope --help` (or `corpusscope <command> --help`) to see the same
options listed below, and `corpusscope --version` to print the version.

### Step-by-step walkthrough

The **connection URL** is a standard
[SQLAlchemy database URL](https://docs.sqlalchemy.org/en/20/core/engines.html#database-urls).
Two common shapes:

```bash
# Microsoft Fabric / Azure SQL analytics endpoint:
export DB_URL="mssql+pyodbc://@<sql-endpoint>.datawarehouse.fabric.microsoft.com/<database>?driver=ODBC+Driver+18+for+SQL+Server&authentication=ActiveDirectoryInteractive"

# A local SQLite file (handy for a trial):
export DB_URL="sqlite:///./mydata.db"
```

**Step 1 — propose a mapping.** Point `autodetect` at your database. It reflects
the schema and guesses which table/columns feed each stream:

```bash
corpusscope autodetect --source "$DB_URL" --out mapping.yaml
```
```text
Proposed mapping -> mapping.yaml
REVIEW IT before profiling: confirm each stream's table/columns and which streams are present:false.
```

> If your patient / encounter key columns aren't named `patient_id` /
> `encounter_id`, tell autodetect: `--patient-id pat_no --encounter-id visit_no`.
> If your tables live under a named schema, add `--schema dbo`.

**Step 2 — review the mapping (the one manual step).** Open `mapping.yaml` and
confirm it against your real schema: fix any column it guessed wrong, fill in the
`corpus:` identity block at the top, and mark any stream you don't have as
`present: false`. See [The mapping file](#the-mapping-file--full-reference) for
every option and two worked examples.

**Step 3 — profile the database.** This does the full read — the exact token pass
plus the scope aggregates — then runs QA and writes the two files:

```bash
corpusscope profile --source "$DB_URL" --mapping mapping.yaml \
    --out-yaml profile.yaml --out-json profile.json
```
```text
Profiling (exact token pass + scope aggregates)…

QA: 0 error(s), 0 warning(s)

  patients   : 48,213
  tokens     : 412,556,190 full  |  210,004,731 clinical (50.9%)  [tiktoken o200k_base]
  wrote -> profile.yaml
  wrote -> profile.json
```

If QA finds an **error**, the run stops and writes nothing:

```text
QA: 1 error(s), 0 warning(s)
  ERROR   [tokens] clinical_content_tokens (…) > total_tokens (…)

QA FAILED — no profile written.
```

So a run either writes a clean, schema-valid profile or stops without writing.
(Both `--out-*` flags are optional; omit them to do a **dry run** that prints QA
and the headline numbers but writes nothing.)

**Step 4 — hand off (and optionally re-check).** The two files *are* the
deliverable. You can re-validate the JSON on its own at any time — no database
needed:

```bash
corpusscope validate --json profile.json
```
```text
valid against the corpus schema.
```

### Command & option reference

**`corpusscope autodetect`** — propose a mapping from a live database.

| Option | Required | Meaning |
| --- | --- | --- |
| `--source <url>` | yes | SQLAlchemy connection URL of the source database. |
| `--out <path>` | yes | Where to write the proposed mapping YAML. |
| `--schema <name>` | no | DB schema/namespace (e.g. `dbo`) if your tables live under one. |
| `--patient-id <col>` | no | Patient key column name (default `patient_id`). |
| `--encounter-id <col>` | no | Encounter key column name (default `encounter_id`). |

**`corpusscope profile`** — build the profile from a mapped database.

| Option | Required | Meaning |
| --- | --- | --- |
| `--source <url>` | yes | SQLAlchemy connection URL of the source database. |
| `--mapping <path>` | yes | Your reviewed mapping YAML. |
| `--out-yaml <path>` | no | Write the human-readable YAML here. |
| `--out-json <path>` | no | Write the schema-valid JSON here. |
| `--schema <name>` | no | DB schema/namespace (e.g. `dbo`). |

**`corpusscope validate`** — check a profile JSON against the schema.

| Option | Required | Meaning |
| --- | --- | --- |
| `--json <path>` | yes | Profile JSON to validate. |

### Exit codes

Every command returns `0` on success and a non-zero code on failure, so it drops
straight into a script or CI pipeline:

| Command | `0` (success) | non-zero (failure) |
| --- | --- | --- |
| `autodetect` | mapping written | connection/reflection error |
| `profile` | QA passed; files written | **any QA error** — nothing written |
| `validate` | JSON is valid | JSON is invalid (errors printed) |

## Using corpusscope from Python

Everything the CLI does is available as a library — the same four moves in code:

```python
import corpusscope as cs

# 1. connect (read-only) and load your reviewed mapping
db = cs.Db(cs.connect("<sqlalchemy-url>"))
mapping = cs.Mapping.from_yaml("mapping.yaml")

# 2. build the profile — exact token pass + scope + one worked patient
profile = cs.build_profile(db, mapping)

# 3. run the QA gates and stop on any error (same gate the CLI enforces)
issues = cs.run_qa(profile)
assert not [i for i in issues if i.level == "error"], issues

# 4. write the deliverable
cs.write_yaml(profile, "profile.yaml")
cs.write_json(profile, "profile.json")
```

`build_profile` returns the profile as a plain Python `dict`, so you can inspect
any number before writing it — e.g. `profile["scale"]["total_tokens"]` or
`profile["scale"]["clinical_content_pct"]`.

---

## The mapping file — full reference

Your tables and columns won't match the canonical names `corpusscope` reports in,
so a small **mapping file** bridges the two. `autodetect` writes a first draft;
you review it. This is the tool's single point of configuration, and it's plain,
auditable YAML.

A mapping has four top-level parts:

```yaml
corpus:                       # identity — copied verbatim into the profile
  name: Example Clinical Corpus
  provider: Example Health
  country: Colombia
  source_system: Example EMR
  source_database: SQL Server 2019
  contact: { name: Jane Doe, email: jane@example.org, role: Data lead }

schema:                       # DB schema/namespace, e.g. dbo — leave blank if none

keys:                         # the columns that link rows to a patient / encounter
  patient_id: patient_id
  encounter_id: encounter_id

streams:                      # one entry per canonical stream (see below)
  ...
```

### Per-stream fields

Each stream tells `corpusscope` where its data physically lives:

```yaml
streams:
  demographics:
    table: tbl_patient
    columns: { age_years: age_years, gender: sex, home_region: home_region }

  encounters:
    table: tbl_encounter
    date_column: encounter_start                 # drives the longitudinal metrics
    columns: { facility_id: care_center_code, specialty_id: specialty_code, visit_type: care_setting }

  diagnoses:
    table: tbl_encounter                         # two streams may share one table
    columns: { icd10_code: admission_diagnosis_code, diagnosis_name: admission_diagnosis }

  lab_results:                                   # analytes stored as columns (wide)
    table: tbl_lab
    layout: wide
    analyte_columns: [hemoglobin, hba1c, creatinine, total_cholesterol, hdl, ldl]

  prescriptions:
    table: tbl_medication
    columns: { generic_name: medication_name, dose: dose, route: admin_route }

  # a stream you don't hold:
  immunizations: { present: false }
```

Every knob a stream can carry:

| Field | Meaning |
| --- | --- |
| `table` | The physical table this stream reads from. Two streams may point at the same table (e.g. `encounters` and `diagnoses`); the tool de-duplicates so a shared table's storage is never counted twice. |
| `present: false` | You don't hold this stream. Recorded as absent in the profile. |
| `columns` | Map of `canonical_field: physical_column`. Only the fields you have. |
| `patient_id_column` / `encounter_id_column` | Override the link columns from `keys` when *this* table names them differently (e.g. notes that link by `admission_id`). |
| `date_column` | The date/datetime column for time-based metrics (used for `encounters` longitudinal coverage). |
| `layout` | For `lab_results`: `long` (one row per analyte) or `wide` (one column per analyte). Both are supported. |
| `analyte_columns` | For a `wide` lab layout: the list of analyte columns to count. |
| `where` | An optional raw SQL filter applied uniformly to every metric over this stream (e.g. `is_annulled = 0` to exclude voided records). |
| `clinical_extra` | Extra free-text columns whose *values* are clinical content but have no canonical field (e.g. `result_interpretation`, `medical_indications`). Counted into the clinical-content tokens. |
| `value_maps` | Per-field value coding. Most important for gender, where single-letter codes conflict across datasets — `m` is *male* in one, *mujer/female* in another. Declaring the coding makes the buckets correct. |

The `value_maps` gender example, showing why it matters:

```yaml
demographics:
  table: pacientes
  columns: { gender: sex }
  value_maps:
    gender:                 # in THIS dataset m = mujer (female), h = hombre (male)
      female: [m, mujer, f]
      male:   [h, hombre]
      other:  [i, unknown]
```

### The 17 canonical streams

`demographics`, `encounters`, `triage_vitals`, `history_notes`, `physical_exam`,
`region_findings`, `impression_notes`, `diagnoses`, `lab_requests`, `lab_results`,
`radiology`, `prescriptions`, `pharmacy_requests`, `procedures`, `immunizations`,
`allergies`, `referrals`.

Map the streams you have; mark the rest `present: false`.

### Worked examples

The `examples/` directory holds two complete, ready-to-edit mappings that show
the same tool adapting to very different schema shapes:

- [`examples/mapping.meridian.yaml`](examples/mapping.meridian.yaml) — a raw
  operational `tbl_*` schema, with labs stored **wide** (one column per analyte).
- [`examples/mapping.anonimizado.yaml`](examples/mapping.anonimizado.yaml) — an
  anonymized delivery schema (Spanish table names, banded ages, year-only dates,
  labs stored **long**), showing `value_maps`, `clinical_extra`, and a `where`
  filter in use.

Start from `autodetect`, then use whichever example is closest as a reference.

---

## The token model (full vs clinical)

Tokens are the headline number. Every patient record is measured on two content
axes and by two encoders.

**Two content axes:**

| | what it counts |
| --- | --- |
| **full-record** | Every stored field, serialized — values **and** labels, ids, flags, timestamps, JSON structure. This is the storage / ingestion cost of the record. |
| **clinical-content** | Only the mapped clinical *values* — diagnoses, results, medications, narrative, vitals, findings. This is the medical signal a model would actually learn from. |

The split between them (e.g. *"51% clinical content"*) tells you how much of the
raw size is real signal versus structural overhead. Which fields count as
clinical content is defined declaratively and auditably — one list per record
section — not buried in the counter.

**Two encoders:** both axes are counted with **`o200k_base`** (the primary,
reported number) *and* **`cl100k_base`** (an independent second count) — a
built-in cross-check on the total.

**Per-patient distribution:** alongside the totals you get min / max, the p50 /
p90 / p99 percentiles, and a 12-bin histogram (`<1k`, `1k-3k`, … `5M+`) of tokens
per patient — so you can see not just the total but how it's spread.

The counting is **streaming**: `corpusscope` tokenises one patient at a time and
keeps only running totals, so an exact count over an arbitrarily large corpus
never loads more than a single record into memory.

---

## Quality gates

Every run checks the finished profile before writing it, against these gates:

- the JSON **validates against the bundled corpus schema**;
- clinical tokens ≤ full tokens, and `structure = full − clinical` exactly;
- the distribution bins sum to the patient count; percentiles are monotonic
  (`p50 ≤ p90 ≤ p99`);
- gender, age-band, exam-outcome, and stream-split shares each sum to ~100%;
- no negative counts anywhere;
- the worked patient is present and non-empty.

An **error** stops the run (non-zero exit, nothing written); a **warning** is
reported for review but doesn't block. The package also ships an end-to-end test
that profiles a synthetic database and checks the numbers against known answers.

---

## Read-only

`corpusscope` reads and measures only. Every SQL statement it runs is a `SELECT` —
no inserts, updates, or deletes — and its entire output is the two profile files.

---

## Requirements & supported databases

- **Python 3.9+**
- Connects via **SQLAlchemy**, so it works against any database SQLAlchemy +
  pyodbc can reach — **Microsoft Fabric / Azure SQL** analytics endpoints,
  **SQL Server**, **PostgreSQL**, and **SQLite** (handy for a local trial). The
  scope queries are written to be dialect-aware across these engines.
- All Python dependencies (`SQLAlchemy`, `pyodbc`, `tiktoken`, `PyYAML`,
  `jsonschema`) install automatically with the package.

---

## License

**Proprietary — © 2026 Meridian Intelligence. All rights reserved.** Not open
source. This software is provided to the counterparty under a **limited,
non-transferable license for use solely within the scope of the parties'
engagement/agreement**, and may not be used, copied, distributed, modified, or
exploited beyond that Purpose. See [LICENSE](LICENSE) for the full terms.
