Metadata-Version: 2.4
Name: meddeid-data
Version: 0.4.2
Summary: Profile-driven multilingual clinical dataset generation and validation for MedDeID
Author: Stig Hellemans
License-Expression: AGPL-3.0-only
Project-URL: Homepage, https://stighellemans.github.io/meddeid/
Project-URL: Documentation, https://stighellemans.github.io/meddeid/workflows/prepare-and-annotate/
Project-URL: Source, https://github.com/stighellemans/meddeid-data
Project-URL: Issues, https://github.com/stighellemans/meddeid-data/issues
Project-URL: MedDeID on Hugging Face, https://huggingface.co/collections/stighellemans/meddeid
Keywords: de-identification,clinical-nlp,synthetic-data,multilingual
Classifier: Development Status :: 4 - Beta
Classifier: Natural Language :: English
Classifier: Natural Language :: Dutch
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Topic :: Scientific/Engineering :: Medical Science Apps.
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: MEDDEID-PRIVATE-FINE-TUNING-EXCEPTION-1.0.txt
Requires-Dist: meddeid-core<0.3,>=0.2
Requires-Dist: meddeid-language-en<0.3,>=0.2
Requires-Dist: meddeid-language-nl<0.3,>=0.2
Requires-Dist: PyYAML>=6
Requires-Dist: python-dotenv>=1.0.1
Provides-Extra: parquet
Requires-Dist: pyarrow>=15; extra == "parquet"
Provides-Extra: llm
Requires-Dist: openai>=2.0; extra == "llm"
Provides-Extra: dev
Requires-Dist: pytest>=8; extra == "dev"
Dynamic: license-file

# meddeid-data

Generate synthetic clinical notes with exact character-offset de-identification
annotations. The package includes locale generation profiles, structured case
models, deterministic renderers, clinical resource pools, Synthea integration,
and dataset-quality checks.

The `nl-BE` profile samples Belgian resources from `meddeid-language-nl`.
The `nl-NL` profile shares only the locale-neutral clinical case structure. It
has a dedicated Netherlands renderer and judge, Netherlands-specific names,
addresses and institutions, and BSN, BIG, EPD/ZIS, postcode, +31 phone, and
`example.nl` conventions. It is available as generation code; this migration
does not generate or publish a corpus.
The unversioned `en-GB` and `en-US` locale profiles sample independently released, audited
name, address, institution, occupation, and terminology records from
`meddeid-language-en`. Bare `en` is rejected because date order, addresses,
identifiers, phone ranges, and clinical terminology are locale-specific.
Belgian DEDUCE is not a runtime dependency.

Start with the suite guide to
[preparing and annotating data](https://stighellemans.github.io/meddeid/workflows/prepare-and-annotate/).
This repository remains authoritative for import, project, split, generation,
and validation commands.

## Installation

```bash
python -m pip install meddeid-data
```

Install Parquet support with
`python -m pip install 'meddeid-data[parquet]'`.

## Generate synthetic data

Generation defaults to the released Dutch/Belgian profile:

```bash
meddeid-data generate --count 100 --output synthetic.jsonl \
  --pretty-output synthetic.pretty.json \
  --judge-report synthetic-report.md
meddeid-data validate synthetic.jsonl
```

Select a language and locale explicitly for multilingual generation:

```bash
meddeid-data generate --language-profile en-GB \
  --count 100 --output synthetic.en-gb.jsonl \
  --judge-report synthetic.en-gb-report.md
```

Use `--language-profile en-US` for the separate United States and territories
profile. English synthetic generation permits exactly the 14 token-classifier
labels and rejects `Anonymize_Other` during rendering, review, and export. The
full taxonomy remains valid for non-generation interoperability.

The local `nl-NL` profile can be selected in the same way. Its availability
does not imply that a Netherlands corpus exists or that the released Dutch
model has been trained or evaluated on Netherlands clinical text.

The ordered English synthetic allowlist is exactly
`Address_Location:Caregiver`, `Address_Location:Other`,
`Address_Location:Patient`, `Age_Birthdate`, `Contactdetails`, `Date`,
`ID:Caregiver`, `ID:Patient`, `Name:Caregiver`, `Name:Other`, `Name:Patient`,
`Organization:Healthcare`, `Organization:Other`, and `Profession`.

Every dataset manifest pins both `language_profile` and the complete
`meddeid.generation-profile.v1` descriptor, including resource hashes. Unknown
profiles fail before generation with the installed-provider discovery result.

Use existing Synthea CSV output or ask the tool to create it:

```bash
meddeid-data generate --count 1000 --output synthetic.jsonl \
  --synthea-csv-dir path/to/synthea/output/csv --require-synthea

meddeid-data generate --count 1000 --output synthetic.jsonl \
  --auto-synthea --require-synthea
```

The two-stage workflow keeps structured cases available for inspection:

```bash
meddeid-data build-cases --language-profile en-GB \
  --count 100 --output cases.jsonl
meddeid-data render-cases cases.jsonl --output synthetic.jsonl
```

Case records pin their generation profile, so `render-cases` resolves it without
requiring the selection again. Passing an incompatible profile is rejected.
`meddeid-data sample` provides a small offline generation run through the same
profile contract.

### Batch-gated production corpora

Use the production state machine for paid generation or any corpus that needs
personal review and a sealed benchmark. Unlike `generate`, it records every
batch transition and refuses to finalize unsigned or changed documents:

```bash
meddeid-data production init ./english-production \
  --profile en-GB,en-US \
  --count 7000 --batch-size 500 --benchmark-count 300 \
  --mode remote --allow-remote --backend english-luna \
  --author-model gpt-5.6-luna --reasoning-effort low \
  --max-cost-usd 32.68

meddeid-data production status ./english-production
meddeid-data production run ./english-production --until-gate
```

`run --until-gate` stops before personal review. Record one content-bound
decision for every document, then sign off that batch:

```bash
meddeid-data production review ./english-production \
  --batch-index 0 --document-id en-gb-synthetic-00001 \
  --decision accept --reviewer reviewer-id
meddeid-data production sign-off ./english-production \
  --batch-index 0 --reviewer reviewer-id --notes 'All 500 reviewed'
```

Decisions are `accept`, `edit`, `regenerate`, or `reject`. Changing a document
invalidates its decision and the batch sign-off. The corpus-wide writer lock
prevents a second batch from spending concurrently. The attempt ledger counts
failed and superseded API requests. `--max-cost-usd` stops new work when the
recorded spend reaches the guardrail and, after the first attempts exist,
before a batch whose projected completion would cross it. The provider account
ledger remains authoritative, so leave headroom for requests already in flight.

Use `replace` for a manually corrected canonical document; the old version and
both hashes remain in the audit trail. Use `invalidate` when the document must
be independently re-authored on the next `run`:

```bash
meddeid-data production replace ./english-production \
  --batch-index 0 --document corrected-document.json \
  --editor reviewer-id --reason 'Corrected two PII boundaries'
meddeid-data production invalidate ./english-production \
  --batch-index 0 --document-id en-gb-synthetic-00001 \
  --reason 'Regenerate with a less common address format'
```

Both operations invalidate the old automated report, document review, batch
sign-off, and any derived final split. After every batch is accepted:

```bash
meddeid-data production finalize ./english-production
```

Finalization repeats canonical-label and diversity checks, refuses exact,
PII-normalized, or configured near duplicates, and creates deterministic
profile/document-family-stratified development and benchmark files. The
English 7,000-document preset resolves to 6,700 development documents and a
300-document benchmark with 150 documents per locale and 25 per
locale/document-family cell.

Remote authoring is a plugin boundary (`meddeid.production_backends`). The
built-in `english-luna` backend implements the current English pipeline;
another language can provide a backend without adding locale branches to the
production state machine.

### Add another language or locale

Generation profiles keep locale-specific case construction, rendering, review,
resources, and provenance outside the CLI. A provider returns a
`meddeid_data.generation_profiles.GenerationProfile` and registers it through
the `meddeid.generation_profiles` entry-point group:

```toml
[project.entry-points."meddeid.generation_profiles"]
fr = "example_meddeid_language_fr.generation:get_generation_profile"
```

The provider function receives `profile_id` and keyword-only `version`. It
returns a profile when supported and raises `ValueError` otherwise. A profile
must implement direct generation, two-stage case building/rendering,
deterministic review, and a hashed resource manifest scoped to the same profile
ID and version. Tests should verify deterministic output, all document types,
exact Unicode-code-point offsets, language metadata, profile inference from case
records, resource hashes, and CLI manifest provenance. The built-in `en-GB`
and `en-US` profiles are the executable conformance examples.

## Create an annotation-ready dataset

Researchers planning to use import, inference, evaluation, and training can
install the complete compatible Python set once with
`python -m pip install 'meddeid[research]'`. The smaller staged installation
below is sufficient when only project creation is needed initially.

No custom conversion script is required for a folder of plain UTF-8 text files
or a CSV, TSV, or Parquet table. For TXT, CSV, or TSV, the only direct package
required for this step is:

```bash
python -m pip install meddeid-data
```

Its schema and language dependencies are installed automatically. Use
`python -m pip install 'meddeid-data[parquet]'` only when reading Parquet.
Neither model weights nor the annotation web application are needed to create
a project. Create the project and import its first dataset in one command:

```bash
meddeid-data project create my-project notes.parquet \
  --namespace hospital-study --language-profile nl-BE \
  --id-column note_id --text-column note_text
```

The same command accepts `notes.csv`, `notes.tsv`, or a directory containing
`.txt` files.

If a source or mapping option is rejected before import, `project create` keeps
the initialized project and its private document-ID key, then prints an
import-only recovery command. Correct the option and either run that
`meddeid-data project import ...` command or repeat `project create` with the
same namespace and language profile. A repeated create continues only an
untouched, matching project scaffold. It never overwrites an imported dataset,
split, assignment, or other project content.

For tables, every column other than the text and ID columns is copied into the
document's `metadata` by default. A column named `metadata` or `metadata_json`
is interpreted as an object and merged, so structured fields such as
`patient`, `caregivers`, and `known_values` reach inference and
post-processing intact. Control this explicitly with repeatable
`--metadata-column`, `--metadata-json-column`, or `--no-metadata`.

### Zero-configuration table shape

A data scientist can avoid mapping options by producing these conventional
columns:

```csv
source_id,text,patient_given_name,patient_family_name,caregiver_1_given_name,caregiver_1_family_name,caregiver_2_given_name,caregiver_2_family_name
n-1,Jan zag Alice en Bob,Jan,Peeters,Alice,Vermeulen,Bob,Janssens
```

`source_id` is used automatically when present. Patient names may instead use a
single `patient` column. Each caregiver may similarly use paired
`caregiver_1_given_name` / `caregiver_1_family_name` columns. The aliases
`first_name` / `last_name` are also accepted. If only a complete name is
available, use the explicit `caregiver_1_full_name` convention. A
`caregivers` column accepts a real list in Parquet or a JSON-array cell in
CSV. Any unrelated columns are preserved as ordinary metadata, but none are
required. No `metadata_json` construction is needed. A complete sample is
available as `examples/zero-config-notes.csv`.

### Reusable mapping for an existing export

Column names do not otherwise need to follow the MedDeID schema. Put the
hospital export mapping in a YAML or JSON file:

```yaml
version: meddeid.import-mapping.v1
text_column: note_body
id_column: export_key
include_metadata: true
patient_given_name_column: subject_first_name
patient_family_name_column: subject_family_name
caregivers:
  - given_name_column: primary_author_first_name
    family_name_column: primary_author_family_name
  - given_name_column: coauthor_first_name
    family_name_column: coauthor_family_name
```

Each `caregivers` entry represents one person. It may instead contain
`full_name_column`; add `delimiter` to that entry only when the same source cell
contains several complete names.

Use it on the first import:

```bash
meddeid-data project create my-project notes.csv \
  --namespace hospital-study --mapping-config import-mapping.yaml
```

MedDeID writes the normalized effective mapping to
`my-project/manifests/import-mapping.json`. Later exports with the same schema
reuse it automatically:

```bash
meddeid-data project import my-project next-export.csv
```

The effective mapping is saved even when the first import used CLI flags or
only the zero-configuration conventions, so mapping remains a one-time project
step. The external YAML is useful when the same hospital export schema should
be reused across several projects.

Supplying a new `--mapping-config` replaces the saved mapping for that import;
individual CLI mapping flags override its fields. The complete example is in
`examples/import-mapping.yaml`.

For a one-off mapping, flags remain available, and the original columns still
remain in metadata:

```bash
meddeid-data project create my-project notes.csv \
  --namespace hospital-study --id-column note_id --text-column note_text \
  --patient-given-name-column patient_first \
  --patient-family-name-column patient_last \
  --caregiver-column author_1_full_name \
  --caregiver-column author_2_full_name
```

Use `--patient-name-column patient_full_name` when the patient name is stored in
one column. Full names use a best-effort comma/whitespace split, so prefer the
separate given/family options when those source columns exist. For one caregiver
column containing several names, add a literal delimiter:

```bash
meddeid-data project create my-project notes.parquet \
  --namespace hospital-study --id-column note_id --text-column note_text \
  --caregiver-column treating_clinicians --caregiver-delimiter ";"
```

The `--caregiver-column` flag always means a complete-name column. Repeat it for
several full-name columns, or add `--caregiver-delimiter` for several complete
names in one cell. Use the structured `caregivers` config shown above when an
existing export stores given and family names separately. Parquet list-valued
caregiver columns are also accepted directly. Arbitrary names such as `author`
or `validator` are never guessed.

Add `--no-metadata` if only the mapped canonical names should be retained and
the original source columns should be discarded. The mapped `patient` and
`caregivers` are still kept because they were explicitly requested.

The project keeps the source-to-document mapping and HMAC key under the
gitignored `private/` directory. Canonical artifacts contain pseudonymous stable
IDs, empty span lists, imported metadata, hashes, and the selected language
profile. The command prints the exact next steps to generate local model
pre-annotations as the initial state of an ordinary annotation assignment:

```bash
python -m pip install 'meddeid>=0.4.1,<0.5'

meddeid batch my-project/artifacts/annotations.jsonl \
  --output my-project/assignments/model-assisted-review.jsonl \
  --model stighellemans/meddeid-dutch-synth --device cpu
```

`meddeid` supplies local inference; the model bundle is downloaded on its first
use. `meddeid-annotate` is a separate web application rather than a pip
package. The default no-checkout route is its public container image:

```bash
docker run --rm -p 127.0.0.1:8787:8787 \
  --read-only --cap-drop ALL --security-opt no-new-privileges \
  -e MEDDEID_ANNOTATIONS_PATH=/input/model-assisted-review.jsonl \
  -v "$PWD/my-project/assignments/model-assisted-review.jsonl:/input/model-assisted-review.jsonl" \
  ghcr.io/stighellemans/meddeid-annotate:0.3.1@sha256:55c6ab5c578ebe1696eb5149c743b8e702470a8f9e1c3cd72e98a5f461bda271
```

Docker downloads the image automatically on first use. When a local source
checkout is configured with `MEDDEID_ANNOTATE_DIR` or detected in the suite
workspace, the CLI instead prints the matching `npm` commands and tells the
user whether `npm ci` is still required.

The predicted spans in `model-assisted-review.jsonl` are the editable starting
state of a review assignment; they are not a separate suggestion type. Its
documents remain unreviewed until a reviewer inspects the text, corrects the
spans, and marks them reviewed using the normal annotation controls.

For a profile without a released inference model, the CLI does not recommend
the Dutch model. It instead prints the command for starting primary annotation
directly from the imported empty-span artifact.

Model downloading happens during `meddeid batch`, not data import. After the
bundle is downloaded, inference is local and document text is not sent to a
server.

The separate lifecycle commands remain available when project initialization
and import need to happen at different times:

```bash
meddeid-data project init my-project --namespace hospital-study --language-profile nl-BE
meddeid-data project import my-project path/to/txt-notes
# or: ... notes.csv --id-column note_id --text-column note_text
# or: ... notes.parquet --id-column note_id --text-column note_text
meddeid-data project package-annotation my-project reviewer-a.jsonl \
  --annotation-set-id hospital-study-round-1 --annotator-id reviewer-7
meddeid-data project split my-project --seed 42 --train .8 --validation .1
```

Reusing the same project preserves document IDs; moving or reordering source
files does not affect downstream identity.

## Prepare training views

If development was reviewed as one assignment, pass it once. The command uses
the project's split manifest to recover the train and validation subsets:

```bash
meddeid-data project prepare-training my-project \
  --development assignments/development-reviewer-a.jsonl \
  --test-gold subannotation/evaluation-bundle/benchmark.jsonl \
  --output prepared-data/experiment-01
```

For workflows that reviewed the two development subsets separately, the
equivalent inputs are `--selection-train ... --selection-validation ...`.

This validates completion, canonical labels, exact split membership, and text
identity before writing:

```text
prepared-data/experiment-01/selection/{train,val,test}.jsonl
prepared-data/experiment-01/refit/{train,val,test}.jsonl
prepared-data/experiment-01/fit/{train,val,test}.jsonl
```

Omit `--output` to retain the compatible default, `my-project/prepared`. Give
each distinct data/split snapshot its own output directory. Multiple model or
hyperparameter runs over the same snapshot can reuse this prepared directory
and should instead receive distinct `meddeid-train --run` directories.

Use the generated `fit` view with `meddeid-train fit` for one-time training.
The selection test file is intentionally empty. The refit train file recombines
the complete reviewed development pool, while its test file contains only the
sealed gold set. Each directory has a checksum and lineage manifest; existing
non-empty output is never overwritten.

## Development

```bash
pip install -e '.[dev]'
pytest
```

## Licence

Code is AGPL-3.0-only with the MedDeID Private Fine-Tuning Exception, version
1.0. The Exception permits private training data and resulting private
fine-tuned weights to remain confidential; MedDeID code modifications remain
subject to AGPL-3.0-only. See `NOTICE` and
`MEDDEID-PRIVATE-FINE-TUNING-EXCEPTION-1.0.txt`. Generated datasets and
incorporated resources retain the terms stated with their respective artifacts
and source notices.
