Metadata-Version: 2.4
Name: dansk-register-platform
Version: 0.16.0
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Rust
Classifier: Topic :: Scientific/Engineering :: Medical Science Apps.
Classifier: Typing :: Typed
Requires-Dist: polars>=1.0
Requires-Dist: pydantic>=2
Requires-Dist: rich>=13
Requires-Dist: rich-argparse>=1.8
Requires-Dist: sas7bdat-polars==0.9.2
Requires-Dist: matplotlib>=3.8
Requires-Dist: dansk-register-platform[study1] ; extra == 'all'
Requires-Dist: dansk-register-platform[study2] ; extra == 'all'
Requires-Dist: dansk-register-platform[study3] ; extra == 'all'
Requires-Dist: pyarrow>=13 ; extra == 'repair'
Provides-Extra: all
Provides-Extra: repair
Provides-Extra: study1
Provides-Extra: study2
Provides-Extra: study3
License-File: LICENSE
Summary: Danish register analysis platform (dansk_register) and the study pipelines built on it: parental income, societal costs, and educational and labour-market trajectories
Keywords: epidemiology,registers,denmark,survival-analysis,polars
Author-email: Tobias Kragholm <tkragholm@gmail.com>
License-Expression: MIT
Requires-Python: >=3.12
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
Project-URL: Issues, https://github.com/tkragholm/register-studies/issues
Project-URL: Repository, https://github.com/tkragholm/register-studies

# dansk-register

A platform for epidemiological studies on the Danish national registers, and
the study pipelines built on it.

The package contains no data. It is the code that reads a register delivery
inside a secure research environment, builds cohorts and outcomes from it, and
estimates and reports the results.

## What is in it

| Package | Role |
|---|---|
| `dansk_register` | The reusable platform: register catalog and I/O, the staged-pipeline runner, matching, estimation, disclosure control, figure styling |
| `societal_costs` | Study II: societal costs of severe chronic disease in childhood |
| `edu_labour_trajectories` | Study III: educational and labour-market trajectories from age 18 |

The dependency direction is one way and enforced by a test: the platform never
imports a study, and studies never import each other. Code that turns out to be
general is lifted into `dansk_register` rather than shared sideways.

## Layout

This directory is the private half of the PhD: the register platform, the three
study pipelines, and the crates they call. Public methods crates are
dependencies, not subdirectories. See `../admin/plans/code_consolidation.md` for
why, and for the migration that produced this shape on 26 August 2026.

- `src/` - the Python packages above plus `parental_income` (Study I) and
  `register_studies` (the orchestration CLI), and `src/rust/` for the pyo3
  extension. All of it ships in one maturin wheel; the platform is not
  separately installable.
- `crates/` - the Rust the studies call. `register_runner` (Parquet in,
  estimators out), `cost_engine`, `cpi_deflate` and `person_years` (the three
  bridge CLIs), `sas7bdat-writer` (the only thing that writes the format the
  Danish register hosts deliver), `register-delivery` (the synthetic delivery
  the end-to-end tests run on), and `sporskifte` (track changes for Typst).
- `checkouts/` - gitignored reading copies of the public repos, cloned by
  `just checkouts`. Never a build input.
- `docs/` - implementation documents only. Science goes to `../studies/`.

Six of the seven crates are members of one cargo workspace with one
`Cargo.lock`. `sporskifte` is not: it is a workspace of its own, and flattening
it would put a vello/wgpu graphics stack and three vendored pdf-rs forks into
this lockfile, which every `--workspace` command in the gate would then compile.

## Dependencies that live elsewhere

| Crate | Source | Pinned by |
|---|---|---|
| `cohort_matching` | crates.io | `0.2.0` + `Cargo.lock` checksum |
| `semicontinuous_models` | crates.io | `0.2.0` + `Cargo.lock` checksum |
| `sas7bdat` | crates.io | `0.8` + `Cargo.lock` checksum |
| `scd` | git, `main` | revision in `Cargo.lock` |
| `did_methods` | git, `main` | revision in `Cargo.lock` |
| `sas7bdat-polars` | PyPI | `==0.9.2`, exact by design |

The rule that produced this table: **public code is consumed by version.** If a
fix is needed in one of these, fix it upstream, release it, bump the pin. Never
vendor it, never path-depend across the boundary. `cohort_matching` was vendored
for a week in August 2026 and the copy cost more than it bought.

`did_methods` stays a git pin because its `honest` feature builds HiGHS from
C++, which is more than a crates.io release is currently worth. `scd` could
follow the other three to a version pin; 0.1.3 is published.

## Install

```
pip install dansk-register-platform                 # everything, all three studies
pip install "dansk-register-platform[study2]"       # names a study; installs the same files
pip install "dansk-register-platform[all]"          # all three, explicitly
```

The distribution was renamed from `dansk-register` at 0.7.0, to get past the
analysis server mirror's 24-hour cache. The import packages are deliberately
unchanged, so nothing in this codebase or on the server imports anything
different; only `pip install` names the new one. The old name still resolves,
to a distribution that stopped being updated before the rename.

The estimators are written in polars and validated against R, so the pipeline
carries no numeric stack of its own. matplotlib is a base dependency because
numpy arrived on every real run anyway; what keeps the boundary real is that its
imports live under a `figures` package and a contract test fails the build if one
escapes into an estimator. Without a renderer every result is still produced and
the figure stage records that it could not draw.

`repair` is the only optional dependency: pyarrow, for `register-repair-utf8`
alone. Install it with `pip install 'dansk-register-platform[repair]'`. The same
kind of contract test keeps pyarrow out of everything else.

## Commands

```
edu-labour --list-profiles                          # what each profile runs
edu-labour-preflight --raw-root PATH                # what a delivery contains
edu-labour --output-dir RUN --profile production \
    --raw-root PATH --allow-expensive-stages --resume
edu-labour-figures --bundle RUN/outputs/export --out figures

societal-costs --capability PATH                     # what this study needs vs. what a delivery has
societal-costs --output-dir RUN --profile production \
    --raw-root PATH --allow-expensive-stages --diagnostics
societal-costs secondary_sensitivity --output-dir RUN   # the arm no profile runs

register-repair-utf8 PATH --report utf8_scan.csv    # find invalid UTF-8 in converted Parquet
register-repair-utf8 PATH --fix --report utf8_fix.csv
```

`--allow-expensive-stages` is required for `production`: `matching` scans the
whole delivery, and without the flag the runner refuses rather than silently
starting a multi-hour match. `--profile production` runs 19 stages but not
`secondary_sensitivity`, the analyse-only sensitivity and subgroup arm, which
reads a completed run and writes into a subdirectory of it.

`register-repair-utf8` needs the `repair` extra and says so if it is missing.

Add `--dry-run` to any `edu-labour` invocation to resolve the plan and settings
without touching data.

## Design notes

**Stages talk through files.** Every stage writes named artifacts into a run
directory and reads its inputs from there, so any stage can be re-run alone
against a previous run's outputs, and `--resume` can skip what is already done.

**Estimators are written against polars.** Aalen-Johansen, Fine-Gray, Cox and
the multi-state occupancy are implemented directly rather than taken from a
modelling library, and each is validated against reference values from R with
those values checked into the tests. That keeps the numbers checkable line by
line and avoids a dependency that may not install in an offline environment.

**Disclosure control is part of the pipeline.** Results leave the secure
environment as a bundle of CSV tables with small cells suppressed and the
estimates resting on them blanked. Figures are rendered from that bundle rather
than from the raw artifacts, so a figure cannot show what the table beside it
was not cleared to show.

## What this covers, and what it does not

Secure-environment research is usually described with the [Five
Safes](https://fivesafes.org/), and it is worth saying plainly which of the five
a piece of software can touch at all.

| | |
|---|---|
| **Safe projects** | Your approval. Nothing here. |
| **Safe people** | Your institution's authorisation. Nothing here. |
| **Safe settings** | The host's environment — Statistics Denmark, Sundhedsdatastyrelsen. Nothing here. |
| **Safe data** | **Partly here.** Person identifiers are replaced with a study-local key, and an export is refused outright if a CPR-shaped value reaches it. |
| **Safe outputs** | **Partly here.** Small cells are suppressed, dependent estimates blanked, file types and sizes checked against the transfer rules, and every bundle carries a manifest saying what was done and what was not. |

"Partly" is the operative word in both rows. The output checks are cell-by-cell
and cannot see a small number recovered by differencing across tables; each
bundle's `MANIFEST.txt` says so in its own text. Running this framework is not a
substitute for your host's output review, and it does not make an
unapproved analysis approved.

## Working on it

`just` lists everything. The gates, which are what CI runs:

```
just lint          # both languages: ruff + three type checkers, rustfmt +
                   # clippy, and the three supply-chain audits
just rust-test     # 207 Rust tests across the three crates, via nextest
just test          # 2,953 Python tests, parallel
just gate          # all three, in CI's order
```

and the loops:

```
just test-fast     # 2,703 of them in ~15 s; everything but the real pipelines
just test-e2e      # the 190 that run a real pipeline over a generated delivery
just types         # ty + pyrefly: 0.7 s from cold, against mypy's 6 s
just typos         # spell-check; not a gate, see _typos.toml
just mutants       # would the tests have noticed? Slow, not a gate
prek install       # optional: run the formatters before each commit
```

and the escape hatch, for when a fix has to be tried in a public crate before it
is released:

```
just checkouts     # clone or fast-forward the public repos in checkouts.toml
just patch-on      # build against those checkouts, not the published versions
just patch-off     # back to the pins
```

`patch-on` spans both toolchains, because the coupling does: a cargo `[patch]`
for `cohort_matching`, `semicontinuous_models`, `scd` and `did_methods`, and a
`[tool.uv.sources]` editable path for `sas7bdat-polars`, which reaches the
pipeline as a PyPI wheel rather than as a cargo dependency. It dirties
`Cargo.toml`, `pyproject.toml` and both lockfiles, deliberately: a patched tree
builds code that is in no lockfile, and that should be visible. `just patch-off`
restores all four, and is what to run before committing, and before trusting a
test result you intend to act on.

Two things worth knowing before the first full run:

**The generated delivery is cached, not rebuilt.** The end-to-end tier runs
against an 854 MB synthetic delivery, and it is byte-reproducible from its
inputs, so it is kept under `target/test-deliveries/` keyed by a hash of the
generator, both catalogs and the parameters. It is built once and shared by
every xdist worker and every later run. `REGISTER_DELIVERY_CACHE=` (empty) turns
that off. See `tests/full_delivery.py` for why it is a cache rather than the
usual filelock recipe.

**One `pytest` at a time.** Overlapping full runs is what filled the disk on
25 August 2026. The suite now refuses to start the end-to-end tier below 8 GB
free, and says how to reclaim; `REGISTER_STUDIES_MIN_FREE_GB=0` overrides it.

## Status

Research code, developed alongside the studies it implements. The public API is
not stable.

## Documentation

[`docs/README.md`](docs/README.md) is the index, and it says which document is a
standing contract and which is a finished pass kept because the code cites it.
Two worth naming: [`docs/open_questions.md`](docs/open_questions.md) is the
open-items register, and [`docs/REUSABILITY_ANALYSIS.md`](docs/REUSABILITY_ANALYSIS.md)
is why the platform is shaped the way it is.

Knowledge rather than implementation — register semantics, methodology, study
protocols — lives in the PhD store, not here. `docs/README.md` states the rule
that decides which is which.

## Licence

MIT. See `LICENSE`.

