Metadata-Version: 2.4
Name: dansk-register-platform
Version: 0.28.18
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Rust
Classifier: Topic :: Scientific/Engineering :: Medical Science Apps.
Classifier: Typing :: Typed
Requires-Dist: polars>=1.41.1,<1.45
Requires-Dist: pydantic>=2
Requires-Dist: rich>=13
Requires-Dist: rich-argparse>=1.8
Requires-Dist: sas7bdat-polars==0.10.0
Requires-Dist: matplotlib>=3.8
Requires-Dist: dansk-register-platform[study1] ; extra == 'all'
Requires-Dist: dansk-register-platform[study2] ; extra == 'all'
Requires-Dist: dansk-register-platform[study3] ; extra == 'all'
Requires-Dist: pyarrow>=13 ; extra == 'repair'
Provides-Extra: all
Provides-Extra: repair
Provides-Extra: study1
Provides-Extra: study2
Provides-Extra: study3
License-File: LICENSE
Summary: Danish register analysis platform (dansk_register) and the study pipelines built on it: parental income, societal costs, and educational and labour-market trajectories
Keywords: epidemiology,registers,denmark,survival-analysis,polars
Author-email: Tobias Kragholm <tkragholm@gmail.com>
License-Expression: MIT
Requires-Python: >=3.12
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
Project-URL: Issues, https://github.com/tkragholm/register-studies/issues
Project-URL: Repository, https://github.com/tkragholm/register-studies

# dansk-register-platform

A platform for epidemiological studies on the Danish national registers, and
the study pipelines built on it.

The package contains no data. It is the code that reads a register delivery
inside a secure research environment, builds cohorts and outcomes from it, and
estimates and reports the results.

## What is in it

| Package | Role |
|---|---|
| `dansk_register` | The reusable platform: register catalog and I/O, the staged-pipeline runner, matching, estimation, disclosure control, figure styling |
| `parental_income` | Study I: parental income after severe chronic disease in childhood |
| `societal_costs` | Study II: societal costs of severe chronic disease in childhood |
| `edu_labour_trajectories` | Study III: educational and labour-market trajectories from age 18 |
| `register_studies` | `register-studies`, the command that runs the studies in one job, stages their results for transfer and unpacks them after download |

The dependency direction is one way and enforced by a test: the platform never
imports a study, and studies never import each other. Code that turns out to be
general is lifted into `dansk_register` rather than shared sideways.

The register runner and the estimators are Rust, compiled into the wheel as one
extension module. The methods crates it builds on, cohort matching,
semicontinuous models, staggered difference-in-differences and the chronic
disease phenotype, are published separately and consumed by version.

## Install

```
pip install dansk-register-platform                 # everything, all three studies
pip install "dansk-register-platform[study2]"       # names a study; installs the same files
pip install "dansk-register-platform[all]"          # all three, explicitly
```

Before 0.7.0 the distribution was published as `dansk-register`. The import
packages did not change; only the name on `pip install` did. The old name still
resolves, to a distribution that stopped being updated at the rename.

The estimators are written in polars and validated against R, so the pipeline
carries no numeric stack of its own. matplotlib is a base dependency because
numpy arrived on every real run anyway; what keeps the boundary real is that its
imports live under a `figures` package and a contract test fails the build if one
escapes into an estimator. Without a renderer every result is still produced and
the figure stage records that it could not draw.

`repair` is the only optional dependency: pyarrow, for `register-check delivery utf8`
alone. Install it with `pip install 'dansk-register-platform[repair]'`. The same
kind of contract test keeps pyarrow out of everything else.

## Commands

To run the studies on the analysis server, read `docs/run_on_server.md` in
the source repository. What changed in each release is in `CHANGELOG.md`
there, and the other documents are indexed in `docs/README.md`. This file is
also the package's page on PyPI, where those files are not published, so they
are named rather than linked.

```
register-studies --dry-run --output-dir RUN --raw-root PATH         # the plan, no data read
register-studies --output-dir RUN --raw-root PATH --only 2,3         # Studies II and III in one job
register-studies --output-dir RUN --raw-root PATH --only 1 --resume  # carry on where it stopped
register-studies unpack DOWNLOADED OUT                               # after download, on the laptop

edu-labour --list-profiles                          # what each profile runs
edu-labour --output-dir RUN --profile production \
    --raw-root PATH --allow-expensive-stages --resume
edu-labour-figures --bundle RUN/outputs/export --out figures

societal-costs --capability PATH                     # what this study needs vs. what a delivery has
societal-costs --output-dir RUN --profile production \
    --raw-root PATH --allow-expensive-stages --diagnostics
societal-costs --output-dir RUN --raw-root PATH --stages secondary_sensitivity   # the arm no profile runs

register-check                                      # every diagnostic check, by when you run it
register-check delivery scan-plan --raw-root PATH   # what a delivery contains
register-check delivery utf8 --raw-root PATH --report utf8_scan.csv   # invalid UTF-8 in Parquet
register-check run silver --silver SILVER --output-dir OUT
```

`edu-labour` and `societal-costs` need `--allow-expensive-stages` for
`production`: `matching` scans the whole delivery, and without the flag the
runner refuses rather than silently starting a multi-hour match.
`register-studies` gives the flag to every study under `--profile production`,
which is its default. `societal-costs --profile production` runs 21 stages and
`edu-labour --profile production` 38. `secondary_sensitivity`, the analyse-only
sensitivity and subgroup arm of Study II, is not among them. It reads a
completed run and writes into a subdirectory of it. `--list-profiles` on each
command gives the current counts.

`register-check` groups the checks by what they read: `delivery` for a raw
delivery before a run, `run` for a run directory, `bundle` for a downloaded
bundle. `register-check GROUP CHECK --help` shows a check's options. The
checks' earlier names, such as `register-verify` and `edu-labour-preflight`,
still run the same code. `register-check delivery utf8` needs the `repair`
extra and says so if it is missing.

Add `--dry-run` to any `edu-labour` invocation to resolve the plan and settings
without touching data.

## Design notes

**Stages talk through files.** Every stage writes named artifacts into a run
directory and reads its inputs from there, so any stage can be re-run alone
against a previous run's outputs, and `--resume` can skip what is already done.

**Estimators are written against polars.** Aalen-Johansen, Fine-Gray, Cox and
the multi-state occupancy are implemented directly rather than taken from a
modelling library, and each is validated against reference values from R with
those values checked into the tests. That keeps the numbers checkable line by
line and avoids a dependency that may not install in an offline environment.

**Disclosure control is part of the pipeline.** Results leave the secure
environment as a bundle of CSV tables with small cells suppressed and the
estimates resting on them blanked. Figures are rendered from that bundle rather
than from the raw artifacts, so a figure cannot show what the table beside it
was not cleared to show.

## What this covers, and what it does not

Secure-environment research is usually described with the [Five
Safes](https://fivesafes.org/), and it is worth saying plainly which of the five
a piece of software can touch at all.

| | |
|---|---|
| **Safe projects** | Your approval. Nothing here. |
| **Safe people** | Your institution's authorisation. Nothing here. |
| **Safe settings** | The host's environment, Statistics Denmark or the Danish Health Data Authority. Nothing here. |
| **Safe data** | **Partly here.** Person identifiers are pseudonymised by Statistics Denmark before delivery, and an export is refused outright if a CPR-shaped value reaches it. |
| **Safe outputs** | **Partly here.** Small cells are suppressed, dependent estimates blanked, file types and sizes checked against the transfer rules, and every bundle carries a manifest saying what was done and what was not. |

"Partly" is the operative word in both rows. The output checks are cell-by-cell
and cannot see a small number recovered by differencing across tables; each
bundle's `MANIFEST.txt` says so in its own text. Running this framework is not a
substitute for your host's output review, and it does not make an
unapproved analysis approved.

## Status

Research code, developed alongside the studies it implements. The public API is
not stable.

## Licence

MIT. See `LICENSE`.

