Metadata-Version: 2.5
Name: dst-parquet-layout
Version: 0.1.4
Summary: Plan and build a long-term Parquet layout for a raw DST register delivery
Author-email: Tobias Kragholm <tkragholm@gmail.com>
License-Expression: MIT
License-File: LICENSE
Keywords: data-management,denmark,parquet,registers,sas7bdat
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Typing :: Typed
Requires-Python: >=3.11
Provides-Extra: convert
Requires-Dist: polars>=1.41; extra == 'convert'
Requires-Dist: sas7bdat-polars==0.10.0; extra == 'convert'
Provides-Extra: dev
Requires-Dist: mypy>=1.11; extra == 'dev'
Requires-Dist: pytest>=8; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Description-Content-Type: text/markdown

# dst-parquet-layout

Plans and builds a long-term Parquet layout for a raw DST register delivery:

```
<delivery>/<provider>/<register>/year=<YYYY>/<source file>.parquet
```

for example `2025/dst/bef/year=2015/bef201512.parquet`. A register whose files
carry no year keeps everything except the `year=` level.

The delivery it was written against is 3,032 GB in 1,469 files. Converting that
is a day's work nobody wants to do twice, and the decisions that would make it
wrong are knowable in advance from the filenames and the headers. So the package
has two halves over one set of rules. `plan` says where every file would go and
what it could not decide alone. `convert` does the work, and refuses any register
the plan left blocked.

## Install

```
pip install dst-parquet-layout            # planning only, no dependencies
pip install "dst-parquet-layout[convert]" # adds polars and the SAS reader
```

The planning half is standard library only, on purpose. The people who have to
agree with a layout should be able to read the plan without a numeric stack, a
compiled reader, or a working package mirror behind a research firewall.

## Use

Review the rules against every naming shape in a delivery, from an exported
register map, with no server access:

```
dst-parquet plan --map .../2026_register_map --out-dir plan/
```

Plan every file, from the map's per-file detail table or from a live walk:

```
dst-parquet plan --files detail/files.csv --out-dir plan/
dst-parquet plan --root /data/rawdata/<delivery> --out-dir plan/
```

Date a delivery directory whose name does not say when it arrived:

```
dst-parquet dates --root /data/rawdata/<delivery> --out-dir dates/
dst-parquet dates --dates dates/DELIVERY_DATES.csv
```

Every `.sas7bdat` carries the timestamp of the SAS session that wrote it, which
is when the extract was run rather than when it was copied. An extraction takes
days and deliveries are months apart, so sorting the timestamps and cutting at
the large gaps recovers the deliveries. It proposes a `[[delivery]]` entry only
where the clusters reproduce the months the dated directories are already named
after.

Convert, once the plan is clean:

```
dst-parquet convert --root /data/rawdata/<delivery> --out /data/parquet \
    --plan plan/ --files detail/files.csv --jobs 4
```

`plan` writes six files. `TREE.txt` is the whole layout as a directory tree,
one line per register directory, which is the form to hand somebody deciding
whether this is the structure they want. `SUMMARY.txt` is the status of
every register directory and every open decision. `REGISTER_PLAN.csv` is one row per
destination directory. `PLACEMENT.csv` is one row per source file.
`REVIEW.csv` is the decisions that need a yes and the ones that block.
`PROVENANCE.json` records the tool version, the time, and digests of the rules
and the file list.

Both commands exit non-zero when something is wrong: `plan` when a register is
blocked, so it can gate a conversion in a script, and `convert` when it skipped
a register or wrote a file whose row count disagrees with its header.

## What a conversion is allowed to do

`convert` takes `--plan`, and refuses to run if the rules or the file list have
changed since that plan was written. A plan is read by people and acted on by a
program, sometimes weeks apart, and nothing about editing `rules.toml` in
between leaves a mark on the plan. Without the check the conversion quietly
builds a tree nobody agreed to. `--accept-rules-change` overrides it, out loud.

Every written file is checked against the row count its source header declared,
and the count that was actually written is recorded in the register directory's
`_layout.json` beside the rules digest that produced it. `--verify-utf8` reads
each output back and counts U+FFFD, the replacement character a lossy decode
leaves behind, which no later repair can undo.

A run is resumable: a file that is already there is left alone, and the partial
files an interrupted run left behind are swept before anything new is written.
`--jobs` converts several register directories at once; they share no state and
no output path. One line per directory goes to stderr as it finishes, with a
running share of the bytes, the time elapsed and an estimate of the time left;
`--quiet` turns it off.

A register directory is written once and then left alone. If a source file is
corrected upstream, the resume check cannot tell that from a finished file: it
sees a `.parquet` of the right name and skips it, so the correction would never
land and nothing would say so. Each source's size and the creation stamp in its
SAS header are recorded beside the data, and a conversion whose sources no
longer match refuses, names the files that differ and writes nothing.

A register that declares a key in `[[series]]` has it checked in every written
snapshot, and a snapshot holding one key twice fails the run. Every written file
also gets a fingerprint of its content in `_layout.json`, independent of row
order and taken column by column, and `dst-parquet resolve --out <tree>` uses
them to name one copy of each snapshot across deliveries and say whether the
others hold the same rows. It writes `RESOLUTION.csv` and moves nothing.

`--new-version bef` then writes the corrected data to the next free
`bef_v2` beside the directory already there, which is left exactly as it is.
A path somebody has already read from never changes meaning.

## What is still open

`OPEN_ITEMS.md`, beside this file in the repository, is the list: one question
only a person can answer, seven decisions the rules already make that want a
yes, nine things left undone in the code on purpose or since closed, and the
fact that none of it has run against a real delivery yet.

## The rules

`src/dst_parquet_layout/rules.toml` holds every judgement, one entry each: which
directory is which delivery and provider, which files are the same table, which
registers are not partitioned by year, how a type conflict is promoted, which
collisions are resolved and how, and what several files in one year of a
register are to each other. Changing a decision is a change to that file,
not to the program. `tests/` pins the current answers, so an edit says out loud
which files it moved.

## What the rules have to survive

Each of these is observed in an August 2026 walk of one delivery.

The register is not the alphabetic prefix of the filename. `soma_l2_t19_proc19`
and `soma_l3_t19_diag19` share a prefix and are different tables. Read as one
register they give a schema in which 197 of 197 columns go missing somewhere.

115 files carry no year, for three different reasons: code lists with no time
dimension, event tables pooled across the whole period, and tables the provider
split into numbered shards. Each needs a different answer, and a register
directory that is partitioned for some of its files and not others cannot be
scanned at all.

A pooled table carries no year in its filename and thirty of them in its rows.
`[undated.partition_on]` names the column for 27 of them, taken from the
register map and each present in every file of its register, and they are cut
into the same `year=` shape as every dated register, in the streaming engine,
without the whole table being held. Where the register offers both `dato_x` and
`datotidspunkt_x` the date is chosen, because the datetime variants are absent
from one of the three files. The choice is the event and not the report: a
reporting timestamp would put a 2019 birth in `year=2021` because that is when
somebody typed it in.

The rest are written whole and `[undated.written_whole]` records why, so a
table with no date column at all reads as settled rather than as an outstanding
decision. A row whose date is missing keeps a null year in the standard
`__HIVE_DEFAULT_PARTITION__` directory: it is never dropped and never given a
year it does not have, and the count is reported.

`dnt_1159.sas7bdat` parses as the year 1159 under any rule that takes four digits
and stops. `soma_l2_t19_drgkont19` carries its year as `19`.
`dimcancergruppering_icd10` carries no year at all. `bef199312` carries a month
that must not be lost.

350 register-years appear in more than one delivery and 139 of those disagree on
row count, usually by a handful of rows: `lmdb` 2023 is 58,161,878 in POPC and
58,161,741 in POPFULD, two extracts of one year three months apart. Deliveries
stay apart and nothing is deduplicated across them. Two files that would land in
one partition with nothing to tell them apart block until a rule says which wins.

A month in a filename is a period in one register and a cut-off in another, and
the filename does not say which. `bef200803` is the population at the end of March 2008
and all four of that year's files are wanted, where `lmdb202406` is every
prescription of 2024 up to the end of June and is already inside `lmdb202412`.
Which day `bef`'s month means is settled by its row counts, each of which is the
population Statistics Denmark publishes for the first day of the following
month, so every `bef` row carries `_ref_date` as the last day of its month.
Read the same way, one of them counts half a year twice. So a register that
arrives more than once a year declares its grain in `[[series]]`, the plan
checks that against the file sizes before acting on it, and a year holding more
than one file whose register declares nothing refuses to convert. For a
cumulative register the conversion then asks the file it kept how many of its
rows fall before the dropped file's cut-off, and fails if that is not the number
the dropped file's header claimed.

A year in a filename is a reading of a filename, not a measurement. A file more
than eight times the median file of its own register is reported with the ratio,
because `udd_grundskole_fravaer_kmd2024` is 39.7 million rows where that
register's years run 3 to 5 million: the whole 2005 to 2024 history under the
delivery's year. One file in a partition is otherwise never asked to explain
itself. Across all 1,469 files the next largest ratio is 3.3.

624 of 2,491 register-columns are absent from at least one file of their own
register, and 60 header changes fall across 29 registers. Under one schema each
absence becomes a null, which is a different claim, so each register directory
carries a `_layout.json` naming the files each column was absent from.

722 column names are not all-uppercase and 37 appear in both cases inside one
register, so `lmdb` offers CPRTJEK and cprtjek as two variables. 721 columns move
position between years. Schemas are therefore built by name, never by order, and
names are uppercased. Five columns change dtype across their years and are
promoted, Float64 and String to String because all four such columns hold codes.

Every file declares windows-1252 and Parquet declares UTF-8. Conversion checks
the declaration per file and refuses a register that declares anything else.

## Getting it onto the analysis server

Either route works. The wheel installs from a package mirror, and the package has
no dependencies unless the `convert` extra is asked for. Failing that, the whole
package fits through `tools/typed_transfer`, which compresses a package into one
checksummed script to type in.

## Development

```
pytest                 # 88 tests
ruff check src tests
mypy
```

Tests run against the source tree without an install, and against the rules that
ship with the package rather than a fixture copy.

Two of them cover the conversion. `test_convert.py` uses a stand-in for the SAS
reader, which needs nothing installed and proves the code agrees with itself.
`test_real_sources.py` runs against six real `.sas7bdat` files under
`tests/fixtures/`, read by `sas7bdat_polars`, which is the only way to find out
whether it agrees with a `.sas7bdat`: that windows-1252 comes through as UTF-8
with `æøå` intact, and that a SAS date, which is a float counting days from
1960, arrives as a date that can be partitioned on. It skips where the reader is
not installed.

The fixtures are committed so the tests need no Rust toolchain. Regenerate them
with the writer that made them, from the repository root:

```
cargo run -p sas7bdat-writer --example pooled_fixture -- \
    tools/dst-parquet-layout/tests/fixtures
```

The writer stamps a fixed creation time, so a re-run is an empty diff.
