Metadata-Version: 2.4
Name: dst-parquet-layout
Version: 0.1.8
Provides-Extra: convert
Summary: Plans and builds a long-term Parquet layout for a raw DST register delivery
Keywords: parquet,sas7bdat,register data,Statistics Denmark
Author: Tobias Kragholm
License-Expression: MIT
Requires-Python: >=3.8
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM

# dst-parquet

Plans and builds a long-term Parquet layout for a raw DST register delivery:

```
<delivery>/<provider>/<register>/year=<YYYY>/<source file>.parquet
```

for example `2025/dst/bef/year=2015/bef201512.parquet`. A register whose files
carry no year keeps everything except the `year=` level.

The delivery it was written against is 3,032 GB in 1,469 files. Converting that
is a day's work nobody wants to do twice, and the decisions that would make it
wrong are knowable in advance from the filenames and the headers. So there are
two halves over one set of rules. `plan` says where every file would go and what
it could not decide alone. `convert` does the work, and refuses any register the
plan left blocked.

It is a single binary. It was first written in Python, and this is that program
ported to Rust: the same commands, flags, rules file and layout, with every plan
file the same byte for byte. The conversion reads each source once, puts it into
its register directory's schema, writes it and measures it in the same pass, and
holds about a gigabyte however large the source is.

## Install

```
pip install dst-parquet-layout
```

The wheel holds the binary and nothing else, so it needs no other package. The
name is the one the Python implementation was published under, so a line that
installed that installs this, `[convert]` included.

`dst-parquet-system-heap` is the same program with the platform's own memory
allocator in place of mimalloc. It exists to time the two against each other on
the machine that runs the conversion.

## Use

Review the rules against every naming shape in a delivery, from an exported
register map, with no server access:

```
dst-parquet plan --map .../2026_register_map --out-dir plan/
```

Plan every file, from a per-file list or from a walk of the delivery. A walk
reads each file's header for its row count:

```
dst-parquet plan --files files.csv --out-dir plan/
dst-parquet plan --root E:/rawdata/<delivery> --out-dir plan/
```

Date a delivery directory whose name does not say when it arrived:

```
dst-parquet dates --root E:/rawdata/<delivery> --out-dir dates/
dst-parquet dates --dates dates/DELIVERY_DATES.csv
```

Convert, once the plan is clean:

```
dst-parquet convert --root E:/rawdata/<delivery> --out E:/workdata/<project>/parquet \
    --plan plan/ --jobs 4
```

On a terminal, `convert` keeps a panel at the bottom of the window: the share of
bytes done, the rate over the last minute, the time left, and a line per file
being read. Each register directory leaves a line above it when it finishes.
With its output sent to a file it prints those lines as they happen and the
panel's figures every five minutes. `--quiet` prints only the summary.

Name one copy of each snapshot across deliveries, and compare the others:

```
dst-parquet resolve --out E:/workdata/<project>/parquet --out-dir resolution/
```

`plan` writes six files. `TREE.txt` is the whole layout as a directory tree, one
line per register directory, which is the form to hand somebody deciding whether
this is the structure they want. `SUMMARY.txt` is the status of every register
directory and every open decision. `REGISTER_PLAN.csv` is one row per destination
directory, `PLACEMENT.csv` one row per source file, and `REVIEW.csv` the
decisions that need a yes and the ones that block. `PROVENANCE.json` records the
version, the time, and digests of the rules and the file list.

Both commands exit non-zero when something is wrong: `plan` when a register is
blocked, so it can gate a conversion in a script, and `convert` when it skipped a
register or found something wrong in what it wrote.

## What a conversion is allowed to do

`convert` takes `--plan`, and refuses to run if the rules or the file list have
changed since that plan was written. A plan is read by people and acted on by a
program, sometimes weeks apart, and nothing about editing `rules.toml` in between
leaves a mark on the plan. `--accept-rules-change` overrides it, out loud.

Every source is read once, and what is checked is checked as the rows go past:

- The rows written are compared with the count in the source's header, less any
  rows the header marks deleted. A header that declares exactly 4,294,967,295
  rows has a saturated counter, which says nothing about the file: it is read to
  its last page and reported, not compared.
- Every string cell is checked for U+FFFD, the replacement character a lossy
  decode leaves behind, which no later repair can undo. The reader decodes every
  windows-1252 byte, so in this delivery it can only appear in a file that is not
  what it declares. A file holding one stays written and fails the run.
  `--verify-utf8` is still accepted, and no longer needed.
- A stock series' key, `PNR` for `bef`, is checked for a person held twice.
- For a cumulative series, the file kept for a year is asked how many of its rows
  fall before each dropped file's cut-off, against the count in the dropped
  file's header.
- Every file gets a fingerprint of its content, independent of row order and
  taken column by column, which `resolve` compares across deliveries.

What was measured goes into the register directory's `_layout.json`, and into each
file's own Parquet footer, so a file whose sidecar is lost still says how many
rows it holds. A file that is already there is not read again, so a run that
resumes it reports again whatever the run that wrote it found wrong, marked as an
earlier run's finding, and fails as that run did.

A run is resumable: a file that is already there is left alone, and the partial
files an interrupted run left behind in the directories being converted are swept
first. `--jobs` converts several register directories at once, largest first.
`--threads` caps what the whole run uses, decoding and encoding together.

A register directory is written once and then left alone. Each source's size and
the creation stamp in its SAS header are recorded beside the data, and a
conversion whose sources no longer match refuses and names the files that differ.
`--new-version bef` then writes the corrected data to the next free `bef_v2`
beside the directory already there, which is left exactly as it is.

`--io-backend` chooses how sources are read. `buffered`, the default, reads each
file in large pieces on several threads at once. `mmap` maps the file into memory
instead, and `auto` maps a local file and reads a network share in pieces. A
conversion reads every byte of a file once, front to back, which is what large
reads in parallel suit.

## How it is written

Files are written with zstd at the level the rules set. Floats are split by byte
stream and dates and times delta-encoded, and a column nearly every value of which
is distinct in its first rows does without a dictionary, which is what lets those
encodings apply. On a synthetic `bef`-shaped file that made the output 28 MB where
the Python implementation's polars writer made 72 MB.

Column types are the reader's own, carried through, with one exception. A SAS
TIME column is seconds since midnight, and the reader gives it as a duration,
because nothing stops a SAS time from being negative or a day and more. It is
written as a time of day, which R, polars and DuckDB read as a clock time; a
value outside 00:00 to 24:00 is written as null, and the summary and the
register's `_layout.json` say how many, in which file and column. The rule
`[layout] sas_time = "duration"` writes every value as it is instead. Where the
rules promote a code column from Float64 to String, a whole number is written
without its decimal point: `1301.0` is `"1301"`.

A tree converted before SAS times became times of day holds them as durations.

```
dst-parquet retime --out E:/workdata/<project>/parquet
```

rewrites each file that has such a column, in place, with the same rule for
values outside a day, and records the counts in the register's `_layout.json`.
`--out` may be the tree or any directory inside or above it: each register's
files are found from where its own `_layout.json` sits, and one whose files are
not there is named. Several files are rewritten at once (`--jobs`), each file's
row groups are decoded in parallel, and the progress panel shows them as
`convert` does. A time hashes as the duration it was, so a file's fingerprint
only changes where a value became null. `--dry-run` reads only the duration
columns, reports what it would change, and writes nothing.

## What is still open

`OPEN_ITEMS.md`, beside this file, is the list: the questions only a person can
answer, the decisions the rules make that want a yes, and what is left undone in
the code on purpose.

## The rules

`rules.toml` holds every judgement, one entry each: which directory is which
delivery and provider, which files are the same table, which registers are not
partitioned by year, how a type conflict is promoted, which collisions are
resolved and how, and what several files in one year of a register are to each
other. The binary carries the file it was built with, and `--rules PATH` reads
another. Changing a decision is a change to that file, not to the program, and
`tests/` pins the current answers, so an edit says out loud which files it moved.

### Changing a few of them

A delivery that does not fit the rules is blocked in the plan, and the reason
says which rule to change and what to write. To see what decides a register or a
file, and the lines that would change it:

```
dst-parquet rules explain forloeb --plan plan/
dst-parquet rules explain bef201912.sas7bdat
```

The changes go in an overrides file, which holds only the entries it changes, in
the sections `rules.toml` uses:

```toml
[undated.partition_on]
forloeb = "DATO_HENVISNING"

[layout]
compression_level = 6
```

A key replaces the key it names. An entry in a list, such as a `[[series]]` or a
`[[delivery]]`, is matched by what identifies it (`register`, `snapshot`) and
only the fields given change; `remove = true` deletes it, and a new
`[[register]]` is tried before the built-in ones. A register named under one of
`[undated]`'s treatments leaves the others. Check the file, and see what it
changes, before planning with it:

```
dst-parquet --overrides overrides.toml rules check
dst-parquet --overrides overrides.toml plan --root E:/rawdata/<delivery> --out-dir plan/
dst-parquet --overrides overrides.toml convert --root ... --plan plan/
```

The plan lists the changes in `SUMMARY.txt`, keeps a copy as `OVERRIDES.toml`
and pins the rules and the overrides together, so a conversion without the same
overrides is refused and told which flag it is missing. `dst-parquet rules show`
prints the built-in rules, comments and all, which is where every default and
its reason is written down.

## What the rules have to survive

Each of these is observed in an August 2026 walk of one delivery.

The register is not the alphabetic prefix of the filename. `soma_l2_t19_proc19`
and `soma_l3_t19_diag19` share a prefix and are different tables. Read as one
register they give a schema in which 197 of 197 columns go missing somewhere.

115 files carry no year, for three different reasons: code lists with no time
dimension, event tables pooled across the whole period, and tables the provider
split into numbered shards. Each needs a different answer, and a register
directory that is partitioned for some of its files and not others cannot be
scanned at all.

A pooled table carries no year in its filename and thirty of them in its rows.
`[undated.partition_on]` names the column for 27 of them, and they are cut into
the same `year=` shape as every dated register. Where a register offers both
`dato_x` and `datotidspunkt_x` the date is chosen, because the datetime variants
are absent from one of the three files. A row whose date is missing keeps a null
year in the standard `__HIVE_DEFAULT_PARTITION__` directory: it is never dropped
and never given a year it does not have, and the count is reported.

`dnt_1159.sas7bdat` parses as the year 1159 under any rule that takes four digits
and stops. `soma_l2_t19_drgkont19` carries its year as `19`.
`dimcancergruppering_icd10` carries no year at all. `bef199312` carries a month
that must not be lost.

350 register-years appear in more than one delivery and 139 of those disagree on
row count, usually by a handful of rows. Deliveries stay apart and nothing is
deduplicated across them. Two files that would land in one partition with nothing
to tell them apart block until a rule says which wins.

A month in a filename is a period in one register and a cut-off in another, and
the filename does not say which. `bef200803` is the population at the end of
March 2008 and all four of that year's files are wanted, where `lmdb202406` is
every prescription of 2024 up to the end of June and is already inside
`lmdb202412`. So a register that arrives more than once a year declares its grain
in `[[series]]`, the plan checks that against the file sizes before acting on it,
and a year holding more than one file whose register declares nothing refuses to
convert.

A year in a filename is a reading of a filename, not a measurement. A file more
than eight times the median file of its own register is reported with the ratio,
because `udd_grundskole_fravaer_kmd2024` is 39.7 million rows where that
register's years run 3 to 5 million.

624 of 2,491 register-columns are absent from at least one file of their own
register, and 721 columns move position between years. Schemas are therefore
built by name, never by order, names are uppercased, and each register directory
carries a `_layout.json` naming the files each column was absent from.

Every file declares windows-1252 and Parquet declares UTF-8. Conversion checks
the declaration per file and refuses a register that declares anything else.

## Development

```
cargo nextest run -p dst-parquet
cargo clippy -p dst-parquet --all-targets -- -D warnings
just dst-parquet-wheel      # the Windows and macOS wheels, sanitised and scanned
just dst-parquet-publish    # uploads them, after asking
```

The conversion tests write real `.sas7bdat` files with `sas7bdat-writer` and read
them through the published reader, so they test the same decode a delivery goes
through. `tests/real_sources.rs` converts a small synthetic delivery of six such
files, written by `tests/synthetic/` into each test's own directory, since no
`.sas7bdat` file is committed.

