Metadata-Version: 2.4
Name: dst-disclose
Version: 0.1.1
Summary: Disclosure control for result tables taken home from Statistics Denmark's research machines
Keywords: disclosure control,small cells,register data,Statistics Denmark
Author: Tobias Kragholm
License-Expression: MIT
Requires-Python: >=3.8
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM

# dst-disclose

dst-disclose prepares result tables for taking home from a Statistics Denmark
research machine. It hides every cell of 1 to 4 people, then hides the further
cells needed so that no hidden cell can be worked out from the cells that are
shown, such as a total. It checks the finished files and never prints a value
that it hid.

You do not need to be a programmer. The tables can come from SAS, Stata, R or
Excel, as long as they are saved as CSV. A guide with worked examples is in
`docs/GUIDE.md`. The declaration file is described in `FORMAT.md`.

## Install

```
pip install dst-disclose
```

This installs the command `dst-disclose`. `dst-disclose --version` checks that
it is there. Commands are shown on one line, which is how cmd and PowerShell
take them. Put a path in quotes when it holds a space.

## Quickstart

Start with a folder of finished tables: CSV, Excel, Stata (.dta), SAS
(.sas7bdat) or Parquet files, one table each, or one table per Excel sheet.
The folder below is called `tables`. Nothing in it is changed.

1. Scan the folder for problems that need no declaration: a count of 1 to 4
   people shown in a table or in a text listing, a person number, a path to
   the server, a file type that is not allowed.

   ```
   dst-disclose scan --bundle tables --counts
   ```

   This finds only what one cell or one row shows. A count that a total and the
   other cells give back needs the declaration of the next steps.

2. Draft a declaration. It says which columns hold people and which rows add up
   to which.

   ```
   dst-disclose spec --tables tables --infer --out disclose.toml
   ```

3. Review `disclose.toml` in a text editor. When most tables count people in
   columns of one name pattern, the draft says so once under `[defaults]`. Every
   line the tool guessed has a comment above it. Check that each column it calls a count is a count of
   people. Each proposed sum (`[[relations]]`) says how many groups it was
   tested in. Keep the ones that are true for your tables in general, not by
   chance, and delete the others. Add what the tool cannot know. `FORMAT.md`
   lists every key. The tool never drafts arms and merges, because they are a
   choice about your paper.

4. Check the declaration against the tables.

   ```
   dst-disclose lint --spec disclose.toml --tables tables
   ```

5. Hide what must be hidden and write the folder to take home.

   ```
   dst-disclose apply --spec disclose.toml --tables tables --out export
   ```

   `apply` writes the tables, then reads them back and audits them as a reader
   would. It prints how many cells it hid and whether any of them could still
   be worked out.

6. Read `export\HIDDEN.csv` and `export\MERGES.csv`. `HIDDEN.csv` lists each
   hidden cell with the reason, never with its value. `MERGES.csv` lists the
   categories that caused further cells to be hidden, so you can decide to
   merge them in the paper instead.

7. Audit the folder as it will be transferred.

   ```
   dst-disclose audit --spec disclose.toml --bundle export
   ```

   The last line counts hidden cells that can be worked out. Zero is the goal.
   Any other number is a list of cells to look at, with the relation that gives
   each away. `docs/GUIDE.md` explains how to read it.

Take home the `export` folder, with `MANIFEST.txt` and the other files in it.

A toy run, with three groups of people and a row of totals, ends like this. The
cell for `A | other` is 4 people. The cell for `A | studying` is hidden as well,
because the total and the other rows would give it back.

```
group,outcome,n_people
A,overall,60
A,working,34
A,studying,..
A,other,..
```

## What a pass means

A clean audit means that, with the relations you declared, no hidden cell can be
worked out from the shown ones. It does not mean that Statistics Denmark will
approve the files. You remain responsible for what you take home, and
Statistics Denmark checks samples by hand.

The audit knows only the relations in the declaration. A relation that is
missing leaves a hidden cell that can be worked out, and the audit cannot see
it. The tool does not know what has been published elsewhere or what is in a
figure, and it knows what you transferred earlier only when you give it those
bundles with `--earlier`. Figures are judged by Statistics
Denmark as the table behind them, so take the table home too.

A share in a CSV or text file is read at the digits it is printed with. `0.067`
in a group of 600 is exactly 40 people, and in a group of 6,000 anything from
399 to 405, so a hidden count next to it is not worked out through it. See
`docs/GUIDE.md`.

## Commands

Every command that reads a declaration takes `--spec`. The threshold is 5 unless
the declaration or `--threshold` says otherwise.

| Command | What it does |
|---|---|
| `scan --bundle PATH [--counts] [--policy FILE] [--report FILE.csv]` | Runs the file checks over a folder: file types, sizes, person numbers, server paths and long digit strings. With `--counts` it also looks through every file and listing for a count of 1 to 4 people, needs no declaration, and writes its findings without values to `--report`. |
| `spec --tables DIR [--infer] [--out FILE]` | Drafts a declaration. With `--infer` it also proposes which rows add up, with its evidence. Without `--out` it prints the draft. In PowerShell 5, use `--out` and not `>`, which writes UTF-16. |
| `lint --spec FILE --tables DIR` | Checks a declaration against the tables and reports what looks wrong. |
| `apply --spec FILE --tables DIR --out DIR [--no-net] [--earlier DIR]... [--members DIR] [--only NAME]...` | Hides cells, writes the folder to take home and audits it. It also uses the relations the tables obey, a label supports and the declaration lacks, and names each cell hidden only through one so you can declare the relation. `--no-net` leaves them out. |
| `explain --spec FILE --tables DIR [--table NAME] [--where COL=VALUE] [--no-net] [--earlier DIR]...` | Says why each cell is hidden, without values, so the answer may leave the server. Takes `--no-net` as `apply` does, so the two agree. |
| `audit --spec FILE --bundle DIR [--net] [--earlier DIR]... [--members DIR]` | Checks a finished folder: can any hidden cell be worked out? Off by default, unlike in `apply`: with `--net` it adds the relations the shown values obey, and finds fewer than `apply` does, since hidden values are unknown. |

`audit` takes two options for older bundles. `--exact-arms` reads every arm as
exact, for a bundle made before unreported arms were rounded. `--older-layout`
leaves out the declared columns a table lacks and the relations that need them,
and prints each one as a warning. `apply` never does this.

## Earlier bundles

A reader has every bundle you took home, not only the latest. Give `audit`,
`apply` and `explain` each earlier bundle of the same study with `--earlier
DIR`, repeated for more. The tool reads them with the same declaration, treats
every value they show as known, and checks the new bundle together with them.

```
dst-disclose audit --spec study.toml --bundle export_2026-11-12 \
    --earlier export_2026-11-07 --earlier export_2026-10-20
```

A cell is the same cell in two bundles when the table, the values of the
declared key columns and the column agree. Row numbers are not used. A cell an
earlier bundle shows is fixed for the reader. A cell it marks `<5` is known to
be 1 to 4. A bundle with `<N` marks is read as the older layout, as
`audit --older-layout` reads it. A table the new run no longer has still counts:
its cells tie others together through the declared relations.

If cells of one table are shown in both bundles with different values, the two
runs are not the same data for that table. The tool warns, names the table,
the earlier bundle and the number of cells that differ, and does not identify
that table's cells. Differences between the two versions are not checked.

`apply` does not hide a cell again that an earlier bundle shows, since the
reader has it, and does not use it to protect another cell. Among the cells that
can protect, it prefers those an earlier bundle also hid, so that the same cells
are hidden from one release to the next. If such a cell is 1 to 4 people, the
earlier bundle already gave it away.

What the earlier bundles give away on their own is not the new bundle's doing.
The tool first audits the earlier bundles together, with every cell of the new
bundle unknown unless an earlier bundle shows it. A hidden cell that can be
worked out, or not shown safe, already then is printed as `already given away
by earlier bundles`. `apply` hides nothing for it. These lines are counted on
their own line, and do not change the exit code, since nothing in the new
bundle can undo them. A hidden cell that only the new bundle's shown cells give
away, including a cell that only an earlier bundle has, is a finding of the new
bundle, and `apply` hides cells of the new bundle to protect it. See
`docs/GUIDE.md`.

## Members

When you know who is behind each count, give the tool the people. For each
table `NAME`, a file `NAME.members.csv` or `NAME.members.parquet` lists the
persons counted in each count cell, and `--members DIR` reads them. The tool then
knows every overlap between the cells of all the tables, so a margin, a
subgroup or a sensitivity analysis that differs from the main one by a few people
needs no declared relation. The people are never written anywhere. `FORMAT.md`
describes the files, and `docs/GUIDE.md` has an example.

A table that is a sensitivity analysis of almost the same people can be written
rounded, with `round_counts = 10` in its `[tables.NAME]` block, or through
`[arms]`. A reader then knows each count only to within the rounding, and the
tool checks, with the people, that this is enough: if the ranges still narrow a
group of 1 to 4 people, it hides more. Without the rounding, the few people who
differ show up in nearly every cell and a lot is hidden.

`apply --only NAME` writes one table of a project at a time and treats the other
tables as taken home before, so that each new table is checked against all the
earlier ones, which cannot change.

## What is written beside the tables

`apply` writes these files into the `--out` folder. None holds a hidden value,
and they are meant to be transferred with the tables.

| File | Holds |
|---|---|
| `HIDDEN.csv` | Each hidden cell: table, row, column and reason, and for a cell hidden to protect another, the relation. |
| `MERGES.csv` | The categories that caused hiding, so that you can decide on a merge. |
| `EXPORT_CHECKS.csv` | One row for each check and table: file type, size, person numbers, server paths, long numbers and floats, with the outcome. |
| `MANIFEST.txt` | The threshold, how many cells were hidden, the arms, rounded tables and merges, and the files with their sizes. |

Tables are written as CSV. A hidden cell is written `..`. An estimate left out
because a number it rests on is hidden is written empty. Zero is shown as 0. A
table over 3,000 KB is split into parts, and so are `HIDDEN.csv` and
`MERGES.csv`: `HIDDEN.part01.csv`, `HIDDEN.part02.csv` and so on, each with the
header.

## Exit codes

| Code | Meaning |
|---|---|
| 0 | Done, and nothing found. |
| 1 | A check failed: a hidden cell that can be worked out, a failed file check, or a table that could not be protected. With `--earlier`, this includes a cell that can be worked out only together with an earlier bundle. A small count an earlier bundle shows, or a cell the earlier bundles already give away, does not count. |
| 2 | The command could not run: a missing file, a malformed table or an invalid declaration. The message follows `error:`. |

## Policy file

`scan --counts` takes `--policy FILE`, a small TOML file:

```toml
threshold = 5            # some registers allow 3
[files]
"income/*.csv" = 3       # a pattern with no "/" matches the file name
```

A declaration sets its own `threshold`, and `--threshold` overrides both.

## Templates

A table of a common shape can name a template instead of writing its roles and
relations: `template = "contrast"`, `"table1"`, `"prevalence"`, `"composition"`,
`"regression"` or `"survival_curve"`. Anything written in the table's block
replaces what the template gives. `FORMAT.md` describes each one.

