Metadata-Version: 2.4
Name: lintle
Version: 0.4.0
Summary: Validate and clean Two-Line Element (TLE) satellite-tracking files
Project-URL: Homepage, https://github.com/elfensky/lintle
Project-URL: Repository, https://github.com/elfensky/lintle
Project-URL: Issues, https://github.com/elfensky/lintle/issues
Project-URL: Changelog, https://github.com/elfensky/lintle/blob/main/CHANGELOG.md
Author-email: Andrei Lavrenov <andrei@lav.ren>
License-Expression: MIT
License-File: LICENSE
Keywords: orbital-mechanics,satellite,sgp4,space-track,tle,two-line-element,validator
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering :: Astronomy
Classifier: Topic :: Utilities
Requires-Python: >=3.14
Requires-Dist: rich<14,>=13
Description-Content-Type: text/markdown

# lintle

A validator and cleaner for **Two-Line Element (TLE)** corpus files exported
from [space-track.org](https://www.space-track.org/).

It audits a TLE file against the standardized TLE specification, repairs the
systematic export defects, and emits a **uniform, de-defected** corpus that any
SGP4 / orbital-mechanics library can ingest directly. Records it cannot safely
repair are quarantined — never silently mangled — into a per-file sidecar
detailed enough to file a defect report with space-track.

---

## What problem it solves

A TLE record is two fixed-width lines, each *exactly* 69 ASCII columns, with a
mod-10 checksum in column 69. Bulk historical exports from space-track carry
two systematic, era-specific defects:

- **Trailing `\` artifact** — almost every `Line 1` has an extra `\` byte
  appended before the newline.
- **Missing checksum digit** — many records were exported without their
  column-69 checksum, leaving 68-column lines.

These appear independently and in combination, and a small fraction of records
are genuinely corrupt (garbled columns, orphaned lines, wrong lengths).
`lintle` distinguishes the safely-repairable from the genuinely-corrupt and
treats each correctly.

## What lintle will and won't reconstruct

**lintle never invents data.** It emits only information that was already in the
record. The single exception is the column-69 checksum — and it is an exception
*precisely because* the checksum carries no information of its own: it is a
deterministic mod-10 function of columns 1–68, so recomputing a missing one
asserts nothing the record didn't already say. This is the **redundancy paradox**
at the heart of the tool: the only field safe to rebuild is the one field that
was redundant to begin with.

Every other defect that could only be "fixed" by guessing a real data character
is **quarantined, never repaired**. A mod-10 checksum has a 1-in-10 chance of
accepting a wrong line by luck, so inventing an orbital-data character risks
emitting a record that *looks* valid but is silently wrong — the one outcome
worse than dropping it. When in doubt, lintle quarantines.

See [How it works](#how-it-works) below for the fix-class table that
operationalises this principle.

## How it works

**One validator, used two ways.** A single module (`tle.py`) defines what a
"perfect" TLE record is — column layout, semantic ranges, and the mod-10
checksum. The `validate` command reports defects against that definition; the
`clean` command reuses the *exact same* validator and only emits records that
pass it.

**The validated-transformation principle.** The cleaner never applies a fix and
hopes. It applies a candidate fix, then re-runs full validation on the result,
and commits the fix *only if it now passes*. Consequently the cleaner cannot
turn a bad record into a wrong-but-valid-looking one, and every line in the
output is valid by construction.

**Five fix classes, in decreasing order of safety:**

| Class | Examples | Action |
|-------|----------|--------|
| Content-preserving | trailing `\`, CRLF, trailing whitespace | auto-fix (checksum survives as an independent check) |
| Reconstructed-checksum | a record exported without its column-69 digit | recompute the checksum from intact columns 1–68 |
| Content-shifting | leading whitespace / BOM | trim, then re-validate; quarantine if it fails |
| Structural | blank / whitespace-only lines | drop, resynchronise pairing |
| Corrupt | bad checksum, wrong length, orphan line, garbled columns | **quarantine** |

**Streaming and parallel.** Files are read in binary, line by line, in constant
memory — a 3 GB file never loads into RAM. Records are paired by a prefix-driven
state machine that resynchronises on every `1 ` line, so one missing line
cannot cascade into mispaired records. Each input file is processed in its own
worker process.

## Requirements

- Python 3.14+
- [`uv`](https://docs.astral.sh/uv/) for environment and dependency management

`lintle`'s only runtime dependency is **`rich`** (`>=13,<14`, terminal rendering for the
`clean` progress UI); everything else is standard library. `sgp4` is a dev-only dependency,
used as a test oracle.

## Installation

```bash
uv sync
```

This creates the virtual environment and installs the dev dependencies. No
build step is needed to run the tool.

## Usage

The console script is `lintle`:

```bash
# Audit only — report defects, write nothing
uv run lintle validate [path]

# Produce cleaned output + quarantine sidecars
uv run lintle clean [path]

# Explain a rule ID or fix tag — definition, examples, source citation
uv run lintle explain <TAG>
```

`python -m lintle ...` is equivalent to `uv run lintle ...`.

**Arguments and options:**

| Option | Default | Meaning |
|--------|---------|---------|
| `path` | `data/source` | A single file or directory. A directory is globbed for `tle*.txt` (tool output `*.cleaned.txt` / `*.broken.txt` is excluded). |
| `--out-dir DIR` | `data/output` | Where `clean` writes its output. Created if absent. |
| `--jobs N` | CPU count | Number of files processed in parallel. Lower it if a slow disk causes I/O contention. |
| `--report text\|json` | `text` | Summary format. |
| `--max-quarantined N[%]` | `0` | Exit non-zero only if MORE than N records were quarantined; or, with a trailing `%`, more than `N%` of routed records (`clean + quarantined`) were quarantined. Default `0` ≡ "any quarantine fails". |
| `--resume` | — | (`clean` only) Resume an interrupted run without prompting, even in a non-interactive context. |
| `--no-resume` | — | (`clean` only) Ignore any checkpoint and start fresh, clearing prior outputs in `--out-dir`. |

**Examples:**

```bash
# Validate the whole corpus
uv run lintle validate data/source

# Clean one file
uv run lintle clean data/source/tle2022.txt --out-dir data/output

# Clean the corpus, capture a machine-readable summary
uv run lintle clean data/source --report json > run-summary.json

# CI gate: tolerate up to 100 quarantined records before failing the job
uv run lintle clean data/source --max-quarantined 100 --report json > run-summary.json

# CI gate (scale-invariant): tolerate up to 1% of routed records
uv run lintle clean data/source --max-quarantined 1% --report json > run-summary.json

# Look up what a rule ID or fix tag means, with a verified example
uv run lintle explain TLE-CHK-001
uv run lintle explain reconstructed-checksum

# Resume an interrupted run (e.g. the laptop slept mid-corpus) — re-run the same command
uv run lintle clean data/source --out-dir data/output  # prompts to resume; auto-resumes in CI

# Start completely fresh, discarding any checkpoint and prior outputs
uv run lintle clean data/source --out-dir data/output --no-resume
```

**Exit codes:**

| Code | Meaning |
|------|---------|
| `0` | Quarantine count (or rate) is at or below `--max-quarantined` (default `0`). |
| `1` | Quarantine count (or rate) exceeded `--max-quarantined`. |
| `2` | Operational error — no input files, disk shortfall, lock held, stale/corrupt/declined resume, or a file that failed to process. |
| `129` | Killed by `SIGHUP` (e.g. terminal closed). |
| `130` | Interrupted with `Ctrl-C` (`SIGINT`). |
| `143` | Terminated by `SIGTERM` (e.g. from a scheduler). |

Repairable defects (including the near-universal trailing `\`) do **not** raise
the exit code above 0 — almost every raw file contains them. `--max-quarantined`
preserves the meaningful `2` (operational error) and `130` (Ctrl-C) signals that
a `lintle ... || true` pipe would swallow.

## Cancelling and resuming

A long `clean` over the full corpus can be interrupted — Ctrl-C, a closed
laptop, or a `SIGTERM`/`SIGHUP` from a scheduler. When that happens lintle
prints the exit code and a one-liner showing how to continue or start over. Just
re-run **the same command** to resume:

```bash
uv run lintle clean data/source --out-dir data/output   # interrupted with Ctrl-C
uv run lintle clean data/source --out-dir data/output   # picks up where it left off
```

On an interactive terminal lintle prompts `Resume interrupted run? [Y/n]`; in
CI or any non-TTY context it auto-resumes and prints a loud notice. Resume is
scoped to the same `--out-dir` — re-running against a different output directory
always starts fresh.

While a run is in flight it maintains a small `.clean-state.json` checkpoint in
`--out-dir`, deleted on successful completion — so its presence marks an
interrupted run, and a finished run leaves none behind. A stale checkpoint (the
`lintle` version or an input file changed since the interruption) is never
silently resumed: interactive → prompt (default No); non-interactive → exit `2`
with the reason and a `--no-resume` hint.

Use `--no-resume` to discard any checkpoint and start fresh — this also clears
prior `cleaned/`/`broken/` outputs so no orphaned files linger. Use `--resume`
to resume without prompting. The two flags are mutually exclusive. This is
single-run resume, not a cross-run cache: each run still re-validates every
record it emits.

## Output

A `clean` run lays `--out-dir` out like this:

```
<out-dir>/
├── cleaned/                tleYYYY.cleaned.txt   — one per input file
├── broken/                 tleYYYY.broken.txt    — one per input file
├── broken-noradids.ndjson  — corpus-wide list of quarantined NORAD IDs
└── report.md               — corpus-wide run report
```

- **`cleaned/tleYYYY.cleaned.txt`** — standard 2-line TLE text, every record
  verified valid: 69 ASCII columns per line, `\n`-terminated, matching
  satellite catalog numbers, valid checksums. World-readable, ready for
  downstream ingestion.

- **`broken/tleYYYY.broken.txt`** — the quarantine sidecar. Each entry records
  the source line number(s), a human-readable reason, and the offending line(s)
  copied **byte-faithfully**. The header carries totals, a timestamp, and the
  tool version — formatted to paste into a space-track defect report.

- **`broken-noradids.ndjson`** — newline-delimited JSON, one
  `{"noradId":N}` object per line, listing every NORAD catalog number whose
  records were quarantined anywhere in the run, deduplicated and sorted
  ascending. Records whose line 1 is itself unreadable are omitted —
  there's no catalog number to recover. Intended for programmatic
  downstream consumers (e.g. a satellite catalog flagging archive gaps)
  that want the affected IDs without parsing `broken/*.txt`. The schema
  is deliberately minimal; future releases may extend each record with
  additional fields, which consumers can ignore safely. Empty file when
  nothing was quarantined.

- **`report.md`** — a Markdown run report aggregating the whole run: corpus
  totals, the percentage cleaned and quarantined, corpus-wide fix counts, the
  defect-rule breakdown (keyed by stable `RuleID` tokens like `TLE-CHK-001`),
  a per-file table, a "Rule reference" section auto-generated from the
  `diagnostics.RULES` registry naming every rule that fired, and a per-NORAD
  breakdown table listing each satellite whose records were quarantined with
  its per-rule counts and the source files it appeared in (sorted by
  quarantined-record count descending, capped at the top 100 with a remainder
  footer pointing at `broken-noradids.ndjson` for the long tail).

A run summary is also printed per file to stdout (and as JSON with
`--report json`):

```
tle2022.txt   8,412,066 records   8,412,064 clean   3 quarantined   (1 orphan, 16,824,134 lines)
  fixes:   trailing-backslash 8,412,064 | reconstructed-checksum 195,293
  quarantined: TLE-CHK-001 1 | TLE-PAIR-001 1 | TLE-COL-001 1
```

The header line separates three input tallies that an earlier version
conflated into a single "records" number (issue #5): `records` is **paired
records** (proper 2-line TLE entries — orphans are not counted here); `clean`
is paired records that passed validation and were written to `cleaned/`;
`quarantined` is everything routed to `broken/`, which includes both failed
paired records and every orphan (an unpaired line is quarantined as
`TLE-PAIR-001`); the parenthetical `orphan` counts unpaired single lines and
`lines` counts every physical line read (including blanks dropped by the
pairing loop). The invariant is `records + orphan == clean + quarantined`,
so `clean + quarantined` can exceed `records` by the orphan count.

Quarantine counts key by the stable `RuleID` registry (e.g. `TLE-CHK-001` for
checksum mismatch, `TLE-PAIR-001` for orphan lines, `TLE-COL-001` for wrong
length) — the same handles cited in `report.md` and the `.broken.txt`
sidecar so a defect surfaces under one identifier across every artifact a
run emits. `reconstructed-checksum` is reported separately from
content-preserving fixes: those records are format-conformant, but their
checksums are *computed*, not independently verified.

`validate` writes nothing — it only prints the per-file summary and the
locations of defective records to stdout.

### Progress

A 30 GB run is not silent. Live progress is written to **stderr** as it goes —
so it never pollutes the stdout summary or a `--report json` pipe:

```
processing 29 file(s) with 10 worker(s)...
  tle2004_7of8.txt: 5,000,000 records...
[3/29] tle2004_3of8.txt — 2,527,820 clean, 183 quarantined
```

A worker emits a record-count line every 1,000,000 records; the main process
prints an `[k/N]` line as each file finishes.

## Disk-space requirements

Every record is **routed** to exactly one of `cleaned/` or `broken/` — never
duplicated — so the finished output is roughly the same total size as the
input, plus negligible metadata (`report.md`, `report.jsonl`,
`broken-noradids.ndjson`, and the transient `.shards/` findings, all tiny
next to the records).

Outputs are written to temporary `.partial` files and atomically renamed on
completion. As a conservative guard, lintle checks free space up front and
**requires roughly twice the total input size** on the `--out-dir` volume
before it starts, aborting with exit `2` if short:

```
error: insufficient disk space in data/output: need ~63,000,000,000 bytes, have 40,000,000,000
```

When free space sits in the borderline band — at or above the 2× floor but
below 2.5× the input size — the run proceeds, but lintle prints a warning
to stderr so you know you are cutting it close:

```
warning: free space in data/output is close to the 2× safety guard: 70,000,000,000 bytes free of ~63,000,000,000 recommended; the run will proceed but may exhaust the disk
```

Rule of thumb: for the bundled ~30 GB corpus, keep **~60 GB free** to clear
the abort floor and **~75 GB free** to clear the warning band. (The 12 GB
`TLEs.zip` is not an input and is never read.)

## Results on the bundled corpus

A full run over the 29-file corpus (`tle2004`–`tle2025`, ~232 million records):

- **99.96 % cleaned** — 187.9 M trailing-`\` artifacts stripped, 71.3 M missing
  checksums reconstructed
- **0.044 % quarantined** (103,228 records) as genuinely corrupt — every
  quarantined record fell into an anticipated category; no unknown defect type
  surfaced

## Development

```bash
uv sync                          # install dev dependencies
uv run pytest                    # run the test suite
uv run pytest --cov=lintle       # with a coverage report
uv run ruff check                # lint
uv run ruff format               # auto-format
```

The suite includes unit tests per module, an asymmetric cross-check against the
trusted `sgp4` parser (a known-good TLE must be accepted by both), and
end-to-end integration tests (golden output, idempotence, re-validation).

Code quality is enforced with [`ruff`](https://docs.astral.sh/ruff/) (lint rule
sets `E`, `F`, `I`, `UP`, `B`, `SIM`; 88-column lines) and coverage is measured
with `pytest-cov`.

## Project layout

```
src/lintle/
  tle.py          # core: defines a "perfect" TLE record (pure, no I/O)
  diagnostics.py  # stable RuleID registry + structured Diagnostic dataclass
  categories.py   # FixClass enum + FixSpec registry — the repair taxonomy
  explain_examples.py # validator-verified examples + citations backing `explain`
  repair.py       # speculative, validated repairs
  pipeline.py     # streaming reader, prefix-driven pairing, per-file routing
  report.py       # FileStats + dataclasses, validate summaries, run report
  report_writers.py # structured-file writers: .broken.txt sidecar, report.jsonl, broken-noradids.ndjson, shard concat
  fsutil.py       # durable_replace — the one atomic+fsync commit path
  term.py         # shared stderr Console + error/warning/note/prompt helpers
  resume.py       # single-run checkpoint for `clean --resume`
  diff.py         # read-only: per-rule delta between two runs (`lintle diff`)
  explain.py      # read-only: renders rule/fix documentation (`lintle explain`)
  cli.py          # argument parsing, parallelism, exit codes
tests/            # pytest suite
ARCHITECTURE.md   # the living design reference
docs/superpowers/
  archive/        # historical design specs, plans, and corpus-run summaries
```

`diagnostics.py` and `categories.py` are pure-data leaves of the dependency
graph — they hold enums and frozen dataclasses, no logic; `explain_examples.py`
is pure data too, composing those leaves into documented examples. `repair`,
`pipeline`, and `report` depend on them; `report_writers.py` is the
structured-file writers leaf (the `.broken.txt` sidecar, the `report.jsonl`
findings shards, the corpus `broken-noradids.ndjson`, and the shard concat)
used by `pipeline` and `cli`, importing the dataclasses and the shared
`_format_diagnostic` renderer from `report.py` one-way — never the reverse, so
no cycle; `fsutil.py` is a stdlib-only I/O leaf
(the durable file-commit helper) that `pipeline`, `report`, `report_writers`,
and `resume` route every output through; `term.py` is a rich-only leaf owning the single stderr
Console and the `error:`/`warning:` emitters that `cli.py` and `diff.py` share;
`diff.py` and `explain.py` are read-only consumers reached only through `cli.py`;
`tle.py` remains the single source of truth for what counts as a valid TLE record.

## Further reading

[`ARCHITECTURE.md`](ARCHITECTURE.md) is the living design reference — the validator
definition, the repair model, streaming/durability/resume, the machine-readable output-format
contracts (`--report json`, `report.jsonl`, the `.broken.txt` sidecar, the checkpoint), and the
runtime-dependency policy. The dated design specs, implementation plans, and corpus-run
summaries are kept for historical rationale under
[`docs/superpowers/archive/`](docs/superpowers/archive/).
