Metadata-Version: 2.4
Name: table2text
Version: 0.1.0
Summary: Token-budgeted structural briefs of messy CSVs, so LLM agents can write correct extractors without opening the file
License: MIT
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: exact-tokens
Requires-Dist: tiktoken; extra == "exact-tokens"
Provides-Extra: dev
Requires-Dist: pytest>=8; extra == "dev"
Requires-Dist: tiktoken; extra == "dev"
Dynamic: description
Dynamic: description-content-type
Dynamic: license
Dynamic: license-file
Dynamic: provides-extra
Dynamic: requires-python
Dynamic: summary

# table2text

Repo maps for tables. `table2text` reads one messy CSV and prints a compact,
token-budgeted **brief** — the structure, the schema, the traps, and the exact
pandas code to load it — so an LLM agent can write a correct extraction script
**without ever opening the file**.

```
$ table2text Freighting_goods.csv
FILE: Freighting_goods.csv — 221 rows x 184 cols, utf-8, delimiter ','
VERDICT: report sheet, not clean — 9 stacked tables inside
MAP: 0-2 banner | 4-5 metadata | 7-20 prose | T1 23-36 | T2 39-64 | ... | 207-219 footnotes
TABLES:
T1 23-36 hdr 23-24 d2/7g 12r x 31c: [0-2] 3 str; [3-30] 28 float
...
QUIRKS:
  9 stacked tables — read each by its own row offsets
  T1-T4, T8: multi-row header (row span on each TABLES line): flatten the column index after load
  T1-T3, T6: forward-fill after load (labels written once per group): [0] Activity, [1] Type
  T1-T9: cols beyond each table's own column count are empty padding (file is 184 wide): trim after load
RECIPE:
```python
import pandas as pd
p = 'Freighting_goods.csv'
specs = [(23,12,2,31), (39,24,2,19), ...]
dfs = [pd.read_csv(p, skiprows=s, nrows=n,
       header=list(range(h)) if h else None).iloc[:, :c]
       for s, n, h, c in specs]
```

That file is a real UK-government spreadsheet: banner rows, a metadata block,
14 rows of prose guidance, nine stacked sub-tables with two-row spanning
headers, forward-filled label columns, and 150 columns of empty padding.
Every conventional schema tool (DuckDB `sniff_csv`, csvkit, qsv, frictionless)
assumes header-on-row-1 and one clean rectangle, and reads it wrong. Agents
figure it out by scrolling and re-reading — an expensive detective loop,
repeated per file, per session. `table2text` runs that loop once,
deterministically, and hands over the conclusions.

## Install

```bash
pip install .            # from a checkout
pip install .[exact-tokens]   # + tiktoken for exact token counting
```

Python ≥ 3.11, stdlib only. `tiktoken` is optional: without it the budget is
enforced against a conservative chars/4 estimate.

## Usage

```bash
table2text data.csv                 # the brief, ≤2000 tokens, to stdout
table2text data.csv --budget 500    # tighter: coarser view of the same facts
table2text data.csv --json          # full table-AST as JSON (for programs)
```

One command, two flags, zero config. Clean CSVs produce naturally tiny briefs
(~150 tokens) at any budget — the budget is a ceiling, never a target.

## Why the default budget is 2000 tokens — do not "helpfully" raise it

The brief must be a **single-gulp artifact**: small enough that every agent
harness ingests it in one untruncated tool-call read, so no agent ever greps
*within* it. The binding constraint is the stingiest mainstream harness —
Codex CLI truncates tool output at 256 lines or ~10 KiB (~2.5k tokens),
head+tail, which silently deletes the *middle* of anything larger (and the
middle of this brief is the schema). 2000 tokens with a 240-line guard is the
largest brief that survives every harness intact. Context-rot research points
the same way: model accuracy degrades with input length from the tens of
thousands of tokens, so a brief should be a negligible fraction of an agent's
remaining healthy context.

If the budget is smaller than a file's structural floor (the irreducible cost
of naming its regions, tables and read parameters), the floor brief ships
anyway, with a warning on stderr — a complete description over budget beats a
within-budget one that omits the map or the recipe.

## How it works — Extract → Rank → Cut (no AI anywhere)

The architecture mirrors Aider's repo map: deterministic structure
extraction, a salience formula, and a greedy fill into a hard token budget.

1. **Extract** (`dialect.py`, `rows.py`, `regions.py`, `header.py`,
   `columns.py`, `ast.py`): sniff the dialect; classify every row
   (banner / metadata / prose / header / data / blank) with Pytheas-style
   type-coherence rules; group rows into regions, splitting stacked tables on
   repeated headers; resolve multi-row headers into a column tree with
   spanning groups and units; compute per-column stats in one streaming pass.
   Output: a lossless **table-AST** — coordinates, names, types, aggregates.
   Bulk cell data is never stored (≤3 sample rows, cells truncated).
2. **Rank** (`rank.py`): structure, boundaries, quirks and recipes are
   never cut; key/enum columns render rich; homogeneous bulk columns collapse
   into pattern lines; individual data rows are represented only by stats.
3. **Cut** (`render.py`, `budget.py`, `brief.py`): a fixed degradation
   ladder (drop samples → drop per-column stats → merge columns into runs)
   renders the richest version that fits `--budget`. Smaller budgets yield a
   coarser view of the same facts, never different facts.

Everything is deterministic heuristics — cheap, fast, reproducible, no LLM
calls in the pipeline.

## Development

```bash
uv run --with pytest,tiktoken python3 -m pytest -q
```

Test fixtures include 24 real UK-government GHG Conversion Factors 2025 CSVs
(gov.uk, Open Government Licence v3.0 — see `tests/fixtures/ghg-2025-condensed/README.md`),
exercising nearly every messy-spreadsheet idiom. The acceptance bar is
executable: extractors derived only from the brief text must reproduce
hand-verified ground-truth rows (`tests/test_acceptance.py`).

Design docs: `CLAUDE.md` (project context), `docs/M3-design-spec.md`
(renderer/budget design and its rationale).
