Metadata-Version: 2.4
Name: jsonxray
Version: 0.1.0
Summary: See what is actually inside a JSON Lines file, including the parts that don't match the rest.
Project-URL: Homepage, https://github.com/CAOShurong/jsonxray
Project-URL: Repository, https://github.com/CAOShurong/jsonxray
Project-URL: Issues, https://github.com/CAOShurong/jsonxray/issues
Project-URL: Changelog, https://github.com/CAOShurong/jsonxray/blob/main/CHANGELOG.md
Author-email: Shurong Cao <170531907+CAOShurong@users.noreply.github.com>
License: MIT License
        
        Copyright (c) 2026 Shurong Cao
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
License-File: LICENSE
Keywords: cli,data-quality,json,json-lines,jsonl,ndjson,profiling,schema,schema-inference,streaming
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Topic :: Utilities
Classifier: Typing :: Typed
Requires-Python: >=3.9
Provides-Extra: docs
Requires-Dist: pillow>=10; extra == 'docs'
Description-Content-Type: text/markdown

# jsonxray

**See what is actually inside a JSON Lines file — including the parts that
don't match the rest.**

[![CI](https://github.com/CAOShurong/jsonxray/actions/workflows/ci.yml/badge.svg)](https://github.com/CAOShurong/jsonxray/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/pypi/v/jsonxray.svg)](https://pypi.org/project/jsonxray/)
[![Python](https://img.shields.io/pypi/pyversions/jsonxray.svg)](https://pypi.org/project/jsonxray/)
[![License](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)

Someone hands you a 4 GB `.jsonl`. You run `head -1 | jq`, write a loader
against what you see, and forty minutes into the job it dies on line 91,204,
where one field arrived as a string instead of a number.

`jsonxray` reads the whole file once, in constant memory, and tells you that
up front — with the line numbers.

![fields](https://raw.githubusercontent.com/CAOShurong/jsonxray/main/docs/fields.png)

```bash
pip install jsonxray
jsonxray data.jsonl
```

No dependencies. Python 3.9+.

## What this tells you that a schema doesn't

A JSON Schema says what shapes are legal. That is a different question from
what your file actually contains, and three gaps between them account for most
of the time people lose:

**Absent is not null.** `{"discount": null}` and a record with no `discount`
key are different, and code that handles one usually mishandles the other. A
schema renders both as optional. `jsonxray` counts them separately: the bar is
how often the key was there at all, and `58 null` is how many of those times
it was explicitly null.

**A percentage needs a denominator.** A field inside an optional object is not
missing from 60% of your records — it is missing from 60% of the records that
had the parent object. A field inside an array is present in some fraction of
*elements*. Quoting either against the record count invents a data quality
problem that isn't there. Every row in the tree is a fraction of its own
parent.

**Averages hide the file's real problem.** The record you need to see is the
one that doesn't look like the others, and it is by definition rare enough to
be invisible in any summary statistic. So the report ranks record shapes by
how far they are from the norm, and hands you line numbers.

```text
Type conflicts
  price
      integer    99.2%  line 1, 2, 3
      string      0.8%  line 58, 200, 333

Record shapes
  32.2% of records share one shape (15 paths)
  least typical records, by distance from that shape:
    • 2x  line 13, 242
        missing items, items[], items[].qty, items[].sku, +3 more
        extra   discount
    • 17x  line 29, 49, 57
        missing shipping, shipping.country, shipping.express
        extra   discount, items[].gift
    • 19x  line 5, 45, 101
        missing shipping, shipping.country, shipping.express
        extra   items[].gift
    • 25x  line 1, 25, 37
        missing shipping, shipping.country, shipping.express
        extra   discount
    • 37x  line 9, 17, 21
        missing shipping, shipping.country, shipping.express
```

Two records out of four hundred are missing `items` entirely. They are ranked
first, ahead of the seventeen-record group, because being *structurally*
unusual matters more than being uncommon.

## Enums you didn't know you had

Fields whose values form a small closed set are worth knowing about — they are
the ones that become a database enum, a validation rule, or a bug when a new
value turns up.

```text
Small value sets
  shipping.express
      false                               71.7%  215
      true                                28.3%  85
  user.verified
      true                                84.8%  339
      false                               15.2%  61
  user.plan
      free                                50.5%  202
      pro                                 31.8%  127
      enterprise                          17.8%  71
  items[].qty
      2                                   26.8%  214
      3                                   25.2%  202
      4                                   24.6%  197
      1                                   23.4%  187
  items[].sku
      SKU-909                             22.2%  178
      SKU-100                             21.0%  168
      SKU-347                             20.5%  164
      SKU-512                             18.2%  146
      SKU-220                             18.0%  144
  shipping.country
      DE                                  23.0%  69
      JP                                  21.0%  63
      US                                  20.0%  60
      BR                                  18.7%  56
      GB                                  17.3%  52
  created_at
      2026-07-01T09:00:00Z                 3.8%  15
      2026-07-02T09:00:00Z                 3.8%  15
      2026-07-03T09:00:00Z                 3.8%  15
      2026-07-04T09:00:00Z                 3.8%  15
      2026-07-05T09:00:00Z                 3.8%  15
      2026-07-06T09:00:00Z                 3.8%  15
      +22 more values
  discount
      0.3                                  7.9%  9
      0.39                                 7.9%  9
      0.24                                 6.1%  7
      0.19                                 5.3%  6
      0.29                                 4.4%  5
      0.32                                 4.4%  5
      +30 more values
```

Once a field exceeds fifty distinct values, the table is **discarded rather
than truncated**. Past that point the tally is no longer a truthful top-N — a
value that is common but first appears late was never counted — and a
plausible wrong answer is worse than no answer.

## Catching drift in CI

Running this once tells you what is in a file. Running it in a pipeline tells
you when that stopped being true.

```bash
# Once, when you are happy with the data
jsonxray exports/monday.jsonl --save baseline.json

# Every night after that
jsonxray exports/today.jsonl --compare baseline.json
```

![drift](https://raw.githubusercontent.com/CAOShurong/jsonxray/main/docs/drift.png)

Exit code `2` means a breaking change: a field disappeared, a new type
appeared, something that was always present became optional, or something that
was never null now is. Exit `1` is reserved for the tool failing, so a
pipeline can tell "the data changed" from "the check is broken" and page
someone for only one of them.

Additive changes — a new field, a new enum value, a type that stopped
appearing — are printed as notes and exit `0`. A check that fires on ordinary
variation gets switched off within a week, at which point it catches nothing.

## Working on files that don't fit in memory

One pass, and every statistic is either O(1) per record or explicitly bounded.
Nothing is retained but the record in hand. Profiling a 77 MB, 300,000-record
file peaks at **0.2 MB** of Python heap (measured with `tracemalloc`); CI
enforces the property directly by profiling a 110 MB file under a hard
`ulimit -v`, which a reader that accumulated records could not survive.

That includes the case that usually defeats this: a **single top-level JSON
array**, pretty-printed across a million lines, which is what most "export to
JSON" buttons produce. `json.load` on one of those is exactly the
out-of-memory failure people hit. `jsonxray` detects it and decodes it
incrementally.

```bash
jsonxray dump.json          # detected automatically
jsonxray dump.json --format array
```

Where a bound bites, the report says so rather than quietly becoming
approximate — a truncated `--limit` run states that its percentages describe
only the records it read.

## Options

| Flag | What it does |
|---|---|
| `--limit N` | Stop after N records; the report says it was truncated |
| `--format jsonl\|array\|auto` | Input shape (default: detected) |
| `--only SECTION` | Print one section; repeatable |
| `--depth N` | How deep to print the field tree |
| `--json` | The whole profile as JSON |
| `--save FILE` / `--compare FILE` | Drift detection |
| `--ascii` | No block characters, for issue trackers that mangle them |
| `--color` | `auto`, `truecolor`, `256`, `16`, `none`. `NO_COLOR` always wins |

Reads stdin when given `-`, or when nothing is piped in:

```bash
zcat events.jsonl.gz | jsonxray -
```

## As a library

```python
from jsonxray import Profile, scan

profile = Profile(source="events.jsonl")
with open("events.jsonl", encoding="utf-8") as handle:
    scan(handle, profile)

for node in profile.conflicts():
    print(node.path, node.non_null_types, node.examples)
```

## Development

```bash
git clone https://github.com/CAOShurong/jsonxray
cd jsonxray
python -m unittest discover -s tests
```

The example file is generated from a fixed seed, so the numbers in this README
are reproducible:

```bash
python docs/make_example.py
python docs/build_docs.py          # regenerate the README and its images
python docs/build_docs.py --check  # what CI runs
```

CI runs the suite on Ubuntu, Windows, and macOS across Python 3.9–3.13, and
fails if this README no longer matches what the tool prints.

## License

MIT. See [LICENSE](LICENSE).
