Metadata-Version: 2.4
Name: jsonl-diff
Version: 0.1.0
Summary: Strict, disk-backed reconciliation of large JSONL/NDJSON datasets for Python.
Author-email: rmoralespp <rmoralespp@gmail.com>
License: MIT License
        
        Copyright (c) 2026 Rolando Morales Perez
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        
Project-URL: Source, https://github.com/rmoralespp/jsonl-diff
Project-URL: Homepage, https://github.com/rmoralespp/jsonl-diff
Project-URL: Changelog, https://github.com/rmoralespp/jsonl-diff/blob/main/CHANGELOG.md
Project-URL: Issues, https://github.com/rmoralespp/jsonl-diff/issues
Project-URL: Documentation, https://rmoralespp.github.io/jsonl-diff/
Keywords: jsonlines,ndjson,jsonl,json,gzip,bzip2,xz,zst,utilities,serialization,deserialization,diff,json-diff,cli
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Information Technology
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Internet
Classifier: Topic :: Utilities
Classifier: Topic :: File Formats :: JSON
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: py-jsonl==1.5.2
Dynamic: license-file

# jsonl-diff

----

<p>
  <img src="https://img.shields.io/pypi/v/jsonl-diff.svg" alt="PyPI">
  <img src="https://img.shields.io/pypi/pyversions/jsonl-diff.svg" alt="Python">
  <img src="https://github.com/rmoralespp/jsonl-diff/workflows/CI/badge.svg" alt="CI">
  <img src="https://codecov.io/gh/rmoralespp/jsonl-diff/branch/main/graph/badge.svg" alt="Coverage">
  <img src="https://img.shields.io/github/license/rmoralespp/jsonl-diff.svg" alt="License">
</p>

`jsonl-diff` performs strict, identity-based reconciliation of JSONL/NDJSON
datasets. It matches records by one or more top-level fields, then classifies
each identity as `equal`, `added`, `deleted`, or `modified`.

## Why jsonl-diff?

A text diff compares lines. That is usually the wrong model for datasets:
reordering unchanged records creates noise, and inserting one record can make
every later line appear different.

`jsonl-diff` fills the gap between line-oriented tools and in-memory dataframe comparisons:

- match records by identity, not physical position;
- compare datasets larger than available RAM using temporary SQLite storage;
- preserve original OLD and NEW line numbers for auditability;
- expose the same behavior through a CLI and a small Python API.

Use it for snapshots, ETL validation, migrations, exports, and CI checks. Use
`diff`, `jq`, or a visual JSON diff instead when you need line-level,
field-level, or JSON Patch output.

The comparison is backed by a temporary SQLite index. Complete inputs and
complete result sets do not need to fit in memory. Process memory is bounded by
the largest record plus parser and SQLite buffers, while indexed data lives on
temporary disk.

## Features

- Single or composite top-level identities, independent of input order.
- Configurable duplicate handling with strict failure by default.
- Exact RFC 6901 object-member ignores.
- Semantic number comparison without binary floating-point rounding.
- Deterministic summaries and changed-identity iteration.
- Original OLD and NEW physical line numbers for every change.
- Local, HTTP/HTTPS, file-like, and supported compressed sources through
  `py-jsonl`.
- The same comparison engine through the CLI and Python API.
- Disk-backed comparison with configurable `jsonl-diff` temporary storage.

## Requirements and installation

`jsonl-diff` supports Python 3.8 through 3.14.

Install it in a project managed by `uv`:

```bash
uv add jsonl-diff
```

Install the CLI as a standalone tool:

```bash
uv tool install jsonl-diff
```

`pip` is also supported:

```bash
python -m pip install jsonl-diff
```

## CLI quick start

Given:

```jsonl
{"id":1,"name":"Ada"}
{"id":2,"name":"Grace"}
```

and a reordered, changed dataset:

```jsonl
{"id":2,"name":"Grace Hopper"}
{"name":"Ada","id":1}
{"id":3,"name":"Linus"}
```

compare them by `id`:

```bash
jsonl-diff old.jsonl new.jsonl --key id
```

Standard output contains only the text summary:

```text
Records:
  equal:     1
  added:     1
  deleted:   0
  modified:  1
```

The result means one record is unchanged, one was added, and one existing
record changed. Physical order does not affect these counts.

Diagnostics are written to standard error. Use `--quiet` when only the exit
status or details file is needed.

### Common pipelines

Filter JSONL with `jq` before comparing it. `-c` keeps one compact JSON object
per line:

```bash
jq -c 'select(has("deleted_at") and .deleted_at == null)' old.jsonl |
  jsonl-diff - new.jsonl --key id
```

Read compressed input directly when no preprocessing is required:

```bash
jsonl-diff old.jsonl.xz new.jsonl.gz --key id
```

For two independent pipelines on Bash-compatible systems, use process
substitution:

```bash
jsonl-diff \
  <(head -n 1000 old.jsonl | jq -c 'select(.active)') \
  <(head -n 1000 new.jsonl | jq -c 'select(.active)') \
  --key id
```

`-` represents stdin. Both inputs cannot use the same stdin stream.

### CLI syntax and options

```text
jsonl-diff [-h] --key KEY [--ignore IGNORE]
           [--duplicates {error,first,last}] [--details FILE] [--quiet]
           [--max-temp MAX_TEMP]
           old new
```

| Argument | Meaning |
|---|---|
| `old` | OLD local path, HTTP/HTTPS source, or `-` for stdin. |
| `new` | NEW local path, HTTP/HTTPS source, or `-` for stdin. |
| `--key KEY` | Required top-level identity field. Repeat it or use a comma-separated value for a composite identity. |
| `--ignore JSON_POINTER` | Exact RFC 6901 pointer to an object member to remove before content comparison. Repeatable. |
| `--duplicates POLICY` | Handle repeated identities with `error` (default), `first`, or `last`. |
| `--details FILE` | Write machine-readable JSONL details to `FILE`; details are never written to stdout. |
| `--quiet` | Suppress the normal stdout summary. Errors still go to stderr. |
| `--max-temp MAX_TEMP` | Limit storage owned by `jsonl-diff` to a positive integer number of bytes. |
| `-h`, `--help` | Show command help and exit. |

These composite-key forms are equivalent:

```bash
jsonl-diff old.jsonl new.jsonl --key country,customerId,type

jsonl-diff old.jsonl new.jsonl \
  --key country \
  --key customerId \
  --key type
```

Ignore volatile fields by exact path:

```bash
jsonl-diff old.jsonl new.jsonl \
  --key id \
  --ignore /updated_at \
  --ignore /metadata/request_id
```

### Exit codes

| Code | Meaning |
|---:|---|
| `0` | The inputs are equal under the configured rules. |
| `1` | At least one added, deleted, or modified identity was found. |
| `2` | A handled input, source, resource, output, or interruption error occurred. |
| `3` | CLI usage or comparison configuration is invalid. |

Invalid argparse usage exits directly with code `3`. The Python API uses
exceptions instead of these process exit codes.

## Details JSONL

`--details FILE` streams a deterministic JSONL report through `py-jsonl` after
both inputs have been completely indexed and validated:

```bash
jsonl-diff old.jsonl new.jsonl --key id --details changes.jsonl
```

For an OLD source containing identities `1` and `2`, and a NEW source
containing changed identity `2` and added identity `3`, the file is:

```jsonl
{"duplicates":"error","ignore":[],"key":["id"],"type":"meta"}
{"key":[1],"old_line":1,"op":"deleted","type":"change"}
{"key":[2],"new_line":1,"old_line":2,"op":"modified","type":"change"}
{"key":[3],"new_line":2,"op":"added","type":"change"}
{"added":1,"deleted":1,"equal":0,"modified":1,"type":"summary"}
```

Record types are:

- **`meta`**: always first. `key` contains normalized identity-field names;
  `ignore` contains the configured ignore pointers.
- **`change`**: one per changed identity. `op` is `added`, `deleted`, or
  `modified`. `old_line` is present for deletions and modifications;
  `new_line` is present for additions and modifications. Equal records are not
  emitted.
- **`summary`**: always last after a successful write, with all four totals.

The normal stdout summary is unchanged when `--details` is used, unless
`--quiet` suppresses it. A write failure can leave a partial details file and
returns exit code `2`.

## Python API

### Comparing sources

```python
from jsonl_diff import ChangeOperation, diff

with diff(
    "old.jsonl.gz",
    "new.jsonl.gz",
    key=("country", "customerId"),
    ignore=("/updated_at",),
    duplicates="error",
    max_temp=2_000_000_000,
) as result:
    print(result.summary)

    for change in result.changes(ChangeOperation.MODIFIED):
        print(change.key, change.old_line, change.new_line)
```

The callable signature is:

```python
from typing import Any, Optional, Sequence, Union

from jsonl_diff import DiffResult, DuplicatePolicy

def diff(
    old: Any,
    new: Any,
    *,
    key: Union[str, Sequence[str]],
    ignore: Sequence[str] = (),
    duplicates: Union[str, DuplicatePolicy] = DuplicatePolicy.ERROR,
    max_temp: Optional[int] = None,
) -> DiffResult:
    ...
```

`diff()` creates a comparison session without opening either source. Entering
its context fully reads, indexes, and validates both sources before exposing
the result. `DiffResult.changes()` is then a lazy iterator over the disk-backed
result rather than a list held in memory.

Using `DiffResult` as a context manager is required. It owns the temporary
resources and removes its private workspace on exit. A result cannot be entered
more than once. Its summary becomes available after entering and remains
available after closing; iterating changes requires the context to remain open.

### Result models

`result.summary` is an immutable `Summary`:

```python
Summary(equal=10, added=2, deleted=1, modified=3)
```

It exposes integer fields `equal`, `added`, `deleted`, and `modified`, plus the
boolean property `different`. The same values are available directly as
read-only `DiffResult` properties.

Each item from `result.changes()` is an immutable `Change` with:

- `operation`: a `ChangeOperation` enum value (`ADDED`, `DELETED`, or
  `MODIFIED`);
- `key`: a typed tuple in normalized identity-field order;
- `old` and `new`: optional `SourceLocation` values;
- `old_line` and `new_line`: convenience properties returning a one-based
  physical line or `None`.

For additions, `old`/`old_line` are `None`; for deletions,
`new`/`new_line` are `None`; modifications have both locations. Location
sources are labeled `OLD` and `NEW`.

Pass a `ChangeOperation` to filter without materializing all changes:

```python
with diff("old.jsonl", "new.jsonl", key="id") as result:
    added = result.changes(ChangeOperation.ADDED)
    for change in added:
        print(change.key, change.new_line)
```

With numeric keys, API key components are `decimal.Decimal` values. For
example, JSON identity `7` is returned as `Decimal("7")`; JSON strings and
booleans retain their types. Details JSONL writes numeric keys as JSON numbers,
including arbitrary-precision integers.

### API errors

The public error hierarchy starts with `JsonlDiffError`:

- `ConfigurationError`: invalid keys, ignore pointers, or `max_temp`;
- `InputError`: an invalid source or record; exposes `source` and optional
  `line`;
- `DuplicateKeyError`: an `InputError` with `key` and the first/repeated
  physical lines in `lines`;
- `ResourceError`: the temporary index cannot be created, written, or kept
  within its configured limit.

Failures during `diff()` clean up the workspace before the exception is
raised.

## Sources and compression

Source opening and decompression are delegated to `py-jsonl`:

| Source | CLI | Python API |
|---|:---:|:---:|
| Local path | Yes | Yes, including path-like values |
| HTTP/HTTPS URL | Yes | Yes |
| File-like object | No | Yes |
| gzip (`.gz`) | Yes | Yes |
| bzip2 (`.bz2`) | Yes | Yes |
| xz (`.xz`) | Yes | Yes |
| Zstandard (`.zst`) | Python 3.14 only | Python 3.14 only |
| ZIP archive | No | No |

Zstandard availability follows `py-jsonl` and its use of Python 3.14's
standard-library zstd support; it is not supported by this project on earlier
Python versions. Compression and HTTP transfer boundaries do not affect the
reported decoded JSONL line numbers.

## Comparison semantics

### Strict JSONL records

- Every physical line must be non-blank, valid JSON containing exactly one
  top-level object.
- Local paths and binary streams are decoded as UTF-8. Text streams are
  already decoded, while URL response charset handling follows `py-jsonl`
  (defaulting to UTF-8 when no charset is declared).
- Blank lines, malformed JSON, duplicate object property names, `NaN`,
  `Infinity`, and `-Infinity` are errors.
- Top-level arrays, strings, numbers, booleans, and `null` are rejected.
- Input errors identify `OLD` or `NEW` and include the physical line when
  available.

No normal summary or details output begins until both inputs pass complete
input and duplicate validation.

### Identity and duplicates

Identity fields are top-level object-member names. Each component must exist
and be a non-null string, number, or boolean; objects and arrays are invalid.
Nested identity paths and automatic key detection are not supported.

Composite identity field names are sorted lexically before their values are
extracted. Consequently, `key=("b", "a")` and `key=("a", "b")` both produce
keys in `(a, b)` order and match identically. Repeated or empty identity-field
names are configuration errors.

JSON types remain significant: `"1"` is different from `1`, and `true` is
different from `1`.

Every normalized identity must be unique within OLD and within NEW by default.
The first duplicate aborts the comparison; `DuplicateKeyError` reports the
first occurrence and repeated physical line. Set `duplicates="first"` to keep
the first occurrence or `duplicates="last"` to keep the last occurrence and
its physical line. These tolerant policies depend on physical input order.

### Ignored object members

Ignore expressions are exact
[RFC 6901 JSON Pointers](https://www.rfc-editor.org/rfc/rfc6901). Escapes such
as `~1` for `/` and `~0` for `~` are supported. An absent path is harmless,
and `/updated_at` does not match `/metadata/updated_at`.

Ignores are applied symmetrically before content canonicalization. A pointer
may remove an entire array-valued object member, but it may not traverse an
array or address an array element. Ignoring a top-level identity field is also
invalid. Wildcards, JSONPath, and recursive name matching are not supported.

### Canonical content and numbers

Records with the same identity are compared after ignored members are removed:

- JSON object property order is insignificant at every depth.
- Array order is significant.
- Unicode strings are compared by code-point sequence; no Unicode
  normalization is applied.
- Numbers are parsed as `Decimal` and compared by mathematical value.
  `1`, `1.0`, and `1e0` are equal, and negative zero equals zero.
- Arbitrary-precision integers and decimal values are not rounded through
  binary floating point.

Canonical content is stored with its length and SHA-256 digest. Matching
canonical bytes are still compared, so a digest match alone is not treated as
proof of equality. Numbers are written using compact scientific notation; a
large exponent does not expand into a large string of zeroes.

### Determinism

Summary counts do not depend on either source's physical record order.
Changes, including details records, use the stable lexical byte order of each
canonical typed identity. This is a reproducibility guarantee, not numeric,
locale-aware, or human-oriented sorting.

## Disk-backed architecture and temporary files

Each record is parsed and validated incrementally, normalized, and inserted
into a private SQLite database under an operating-system temporary directory.
The database stores the typed canonical identity, original line, canonical
content, content length, and digest. A uniqueness constraint detects duplicate
identities. SQL joins calculate the summary, and ordered SQLite cursors drive
lazy change iteration.

This architecture bounds memory by the records currently being processed and
database buffers; it does not keep the complete decoded inputs or all changes
in RAM. It does require temporary disk space. Canonical records and SQLite
indexes remain until the `DiffResult` is closed, so temporary usage can exceed
twice the combined decompressed input size.

`max_temp` / `--max-temp` accepts a positive byte count. The implementation
limits and checks files in the workspace owned by `jsonl-diff`, raising
`ResourceError` (CLI exit `2`) when the index cannot stay within that budget.
Choose a limit with room for SQLite pages and index overhead.

`py-jsonl` may create its own temporary staging files for remote or compressed
sources. Those files follow `py-jsonl`'s resource policy and are not counted by
`jsonl-diff`'s `max_temp` limit. Total system temporary usage can therefore
exceed the configured value.

The private workspace is removed on context-manager exit, explicit `close()`,
or a handled failure during construction. Cleanup failures are emitted as
warnings rather than replacing the primary error.

## Limitations and non-goals

`jsonl-diff` intentionally does not provide:

- line- or physical-order comparison;
- equal-record events or complete record payloads in details output;
- field-level diffs, JSON Patch, move detection, or rename detection;
- nested identities, fuzzy matching, or key autodetection;
- numeric tolerances, unordered-array comparison, or Unicode normalization;
- wildcard/JSONPath ignores or ignores that traverse arrays;
- schema validation or repair of malformed input;
- ZIP input, database-table input, cloud-provider SDK integrations, GUI, or
  HTML reports.

It is a reconciliation tool for strict JSONL datasets, not a general-purpose
visual JSON diff.

## Development

Clone the repository and sync the test and lint dependency groups with `uv`:

```bash
uv sync --group test --group lint
```

Run the test suite and Ruff:

```bash
uv run pytest
uv run ruff check --quiet --output-format=concise .
```

The tests cover CLI behavior, identity and canonicalization rules, strict
validation, details output, source types, compression, temporary limits, and
resource cleanup.

## License

See [LICENSE](LICENSE).
