Metadata-Version: 2.4
Name: jsonl-diff
Version: 0.1.2
Summary: A lightweight, pure-Python tool for comparing large JSONL/NDJSON datasets without loading them entirely into memory.
Author-email: rmoralespp <rmoralespp@gmail.com>
License: MIT License
        
        Copyright (c) 2026 Rolando Morales Perez
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        
Project-URL: Source, https://github.com/rmoralespp/jsonl-diff
Project-URL: Homepage, https://github.com/rmoralespp/jsonl-diff
Project-URL: Changelog, https://github.com/rmoralespp/jsonl-diff/blob/main/CHANGELOG.md
Project-URL: Issues, https://github.com/rmoralespp/jsonl-diff/issues
Project-URL: Documentation, https://rmoralespp.github.io/jsonl-diff/
Keywords: jsonlines,ndjson,jsonl,json,gzip,bzip2,xz,zst,utilities,serialization,deserialization,diff,json-diff,cli
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Information Technology
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Internet
Classifier: Topic :: Utilities
Classifier: Topic :: File Formats :: JSON
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: py-jsonl==1.5.2
Requires-Dist: jmespath>=1.0.0
Dynamic: license-file

# jsonl-diff

<p>
  <img src="https://img.shields.io/pypi/v/jsonl-diff.svg" alt="PyPI">
  <img src="https://img.shields.io/pypi/pyversions/jsonl-diff.svg" alt="Python">
  <img src="https://github.com/rmoralespp/jsonl-diff/workflows/CI/badge.svg" alt="CI">
  <img src="https://codecov.io/gh/rmoralespp/jsonl-diff/branch/main/graph/badge.svg" alt="Coverage">
  <img src="https://img.shields.io/github/license/rmoralespp/jsonl-diff.svg" alt="License">
</p>

Compare large JSONL/NDJSON datasets **by record identity instead of line
position**, without loading the complete inputs into memory.

`jsonl-diff` matches records using one or more top-level fields and reports:
`equal`, `added`, `deleted`, `modified`, and tolerated duplicates in each
source.

It is useful for snapshots, ETL validation, migrations, exports, and CI checks.

> For line-level, field-level, JSON Patch, or visual diffs, use a tool designed for those purposes instead.

## Installation

```bash
pip install jsonl-diff
```

## Quick start

Given:

```jsonl
{"id":1,"name":"Ada"}
{"id":2,"name":"Grace"}
```

and:

```jsonl
{"id":2,"name":"Grace Hopper"}
{"name":"Ada","id":1}
{"id":3,"name":"Linus"}
```

compare them by `id`:

```bash
jsonl-diff old.jsonl new.jsonl --key id
```

```text
Records:
  equal:     1
  added:     1
  deleted:   0
  modified:  1
  OLD duplicates:  0
  NEW duplicates:  0
```

Record order does not matter.

## Features

* Single or composite top-level identities.
* Disk-backed comparison for large datasets ([architecture](https://github.com/rmoralespp/jsonl-diff/blob/main/docs/architecture.md)).
* Optional **JMESPath** filtering with `--where`.
* Configurable duplicate handling with counts and diagnostics: `error`, `first`, or `last`.
* Exact **RFC 6901** JSON Pointer ignores.
* Optional disk-backed observed-schema diff for fields, types, nullability, and requiredness.
* Semantic number comparison using `Decimal`.
* Deterministic summaries and change iteration.
* Original OLD/NEW physical line numbers.
* Machine-readable JSONL change log with `--details` ([format](https://github.com/rmoralespp/jsonl-diff/blob/main/docs/details-format.md)).
* Local, HTTP/HTTPS, file-like, and supported compressed sources.
* CLI and Python API using the same comparison engine.

## CLI

```text
jsonl-diff [-h] --key KEY [--ignore IGNORE] [--where EXPRESSION]
           [--duplicates {error,first,last}] [--details FILE] [--quiet]
           [--schema-diff] [--schema-ignore POINTER] [--max-temp MAX_TEMP]
           old new
```

| Option                | Description                                                                    |
| --------------------- | ------------------------------------------------------------------------------ |
| `old`, `new`          | Local path, HTTP/HTTPS source, or `-` for stdin                                |
| `--key KEY`           | Required top-level identity field; repeat or comma-separate for composite keys |
| `--ignore POINTER`    | RFC 6901 pointer to exclude from content comparison                            |
| `--where EXPRESSION`  | JMESPath filter applied to each record                                         |
| `--duplicates POLICY` | `error` (default), or select and report duplicates with `first`/`last`          |
| `--details FILE`      | Write deterministic machine-readable JSONL changes                             |
| `--schema-diff`       | Compare observed fields, types, nullability, and requiredness                   |
| `--schema-ignore`     | RFC 6901 pointer to exclude from observed-schema profiling                      |
| `--quiet`             | Suppress the normal summary                                                    |
| `--max-temp BYTES`    | Best-effort budget for `jsonl-diff` workspace temporary storage                |

Examples:

```bash
# Composite identity
jsonl-diff old.jsonl new.jsonl --key country,customerId,type

# Ignore volatile fields
jsonl-diff old.jsonl new.jsonl \
  --key id \
  --ignore /updated_at \
  --ignore /metadata/request_id

# Filter records
jsonl-diff old.jsonl new.jsonl \
  --key id \
  --where 'deleted_at == `null`'

# Detect observed schema drift
jsonl-diff old.jsonl new.jsonl \
  --key id \
  --schema-diff \
  --schema-ignore /metadata

# Compressed input
jsonl-diff old.jsonl.xz new.jsonl.gz --key id
```

### Exit codes

| Code | Meaning                                                    |
| ---: | ---------------------------------------------------------- |
|  `0` | No record, duplicate, or requested observed-schema issues   |
|  `1` | Record differences, tolerated duplicates, or schema changes |
|  `2` | Input, resource, output, or runtime error                  |
|  `3` | Invalid CLI configuration or usage                         |

## Python API

```python
from jsonl_diff import ChangeOperation, diff

with diff(
    "old.jsonl.gz",
    "new.jsonl.gz",
    key=("country", "customerId"),
    ignore=("/updated_at",),
    where='country == `"ES"`',
    schema_diff=True,
) as result:
    print(result.summary)

    for change in result.changes(ChangeOperation.MODIFIED):
        print(change.key, change.old_line, change.new_line)

    for change in result.schema_changes():
        print(change.operation, change.path)
```

`diff()` returns a disk-backed `DiffResult`, used as a context manager.
Results are streamed lazily through `changes()` rather than materialized in
memory.

The main result models are:

```python
Summary(equal=10, added=2, deleted=1, modified=3)
```

and immutable `Change` objects containing the operation, typed identity, and
OLD/NEW source locations.

The public exception hierarchy is rooted at `JsonlDiffError`:

* `ConfigurationError`
* `InputError`
* `DuplicateKeyError`
* `ResourceError`

See the [Python API reference](https://github.com/rmoralespp/jsonl-diff/blob/main/docs/python-api.md) for the full callable
signature, result models, `Decimal` key semantics, and error hierarchy.

## Comparison semantics

`jsonl-diff` is intentionally strict:

* Every input line must contain exactly one valid top-level JSON object.
* Blank lines, malformed JSON, duplicate properties, `NaN`, and infinities are rejected.
* Identity fields must be top-level scalar values (`null` allowed only as one component of a composite key).
* Identity types remain significant: `"1"` ≠ `1`, `true` ≠ `1`.
* Duplicate identities fail by default; `--duplicates first`/`last` select one
  occurrence, report every discarded occurrence, and return exit code `1`.
* Object property order is ignored; array order is significant.
* Numbers are compared by mathematical value: `1`, `1.0`, and `1e0` are equal.
* Unicode strings are compared without normalization.
* Changes are reported in deterministic identity order.

`--where` selects which records participate; `--key` defines identity;
`--ignore` removes fields from content comparison. With `--schema-diff`,
`--schema-ignore` independently removes fields from observed-schema profiling.
See
[Comparison semantics](https://github.com/rmoralespp/jsonl-diff/blob/main/docs/comparison-semantics.md) for the full rules,
including duplicate handling, ignore-pointer edge cases, `--where` evaluation
order, and canonical number formatting.

## Sources & compression

Supported sources include local paths and HTTP/HTTPS URLs. The Python API
also accepts file-like objects.

Supported compression:

* gzip
* bzip2
* xz
* Zstandard on Python 3.14

See [architecture](https://github.com/rmoralespp/jsonl-diff/blob/main/docs/architecture.md) for the full source/compression
support matrix and how sources are delegated to `py-jsonl`.

## Limitations

`jsonl-diff` does **not** provide:

* line/position-based diffs
* field-level diffs or JSON Patch
* move/rename detection
* nested identities or automatic key detection
* fuzzy matching or numeric tolerances
* unordered-array comparison
* validation against a declared schema or input repair
* ZIP, database, or cloud-provider inputs
* GUI or HTML reports

It is a **dataset reconciliation tool**, not a general-purpose visual JSON diff.

## Further reading

* [Comparison semantics](https://github.com/rmoralespp/jsonl-diff/blob/main/docs/comparison-semantics.md) — identity, duplicates, ignores, `--where`, canonical numbers, determinism.
* [Disk-backed architecture](https://github.com/rmoralespp/jsonl-diff/blob/main/docs/architecture.md) — SQLite index, `max_temp`, cleanup, sources and compression.
* [Details JSONL format](https://github.com/rmoralespp/jsonl-diff/blob/main/docs/details-format.md) — machine-readable `--details` output schema.
* [Python API reference](https://github.com/rmoralespp/jsonl-diff/blob/main/docs/python-api.md) — full signature, result models, and error hierarchy.

## Development

```bash
uv sync --group test --group lint

uv run pytest
uv run ruff check --quiet --output-format=concise .
```

The test suite covers CLI behavior, identity/canonicalization, validation, details output, sources, compression, temporary limits, and cleanup.

## License

See [LICENSE](LICENSE).
