Metadata-Version: 2.4
Name: oxedi
Version: 0.4.0
Classifier: Development Status :: 4 - Beta
Classifier: Programming Language :: Rust
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Operating System :: POSIX :: Linux
Classifier: Operating System :: MacOS
Classifier: Operating System :: Microsoft :: Windows
Classifier: Topic :: Office/Business :: Financial
Requires-Dist: pandas>=2.0.3,<3 ; extra == 'edi-835-parser'
Requires-Dist: pyarrow>=14 ; extra == 'edi-835-parser'
Requires-Dist: pandas>=2 ; extra == 'pandas'
Requires-Dist: pyarrow>=14 ; extra == 'pandas'
Requires-Dist: polars>=1.0 ; extra == 'polars'
Requires-Dist: pyx12>=4.0,<5 ; extra == 'pyx12'
Requires-Dist: pytest>=8 ; extra == 'test'
Requires-Dist: polars>=1.0 ; extra == 'test'
Requires-Dist: pyarrow>=14 ; extra == 'test'
Requires-Dist: pandas>=2.0.3,<3 ; extra == 'test'
Requires-Dist: edi-835-parser==1.8.0 ; extra == 'test'
Requires-Dist: duckdb>=1.1 ; extra == 'test'
Provides-Extra: edi-835-parser
Provides-Extra: pandas
Provides-Extra: polars
Provides-Extra: pyx12
Provides-Extra: test
License-File: LICENSE
License-File: THIRD_PARTY_NOTICES
Summary: Lossless, fast, data-driven EDI 835 parser
License-Expression: MIT
Requires-Python: >=3.11
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
Project-URL: Changelog, https://github.com/javillegasna/oxedi/blob/master/CHANGELOG.md
Project-URL: Issues, https://github.com/javillegasna/oxedi/issues
Project-URL: Repository, https://github.com/javillegasna/oxedi

# oxedi

A fast X12 EDI parser for Python, written in Rust. Today it reads the 835 (electronic
remittance advice).

- **Lossless.** Every byte of the file is kept, so `write()` returns the file exactly as it
  was read.
- **Ready-made tables.** Payments, claims, service lines and adjustments come out as typed
  tables that open in Polars, pandas, pyarrow or DuckDB without copying the data.
- **Problems are reported, not raised.** A file that is valid EDI but has questionable data
  still parses; each issue is returned as a diagnostic that names the rule, the position and
  the value.
- **Large files.** `stream` yields one transaction at a time, so memory stays bounded.
- **Extensible.** Add segments or table columns with a small JSON patch, without touching
  the code.

## Install

```bash
pip install oxedi               # no Python dependencies
pip install "oxedi[polars]"     # with Polars
pip install "oxedi[pandas]"     # with pandas and pyarrow
```

Requires Python 3.11 or later. Wheels are available for Linux (x86_64 and aarch64, plus
musl on x86_64), macOS (Intel and Apple silicon) and Windows (x86_64).

## Quick start

```python
import oxedi

result = oxedi.parse_file("remittance.835")

claims = result.tables["claims"].to_polars()      # or .to_pandas()
services = result.tables["services"].to_polars()

print(result.count_claims(), result.sum_payments())  # sum_payments() is a Decimal
print(result.payer["name"], result.payee["name"])

for diagnostic in result.diagnostics:
    print(diagnostic)
```

`oxedi.parse(data)` does the same from `bytes`. A file that is not an 835 at all (for
example, one that does not start with an `ISA` segment) raises `oxedi.ParseError`.

### Tables

| Table | One row per | Linked by |
|---|---|---|
| `payments` | payment (transaction) | — |
| `claims` | claim | `payment` |
| `services` | service line | `payment`, `claim` |
| `adjustments` | claim or service adjustment | `payment`, `claim`, `service` |
| `provider_adjustments` | provider-level adjustment | `payment` |

`result.tables.keys()` lists them and `table.columns` lists a table's columns. Amounts are
decimals, dates are dates, and text fields are kept as the raw bytes from the file (Arrow
`binary`), so nothing is lost to an encoding guess. Tables follow
the Arrow PyCapsule interface, so any Arrow-aware library reads them directly:

```python
import duckdb, polars as pl

claims = result.tables["claims"]
duckdb.sql("select claim_status, sum(payment_amount) from claims group by 1")
pl.DataFrame(result.tables["services"])
```

### Large files

```python
from pathlib import Path

for batch in oxedi.stream(Path("big.835").read_bytes()):
    services = batch.tables["services"].to_polars()
```

Each batch holds the tables and diagnostics of one transaction. Pass `by="2100"` (or any
other loop id) to get one batch per claim instead. Parsing releases the GIL, so several
files can be parsed in parallel from threads.

### Writing the file back

```python
data = Path("remittance.835").read_bytes()
assert oxedi.parse(data).document.write() == data
```

## Writing

`oxedi.write` turns tables back into an 835: the tables of a parse, or your own Arrow, Polars or
pandas tables with the spec's columns. Parse, change a value, write:

```python
import datetime
import polars as pl
import oxedi

result = oxedi.parse_file("remittance.835")
frames = {name: result.tables[name].to_polars() for name in result.tables.keys()}
frames["payments"] = frames["payments"].with_columns(trace_number=pl.lit("CHK100235"))

envelope = oxedi.Envelope(
    sender_id="ACMEPAYER",
    receiver_id="SUNRISECLINIC",
    date=datetime.date(2024, 1, 10),
    time=datetime.time(9, 0),
    usage_indicator="T",  # "P" (production) is the default
)
data = oxedi.write(frames, envelope)  # bytes, one interchange
```

The spec is the one that parsed the tables, else the built-in 5010 spec. A dictionary of frames,
as above, does not carry the parse's spec, so pass the one that matches the file's version: for
a 4010 file, `oxedi.write(frames, envelope, spec=oxedi.Spec.builtin("4010"))`; otherwise it is
written as 5010 and refused. A table left out has no rows and a column left out is null. Rows nest by their `payment`, `claim` and `service` columns, the row
number of their parent. Only what the tables hold is written: segments no column reads (for
example the payer's `PER*CX` or the bank details of `BPR`) are left out. To reproduce a parsed
file byte for byte, use `result.document.write()` instead.

**Envelope.** `oxedi.Envelope` gives what the tables cannot: `sender_id` and `receiver_id` with
their qualifiers (`sender_qualifier`, `receiver_qualifier`, default `"ZZ"`), `date` and `time`,
`usage_indicator`, the first `control_number` (default 1), optional `application_sender` and
`application_receiver` for the group header, `delimiters` (default `*`, `:`, `~` and repetition
`^`) and `line_break` (a line break after each segment). The writer works out the rest: the
fixed-width `ISA`, the control numbers that must match (`ISA13`/`IEA02`, `GS06`/`GE02`,
`ST02`/`SE02`) and the counts (`SE01`, `GE01`, `IEA01`).

**Strict by default.** The written file is read back with the spec, and every diagnostic of that
read is a finding, as is anything the writer cannot place: for example a required element or
occurrence without a value, a value outside its code list or length, a row whose parent does not
exist or that comes out of order, a value holding a delimiter, or money that does not balance.
A value holding a delimiter is reported first: the file then splits where the data does not, so
it is not read back, and the other findings show once that value is fixed. Any finding raises
`oxedi.WriteError`, a `ValueError` whose message lists every finding and whose `findings` holds
them as `oxedi.WriteFinding` (each names the table, row and column, or the envelope field, with
the diagnostic behind it); nothing is written. With `allow_findings=True` the call returns
`(data, findings)` instead, for example to produce invalid files for tests:

```python
try:
    data = oxedi.write(frames, envelope)
except oxedi.WriteError as error:
    for finding in error.findings:
        print(finding.table, finding.row, finding.column, finding)

data, findings = oxedi.write(frames, envelope, allow_findings=True)
```

The writer never changes money: the balancing rules (`BPR02` against the claims' payments and
the `PLB` adjustments, each claim's and each service's charge minus payment against their `CAS`
adjustments) are checked, never used to fill a total. The claim rule does not model interest
(`AMT*I`): a claim whose payment includes interest does not balance and is refused unless
`allow_findings` is set.

**Null and empty text.** In text columns, `null` is an element the segment does not have and `""`
one it has but leaves empty. A null cell in the middle of a segment is still written as an empty
element, so it reads back as `""`; only trailing nulls read back as `null`. The tables of a
parse write back to the same tables.

## Extending the spec

The structure of the 835 and the columns of each table are defined by a JSON spec. A patch
in JSON Merge Patch format (RFC 7386) adapts it to a payer's variations:

```python
spec = oxedi.Spec.builtin().patch({
    "tables": {"claims": {"columns": {
        "contract_class": {"segment": "REF", "where": {"1": "CE"}, "element": 2}
    }}}
})
result = oxedi.parse(data, spec=spec)   # claims now has a contract_class column
```

Objects merge key by key, while arrays are replaced whole. A loop's segments are named
occurrences, so a patch adds one (or changes or removes it with `null`) by its name:

```json
{"loops": {"1000A": {"occurrences": {"xx": {"segment": "XX", "pos": 11400}}}}}
```

`pos` orders the occurrences of a loop. The transaction and every loop below it share one
position space: the built-in spec numbers a segment of the transaction's n-th table at
n × 10000 plus its implementation-guide position (1000A's N1 is 10800, 2100's CLP 20100), and
a child loop's occurrences sit at their own positions inside that space. The occurrence a
loop opens on comes first: every other occurrence of the loop has a higher `pos`.

A column reads an element of a segment chosen by `segment` and optional `where`
conditions, or of a named occurrence (`{"occurrence": "patient_name", "element": 3}`). `loop`
reads a loop inside the table's anchor or above it: in the services table,
`{"loop": "2100", "occurrence": "claim_payment_information", "element": 1}` gives each service
its claim's id. When an occurrence repeats, `pick` chooses `"first"` (the default), `"last"` or
the n-th match, counting from 1, among the segments read while the row's loop instance is
open; a loop above the anchor offers the segments its enclosing instance read before the row's
instance opened.

`parse` and `stream` both accept `spec=`.

## Spec versions

`parse`, `parse_file` and `stream` read the version a file declares (ISA12 and GS08) and use
the matching built-in spec: 5010 (`005010X221A1`) by default, 4010 (`004010X091A1`) for 4010
files. Pass `spec=` to override, or pick one yourself:

```python
spec = oxedi.Spec.builtin(version="4010")
```

## Validating with pyx12

`pip install "oxedi[pyx12]"` adds [pyx12](https://github.com/azoner/pyx12)'s
implementation-guide validation. `parse` never calls it; `validate` does, on demand:

```python
import oxedi
from oxedi.pyx12 import validate

result = oxedi.parse(data)
findings = result.diagnostics + validate(data)    # bytes, a path or a binary file object
for d in sorted(findings, key=lambda d: d.level):
    print(d.origin, d.code, d)                    # "oxedi" or "pyx12"; pyx12's own code
    if d.segment is not None:
        start, end = result.document[d.segment].span  # the segment's bytes in your file
```

`validate` returns `oxedi.Diagnostic`, the type `parse` returns, so the two lists mix, sort
by `level` and filter by `origin`. A pyx12 finding has `kind == "External"`,
`origin == "pyx12"`, `code` set to pyx12's error code and an empty `path` (pyx12 does not report
the loop); interchange, group and transaction findings are level 1, segment and element
findings level 2. A file pyx12 cannot read, or an exception inside pyx12, comes back as one
level 1 finding with no `code` whose `rule` starts with `could not finish validating`,
instead of a traceback.

## Coming from another library

Coming from edi-835-parser? See the [migration guide](https://github.com/javillegasna/oxedi/blob/master/docs/migrating-from-edi-835-parser.md).

## Versioning

The project follows semantic versioning. While the version is `0.x`, a minor release may change the API. See the [changelog](https://github.com/javillegasna/oxedi/blob/master/CHANGELOG.md).

## License

MIT. See [`LICENSE`](https://github.com/javillegasna/oxedi/blob/master/LICENSE) and [`THIRD_PARTY_NOTICES`](https://github.com/javillegasna/oxedi/blob/master/THIRD_PARTY_NOTICES).

