Metadata-Version: 2.4
Name: super-xml
Version: 0.1.1
Requires-Dist: pyarrow>=14.0
Requires-Dist: pyspark>=4.0 ; extra == 'spark'
Provides-Extra: spark
License-File: LICENSE
Summary: Distributed, splittable XML record parsing on the succinct DOM — a PySpark DataSource with no JVM native code (serverless-compatible).
License-Expression: MIT
Requires-Python: >=3.9
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM

# super-xml

Distributed, **splittable** XML record parsing built on the succinct DOM —
packaged as a **PySpark custom DataSource with no JVM native code**, so it runs
on Databricks Serverless where JNI/JAR-native data sources are blocked.

```python
import superxml

COLUMNS = [
    ("title", "text", ["title"], "", "utf8"),
    ("id",    "attr", [],        "id", "utf8"),
]

# Split a large file by byte range across executors (default).
df = superxml.read(spark, "/Volumes/.../wiki_dump.xml", "page", COLUMNS)

# Same call, one flag, for a directory of many small files.
df = superxml.read(spark, "/Volumes/.../manyfiles/", "order", COLUMNS,
                   many_small_files=True)

df.count()  # triggers the distributed parse
```

## Why this exists

The succinct DOM parses a real 5.24 GB XML file (DBLP) in ~45 s at 0.75× the
file size in memory — beating lxml, roxmltree, and ElementTree on both memory
and speed, and the only one of them that parsed that file at all. super-xml
makes that engine **usable from Spark** for files too big for one machine, on
serverless runtimes that forbid JVM-native data sources.

## The design in one paragraph

Spark hands each task a *byte range* of a huge file. `parse_partition`
realigns the range to whole-record boundaries (standard Hadoop
`XmlInputFormat` ownership: a partition owns records whose opening tag begins
in its range), wraps the owned records in a synthetic root, and feeds that to
the **unmodified** hardened core parser. So every bit of the core's
correctness work (namespaces, entities, UTF-8 validation, fuzzing) carries
over, and splitting is provably lossless — see the exhaustive "cut at every
byte offset" parity test in `src/lib.rs`.

Distribution rides on Spark's **Python DataSource API** (Spark 4.0+ / DBR
15.4+ / serverless env v2): `partitions()` plans byte ranges on the driver,
`read()` calls the Rust core (via PyO3, GIL released) on executors. Pure
Python + a CPython native extension — no JVM `.so`, so serverless allows it.

## Install

```bash
pip install super-xml        # import name is `superxml`
```

The distribution ships a prebuilt `abi3` manylinux wheel (CPython ≥ 3.9), so
no Rust toolchain is needed to install. `read_many_small_files` additionally
needs `pyspark>=4.0` and a runtime with `pandas>=2.2.0` (Databricks and any
real Spark cluster already provide these — see `pyproject.toml` for why they
are not hard-pinned).

## Results

All on Databricks Serverless (environment v2), input `wiki_139972.xml`
(1,753 MB MediaWiki dump, 139,972 `<page>` records). Full detail:
[`databricks/RESULTS.md`](databricks/RESULTS.md).

| workload | reader | records | time |
|---|---|---|---|
| 1.8 GB single file, byte-range split | **super-xml** | 139,972 / 139,972 | **46.0 s** |
| 1.8 GB single file, byte-range split | native Spark built-in `xml` | 139,972 | 79.1 s |
| 65,536 × 156 KB small files (10 GB) | `read_many_small_files` | — | 237 s |
| nested/structural, ~100k files | super-xml vs lxml | — | at parity/slight edge |

- Lossless: **139,972 / 139,972** parsed, zero loss or duplication across the
  distributed byte-range partitions.
- **1.72× faster** than native Spark XML on the same `count()` (full parse +
  extract), with exactly matching record counts — a fair speed comparison, not
  accuracy-for-speed. Native XML runs entirely in the JVM with no Python-worker
  hop or FFI copies; super-xml wins despite paying all of that.

## One call, two paths

`superxml.read(spark, path, row_tag, columns, *, many_small_files=False)` is the
single entry point. It dispatches to one of two underlying readers:

- **default** — the `SuperXmlSource` DataSource (`format("superxml")`). Splits
  large files by byte range across executors. Use for files too big for one
  machine. Extra `**kwargs` pass through as DataSource options.
- **`many_small_files=True`** — routes a directory of many small files through
  Spark's JVM-side `binaryFile` listing/reads (~3.2× faster than the DataSource
  on 65k small files). Refuses large files it cannot split, via a size guard
  (opt out with `skip_size_stats=True`). Extra `**kwargs` forward to it.

The two paths can't be a single `format(...).option(...)` call: the DataSource
reader runs on an executor with no `SparkSession`, so it can't invoke
`binaryFile` itself — that must happen on the driver. `read` hides that split
behind one flag. You can still call either underlying reader directly.

### Which do I use?

| your input | flag | what happens |
|---|---|---|
| one big file, or a few big files | default (`many_small_files=False`) | each file is byte-range split across executors |
| a directory of **uniformly small** files | `many_small_files=True` | whole-file reads via `binaryFile`, ~3.2× faster on 65k files |
| **mixed** small + huge files | default (`many_small_files=False`) | per-file routing: small files bin-packed together, huge files split — all into one DataFrame |

Rule of thumb: **reach for `many_small_files=True` only when every file is
small** (comfortably under one executor's memory). For anything else — a single
huge file, or a directory that mixes small and huge — use the default; it is the
only path that decides *per file*.

Why the flag is not "auto": the fast path loads each file **whole** into one
task (no byte-range splitting), so it cannot handle a file bigger than a task's
memory. It guards against this — with `many_small_files=True`, a single file
over `max_file_bytes` (default 256 MB) **raises `ValueError`** and points you at
the default reader, rather than silently OOM-ing. (Setting `skip_size_stats=True`
removes that guard *and* the guard's one-time size scan — only do that when you
already know every file is small.) The default DataSource has no such limit: its
driver-side router bin-packs the small files and byte-range-splits the huge ones
in the same pass, so a mixed directory "just works".

## Build & test (from source)

Building from source needs the sibling Rust crate `xml-dom-compress` checked
out alongside this repo (it is a path dependency), plus `maturin`.

```bash
cargo test                        # pure-Rust core + parity suite, no Python needed
maturin build --release --features python --out dist   # build the wheel
```

## Known v1 limitations

- **Single `rowTag`.** Heterogeneous record types (DBLP's
  `article`/`inproceedings`/…) need a set of tags; v1 supports one (perfect for
  wiki `<page>`). Multi-tag is a planned extension.
- **UTF-8 input.** ISO-8859-1 documents (like raw DBLP) need transcoding to
  UTF-8 first; the core rejects non-UTF-8 by policy.

## License

MIT — see [`LICENSE`](LICENSE).

