Metadata-Version: 2.4
Name: fastsimdjson
Version: 2.0.1
Summary: Fast JSON parsing for Python, built on simdjson
Author-email: Daniel Lemire <daniel@lemire.me>
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/simdjson/fastpysimdjson
Project-URL: Repository, https://github.com/simdjson/fastpysimdjson
Project-URL: Issues, https://github.com/simdjson/fastpysimdjson/issues
Keywords: json,simdjson,parser,simd,performance
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: C++
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Programming Language :: Python :: 3.15
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Topic :: Software Development :: Libraries
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Provides-Extra: test
Requires-Dist: pytest; extra == "test"
Requires-Dist: msgpack; extra == "test"
Requires-Dist: typing_extensions; extra == "test"
Provides-Extra: bench
Requires-Dist: orjson; extra == "bench"
Requires-Dist: jiter; extra == "bench"
Requires-Dist: msgspec; extra == "bench"
Requires-Dist: pydantic; extra == "bench"
Requires-Dist: cattrs; extra == "bench"
Requires-Dist: pysimdjson; extra == "bench"
Requires-Dist: cysimdjson; extra == "bench"
Requires-Dist: pyarrow; extra == "bench"
Requires-Dist: polars; extra == "bench"
Requires-Dist: duckdb; extra == "bench"
Requires-Dist: msgpack; extra == "bench"
Requires-Dist: ormsgpack; extra == "bench"

# fastsimdjson

Fast JSON for Python, built on [simdjson](https://github.com/simdjson/simdjson),
the parser that reads JSON at gigabytes per second. fastsimdjson is a drop-in
replacement for the `json` module (`loads`, `load`, `dumps`, `dump`) that
gives the same results 3.5 to 4.3 times faster, and it goes further: lazy
parsing, streams of documents on many threads, Apache Arrow tables and
MessagePack, in one package with no dependencies.

## How it compares

fastsimdjson is often two to four times faster than the alternatives:

<picture>
  <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/simdjson/fastpysimdjson/main/doc/perf_summary_dark.png">
  <img src="https://raw.githubusercontent.com/simdjson/fastpysimdjson/main/doc/perf_summary.png" width="80%" alt="fastsimdjson against the competition, speed on one core relative to a fast alternative: Python objects to JSON str, 4.3 times json.dumps; MessagePack to JSON, 4.2 times ormsgpack with orjson; JSON to MessagePack, 3.6 times orjson with ormsgpack; NDJSON to Arrow, 3.6 times polars; Arrow to NDJSON, 2.4 times polars; a few fields of a document, 2.2 times msgspec; JSON to Python objects, 1.3 times orjson">
</picture>

Against the standard `json` module, `loads` is 3.5 times faster and `parse`
25 times faster. The files, the versions and the scripts are under
[Benchmarks](#benchmarks).

* **Same results.** `loads` returns what `json.loads` returns, `dumps` the
  same `str` as `json.dumps`, character for character, and `dumpb` the same
  bytes as `orjson.dumps`. Switching changes the speed, not the output.
* **Typed.** `loads(data, type=T)` builds dataclasses, TypedDicts and
  other typed values straight from the parser, with the types checked: 1.6
  to 2.1 times as fast as msgspec, 3.9 to 5.2 times as fast as pydantic.
* **Only what you need.** `parse` gives lazy views and builds only the
  values you read; `fields=` extracts only some fields from each document of
  a stream.
* **Many threads.** On free-threaded Python, streams of documents (NDJSON,
  JSON Lines) are read and written on many threads: up to 21 GB/s on 64
  cores. Arrow and MessagePack conversions release the GIL and use many
  threads on any Python: on 32 cores, `loads_arrow` reads NDJSON into Arrow
  at up to 28.6 GB/s, 2.3 to 2.8 times as fast as pyarrow.
* **Arrow without pyarrow.** `loads_arrow` and `dumpb_arrow` convert NDJSON
  to Arrow tables and back, for pyarrow, polars or DuckDB, without a copy.
* **MessagePack without Python objects.** `json_to_msgpack` and
  `msgpack_to_json` convert directly, without building Python objects.
* **Minify without parsing.** `minify` strips insignificant whitespace
  and returns `bytes`, without building Python objects.
* **Everywhere.** No dependencies; wheels for Linux (x64 and ARM), macOS
  (Intel and Apple silicon) and Windows, Python 3.10 to 3.15, including
  free-threaded Python, and for Pyodide (WebAssembly).

## Installation

With pip:

```sh
pip install fastsimdjson
```

With uv:

```sh
uv pip install fastsimdjson      # in a uv project: uv add fastsimdjson
```

Wheels are available for Linux, macOS and Windows, for Python 3.10 to 3.15,
including free-threaded Python 3.14 and 3.15.

Wheels are also available for Pyodide (WebAssembly, in the browser or in
Node), for Python 3.13 to 3.15:

```python
import micropip
await micropip.install("fastsimdjson")
```

Under Pyodide, Python cannot start threads: `threads=` runs on the calling
thread.

## Usage

### `loads`: the whole document

```python
import fastsimdjson
fastsimdjson.loads(b'{"a": [1, 2.5, "x", true, null]}')
# {'a': [1, 2.5, 'x', True, None]}
```

`loads(data)` accepts `bytes`, `bytearray`, `memoryview` and `str`, and
returns the same value as `json.loads`, with the same types and key order.
Integers that do not fit in 64 bits become exact Python ints. `NaN`,
`Infinity` and `-Infinity` are accepted, in any capitalization.

Invalid input raises `fastsimdjson.JSONDecodeError`, a subclass of
`json.JSONDecodeError`. When simdjson rejects a document, the same input is
parsed with `json.loads`. If that succeeds, `loads` returns its value (this
is how a number that overflows a double becomes `inf`). If it raises
`JSONDecodeError`, the exception is re-raised with Python's message and byte
position. Any other exception from `json.loads` propagates.

### `parse`: lazy views

When you need only part of a document, `parse` avoids building the rest.
It accepts the same inputs as `loads` and returns read-only views:
`fastsimdjson.Object` (a `Mapping`) and `fastsimdjson.Array` (a
`Sequence`). Values are converted when you access them; nested objects and
arrays are returned as views. A scalar root is returned as a plain value.

```python
doc = fastsimdjson.parse(open("twitter.json", "rb").read())
ids = [(s["id"], s["user"]["screen_name"]) for s in doc["statuses"]]
doc.at_pointer("/statuses/0/user/name")   # JSON Pointer (RFC 6901)
doc["search_metadata"].as_dict()          # convert a subtree, like loads
```

`Object` supports `obj[key]`, `get`, `in`, `len`, iteration over the keys,
`keys()`, `values()`, `items()` (iterators), `at_pointer` and `as_dict()`.
`Array` supports `arr[i]` (negative indexes and slices), `len`, iteration,
`at_pointer` and `as_list()`. Both work with `match` statements.

* A view keeps its document alive; the document owns its own buffers, so
  it remains valid while other documents are parsed.
* A key lookup scans the object. With duplicate keys, lookups return the
  first value, whereas `as_dict()` (like `json.loads`) keeps the last.
* Indexing an array walks it from the last index reached, so a loop over
  `arr[i]` is linear; iteration is the fastest way to visit an array.
* Errors are handled as in `loads`. A document that simdjson rejects but
  `json.loads` accepts (an overflowing number, an unpaired surrogate) is
  returned as plain Python objects, as `loads` would return it.

### `type=`: typed decoding

`loads(data, type=T)` returns the document as an instance of `T`, checked
on the way. It builds dataclasses straight from the simdjson tape, without
an intermediate dict, and is 1.6 to 2.1 times as fast as msgspec on one core
(see [Typed decoding](#typed-decoding)).

<picture>
  <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/simdjson/fastpysimdjson/main/doc/perf_typed_dark.png">
  <img src="https://raw.githubusercontent.com/simdjson/fastpysimdjson/main/doc/perf_typed.png" width="80%" alt="JSON to dataclasses on one core, GB/s. twitter.json, a few fields: fastsimdjson 2.59, msgspec Struct 1.53, msgspec dataclasses 1.43, pydantic 0.66, orjson with cattrs 0.57. citm_catalog.json typed in full: fastsimdjson 1.00, msgspec Struct 0.64, msgspec dataclasses 0.47, pydantic 0.19, orjson with cattrs 0.19. GH Archive NDJSON, 6 fields: fastsimdjson 2.05, msgspec Struct 1.29, msgspec dataclasses 1.16, pydantic 0.52, orjson with cattrs 0.42">
</picture>

```python
from dataclasses import dataclass, field
from typing import Literal, Optional

@dataclass
class User:
    id: int
    login: str
    followers: int = 0

@dataclass
class Repo:
    id: int
    name: str
    owner: User
    topics: list[str] = field(default_factory=list)
    visibility: Literal["public", "private"] = "public"
    license: Optional[str] = None

repo = fastsimdjson.loads(data, type=Repo)
repos = fastsimdjson.loads(data, type=list[Repo])
events = list(fastsimdjson.loads_many(ndjson, type=Repo))
```

`type` accepts:

* `int`, `float` (an integer becomes a float), `str`, `bool`, `type(None)`
  (JSON `null`; `None` too inside a type, as in `list[None]`), and `Any` or
  `object` (no check: the value as `loads` returns it). `type=None`, the
  default, means no type;
* `list`, `tuple` (fixed length or `tuple[T, ...]`), `set`, `frozenset`,
  `dict` with `str` keys, and their abstract counterparts (`Sequence`,
  `Mapping`, ...);
* `Optional[T]`, unions (`int | str`, at most one type per JSON kind; an
  integer goes to `int` rather than to `float`) and `Literal` (of `str`,
  `int`, `bool` and `None`: JSON `1` is not `True`);
* enums (by value: `str`, `int`, `bool` or `None`), dataclasses (including
  frozen ones and those with `__slots__`), `TypedDict` (from `typing` or
  `typing_extensions`) and `NamedTuple` (from an array);
* `Annotated`, `NewType` and `type` aliases, and recursive types.

Dataclass fields take their defaults (and default factories) when the key
is missing, keys that are not fields are ignored, `__post_init__` runs, and
`init=False` fields get their default. The fields are set as the generated
`__init__` sets them: an `__init__` of your own is not called. A `TypedDict` becomes a `dict` with
its keys only. A value of the wrong type raises
`fastsimdjson.ValidationError` (a `ValueError`) with its path:

```
ValidationError: Expected `int`, got `str` - at `$.owner.id`
```

A `ValueError` or `TypeError` raised by `__post_init__` becomes a
`ValidationError` too, with the path. Invalid JSON raises
`JSONDecodeError`, and the documents that only `json` accepts (`NaN`, a
number that overflows a double) are converted the same way, from what
`json` returns; past what Python's stack allows, those raise
`RecursionError`, as `json` does. A set of values that cannot be hashed
(`set[list[int]]`) or a union that contains itself is a `TypeError`. The
description of each type is computed once and cached (the cache starts over
past 1024 types, so that types made on the fly do not pile up). `load`,
`load_file`, `loads_many` and `map_many` accept `type=` too; a
`ValidationError` ends the iteration of `loads_many`, as any error does. With
`threads=N` on free-threaded Python, the instances are built on N threads
(not with `fields=` or `lazy=True`).

### `dumps` and `dump`: like `json.dumps`

```python
fastsimdjson.dumps({"a": [1, 2.5, None]})            # '{"a": [1, 2.5, null]}'
fastsimdjson.dumps(obj, indent=2, sort_keys=True)
with open("out.json", "w", encoding="utf-8") as f:
    fastsimdjson.dump(obj, f)
```

`dumps(obj, **kw)` takes the arguments of `json.dumps` and returns the same
`str`, character for character, including the float format (`repr`), the
escapes and the default separators. `ensure_ascii`, `indent`, `separators`,
`sort_keys`, `allow_nan` and `default` are handled in C. Everything else is
passed to `json.dumps` itself, which produces the result or raises its usual
exception: a `cls` argument or other encoder options, `skipkeys`, a circular
reference, `NaN` with `allow_nan=False`, a key or a value that `json` cannot
serialize. `dump(obj, fp, **kw)` writes `dumps(obj, **kw)` to `fp`.

### `dumpb`: like `orjson.dumps`

```python
fastsimdjson.dumpb({"a": [1, 2.5, None]})            # b'{"a":[1,2.5,null]}'
fastsimdjson.dumpb(obj, option=fastsimdjson.OPT_INDENT_2 | fastsimdjson.OPT_SORT_KEYS)
fastsimdjson.dumpb({1, 2}, default=sorted)           # b'[1,2]'
```

`dumpb(obj, default=None, option=None)` has the arguments, the output and the
errors of `orjson.dumps` (orjson 3.12): `orjson.dumps(obj, ...)` can be
replaced by `fastsimdjson.dumpb(obj, ...)` for the types below. It returns
compact UTF-8 `bytes`, writes floats as orjson does (`1e-6`, `1e+16`, `NaN`
and infinities as `null`), and raises `TypeError` with orjson's messages:
integers beyond 64 bits, a dict key that is not a `str`, invalid UTF-8 (a lone
surrogate), nesting deeper than 254, a type it cannot serialize. `orjson.JSONEncodeError` is a subclass of `TypeError`, so
`except TypeError` catches the errors of both.

It serializes `str`, `int`, `float`, `bool`, `None`, `dict`, `list`, `tuple`,
enums, and subclasses of `str`, `int`, `dict` and `list`. Anything else goes
to `default`, as in orjson: its result is serialized in place of the object,
and an exception it raises becomes the `__cause__` of the `TypeError`. The
options are exported under orjson's names and values (`OPT_APPEND_NEWLINE`,
`OPT_INDENT_2`, `OPT_NON_STR_KEYS`, `OPT_PASSTHROUGH_SUBCLASS`,
`OPT_SORT_KEYS`, `OPT_STRICT_INTEGER`, ...). Unlike orjson, `dumpb` does not
serialize dataclasses, `datetime`, `date`, `time`, `UUID`, numpy arrays or
`orjson.Fragment` itself: they go to `default`, as if
`OPT_PASSTHROUGH_DATACLASS` and `OPT_PASSTHROUGH_DATETIME` were set, and the
options that concern only these types are accepted and have no effect.

### `minify`: strip whitespace without parsing

```python
fastsimdjson.minify(b'{ "a" : [1, 2] }')    # b'{"a":[1,2]}'
```

`minify(data)` removes insignificant whitespace (spaces, tabs, newlines and
carriage returns outside of strings) and returns `bytes`. It accepts `bytes`,
`bytearray`, `memoryview` and `str`. It does not parse the document and does
not validate it: numbers, escapes, duplicate keys and key order are left as
they are, and input that is not valid JSON is minified the same way. An
unterminated string raises `JSONDecodeError` (the minifier does not report
where the string started, so the position is 0). The call releases the GIL.

### `dumpb_many`: NDJSON, on several threads

```python
data = fastsimdjson.dumpb_many(records)               # b'{...}\n{...}\n'
data = fastsimdjson.dumpb_many(records, threads=16)   # free-threaded Python
```

`dumpb_many(records, default=None, option=None, *, threads=None)` returns
the records (a list, a tuple or any iterable) as NDJSON `bytes`: each record
as `dumpb` writes it, followed by a newline, so that the output reads back
with `loads_many(data, format="lines")`. `default` and `option` are those of
`dumpb`, except `OPT_INDENT_2`, which would spread a record over several
lines (`ValueError`); `OPT_APPEND_NEWLINE` changes nothing.

On free-threaded Python, `threads=N` writes chunks of records on N threads at
once (`threads=0` or `None`: one per processor), with the worker threads of
`map_many`. The output and the errors are those of one thread: when records
cannot be serialized, the error of the first of them is raised (`default` may
then have been called on later records, in other threads, and it must be
safe to call from several threads at once). With the GIL, the calling thread
does all the work. Like `dumpb`, `dumpb_many` does not crash when other
threads change the records meanwhile, but the output then reflects some of
the changes and not others.

### Files

```python
with open("data.json", "rb") as f:
    doc = fastsimdjson.load(f)          # like json.load: f.read(), then loads
doc = fastsimdjson.load_file("data.json")
view = fastsimdjson.parse_file("data.json")   # lazy, like parse
```

`load_file(path)` and `parse_file(path)` accept a `str`, `bytes` or
`os.PathLike` path and raise `OSError` (e.g. `FileNotFoundError`) when the
file cannot be read.

### `loads_many` and `parse_many`: streams of documents

```python
for record in fastsimdjson.loads_many(open("log.ndjson", "rb").read()):
    ...
for view in fastsimdjson.parse_many(data):    # lazy views, like parse
    ...
```

Both return an iterator over the documents of `data` (`bytes`, `bytearray`,
`memoryview` or `str`). The `format` keyword selects how documents are
separated:

| `format` | input |
|---|---|
| `"whitespace"` (default) | documents separated by white space, including NDJSON and JSON Lines |
| `"lines"` | one document per line (NDJSON, JSON Lines) |
| `"json_seq"` | RFC 7464 JSON text sequences (each document preceded by `\x1e`) |
| `"comma"` | documents separated by commas: `{...}, {...}` (simdjson also accepts white space between them) |
| `"array"` | the elements of one array: `[{...}, {...}]` |

To read a file without copying it, map it in memory:

```python
import mmap
with open("log.ndjson", "rb") as f, mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) as m:
    names = fastsimdjson.map_many(fn, memoryview(m), threads=64)
```

A `memoryview` of `bytes`, or of a file mapped with `mmap.ACCESS_READ` (as
here), is read in place, and the mapped file must not change meanwhile;
other buffers (a `bytearray`, a view of a writable mmap) are copied first.

simdjson parses the input in batches (`batch_size`, 1 MB by default); a
larger document is handled automatically. In every format, documents that
simdjson rejects are handled as in `loads`: a document that `json` accepts is
returned, otherwise `JSONDecodeError` reports `json`'s message and the
position in the whole input. A truncated last document is an error. The
views returned by `parse_many` remain valid after the iterator moves on.

### Streams on several threads: `threads=N` and `map_many`

On free-threaded Python (3.14t and newer), a stream can be parsed by several
threads at once:

```python
docs = fastsimdjson.loads_many(data, threads=16)        # a list, in order
views = fastsimdjson.parse_many(data, threads=16)       # a list of views
# fn runs in the threads too, on each document; its results, in order:
scores = fastsimdjson.map_many(score, data, threads=16)
```

With `threads`, `loads_many` and `parse_many` return a list of all the
documents instead of an iterator. `map_many(fn, data, **kw)` returns
`[fn(doc) for doc in loads_many(data, **kw)]` (`parse_many` with
`lazy=True`), with `fn` called in the threads. `threads=0` (or `None`) uses
one thread per processor; small inputs use fewer threads (about one per
512 KB). The workers share one counter. Each claims the next 256 KB and
moves both ends forward to a newline (to the next `\x1e` for `json_seq`);
the pieces are adjacent. On `lines` and `json_seq` a document is never cut,
and one longer than 256 KB is parsed with the piece that started before it.
`threads` works with the formats `"whitespace"`, `"lines"` and `"json_seq"`.

The results, and the exceptions, are those of one thread: when a chunk
contains an error or a document that only `json` accepts, the documents
before it are kept and the rest is parsed in the calling thread, as without
threads. An exception raised by `fn` is raised again, for the first document
where it happened; `fn` may then have been called on later documents, in
other threads, and it must be safe to call from several threads at once.

The worker threads are started by the first call that needs them and kept
for later calls (they wait, detached from the interpreter, between calls,
and stop at exit). `map_many` scales best: each document is built, used and
freed in the same thread, and only `fn`'s results come back. `loads_many` must return every
document, and the Python objects of all of them, to the calling thread.
With the GIL (the default build of Python), the calling thread does all the
work: `threads` has no effect on speed.

With `drop_none=True`, `map_many` leaves out the results that are `None`:
when `fn` keeps only a few documents, the threads do not build a list
entry for each of the others, and the calling thread has fewer to collect.

```python
def starred(event):
    kind, repo = event
    return repo if kind == "WatchEvent" else None

names = fastsimdjson.map_many(starred, data, threads=64, drop_none=True,
                              fields=["/type", "/repo/name"])
```

### Only some fields: `fields=`

When you need a few values from each document, `fields` extracts them
without building the documents: only the values found become Python
objects.

```python
events = fastsimdjson.loads_many(data, fields=["/type", "/payload/action"])
# (type, action) for each document

def count(t):
    kind, action, language = t
    ...
fastsimdjson.map_many(count, data, threads=64,
                      fields=["/type", "/payload/action",
                              "/payload/pull_request/base/repo/language"])
```

`loads_many` and `map_many` accept `fields`: a JSON Pointer (RFC 6901), or a
list or tuple of them. Each document then becomes the value at the pointer,
or a tuple of the values at the pointers, in order. A value that is an
object or an array is built in full. A pointer that leads nowhere (a
missing key, an index past the end, or a step into a scalar) gives
`default`, `None` unless you pass another value; a JSON `null` gives `None`.
As with `at_pointer`, a key that appears twice in an object refers to its
first occurrence. `fields` cannot be combined with `lazy=True`.

With the formats `"whitespace"` and `"lines"`, `fields` reads the documents
with simdjson's On-Demand API: the top level of each document, the keys on
the way to the pointers and the values found there are read and validated,
so a malformed or incomplete document is still an error. The values that no
pointer reaches are skipped without being checked, so an invalid value
elsewhere in a document (`{"a": 1, "b": tru}` with `fields="/a"`) goes
unnoticed. A value that is read is handled as without `fields`: an error
raises `JSONDecodeError`, and a document that only `json` accepts is decoded
by `json`. The whole documents are parsed (and validated) with the other
formats, and when a pointer is `""` or leads into the value of another
pointer (`["/a", "/a/b"]`).

On 7.6 GB of GitHub events, `map_many` with three fields ran at 3.0 GB/s on
one core of an Intel Xeon Gold 6548N, against 1.8 GB/s with lazy views
(`lazy=True`). With 64 threads (free-threaded Python 3.14.10t), it ran at
113 GB/s (lazy views: 81 GB/s).

### Arrow: `loads_arrow`, `load_arrow_file` and `dumpb_arrow`

```python
import pyarrow as pa
table = pa.table(fastsimdjson.loads_arrow(data))             # inferring the schema
schema = pa.schema([("id", pa.int64()), ("name", pa.string())])
table = pa.table(fastsimdjson.load_arrow_file("log.ndjson", schema, threads=8))
frame = polars.DataFrame(fastsimdjson.loads_arrow(data, schema))
data = fastsimdjson.dumpb_arrow(table)                       # b'{"id":1,"name":"x"}\n...'
data = fastsimdjson.dumpb_arrow(frame, threads=8)            # a polars.DataFrame
```

`loads_arrow(data, schema=None, *, block_size=1048576,
error_on_unknown_fields=False, validate_skipped=True, threads=None)` reads
NDJSON (one object per record) into Arrow record batches. The input is
`bytes`, `bytearray`, `memoryview` or `str`, read in place.
`load_arrow_file(path, ...)` maps the file in memory and reads it the same way
(the file must not shrink meanwhile). Both return a `fastsimdjson.ArrowStream`,
which goes to any library that supports the [Arrow PyCapsule
interface](https://arrow.apache.org/docs/format/CDataInterface/PyCapsuleInterface.html),
without a copy: `pyarrow.table(s)`, `pyarrow.RecordBatchReader.from_stream(s)`,
`polars.DataFrame(s)`, `duckdb.from_arrow(s)`... Like a `RecordBatchReader`,
it hands over its batches once (`ValueError` afterwards); `pyarrow.schema(s)`
gives its schema at any time, and `s.num_rows` the number of records (`None`
once the batches are taken). fastsimdjson does not depend on pyarrow.

* The schema is a `pyarrow.Schema` (or any object with `__arrow_c_schema__`)
  whose fields are booleans, 64-bit integers, unsigned 64-bit integers,
  doubles, strings, timestamps, structs and lists of these, or null. A field absent from a
  record is null; a key absent from the schema is skipped, or an error with
  `error_on_unknown_fields=True`. Skipped values are checked to be valid
  JSON, unless `validate_skipped=False` (then only their structure is).
* Without a schema, it is inferred as pyarrow's JSON reader does: `null`,
  `true`/`false`, integers that fit in 64 bits (`int64`), other numbers
  (`double`), strings, objects (structs, with the union of their fields in
  order of first appearance) and arrays (lists). A field of nulls takes the
  type of its other values and integers become doubles next to other
  numbers; other mixes of types raise `ValueError`. Strings stay strings (no
  timestamps).
* A timestamp field (`pa.timestamp("us", tz="UTC")`, any unit, with or
  without a time zone) reads ISO 8601 strings, such as the `created_at` of
  GH Archive events (`"2015-01-01T15:00:00Z"`): a date (a four-digit year), then optionally a
  time (`T` or a space, then `15`, `15:00`, `15:00:00` or
  `15:00:00.123456`), then optionally `Z` or an offset (`+05:00`). A time
  without `Z` or an offset is UTC. An integer is a count of units since 1970.
  As with `pyarrow.json.read_json` (which returns the same table from the
  strings it accepts; a lowercase `t` or `z` and a comma before the fraction
  are accepted here too), a fraction
  with more digits than the unit raises `ValueError`, as does an invalid
  date. Reading the timestamps costs next to nothing: GH Archive with
  `created_at` as `timestamp[us, UTC]` reads at 2.11 GB/s on one core, as
  with strings, against 0.45 GB/s for pyarrow and 0.21 GB/s for polars.
* A new batch starts every `block_size` bytes of input. With `threads=N`
  (`threads=0` or `None`: one per processor), N threads read blocks of
  `block_size` bytes, cut at line ends: a record must then fit on one line.
  When that fails, the input is read again on one thread, so that the errors
  (`ValueError`, with the record number and the field) are the same with any
  number of threads.

`dumpb_arrow(data, *, write_nulls=True, threads=None)` writes Arrow data as
NDJSON `bytes`, one object per row: a `pyarrow.Table`, `RecordBatch`,
`RecordBatchReader` or `StructArray`, a `polars.DataFrame`, a DuckDB
relation, an `ArrowStream`, or any object with `__arrow_c_stream__` or
`__arrow_c_array__`. It writes integers of 8 to 64 bits, floats and doubles
(as `dumpb` writes them, NaN and infinities as `null`), booleans, strings
(including large strings and string views), timestamps (ISO 8601 in UTC,
`"2015-01-01T15:00:00.250000Z"`, with `Z` when the timestamp has a time zone
and the fraction when it is not zero), dates (`"2015-01-01"`), structs and
lists (including large and fixed-size lists); other types raise
`ValueError`. With
`write_nulls=False`, null fields are omitted. With `threads=N`, N threads
write ranges of rows into the result. The output reads back with
`loads_arrow`, and with `loads_many(data, format="lines")`, except for
timestamps and dates outside the years 0000 to 9999: their year is written
with more digits or a minus sign, which `loads_arrow` does not read (nor
does pyarrow).

These functions release the GIL and run their threads outside Python, with
or without free threading. They are built on
[simdjson_arrow](https://github.com/simdjson/simdjson_arrow), a C++ library
(vendored).

### MessagePack: `json_to_msgpack` and `msgpack_to_json`

```python
packed = fastsimdjson.json_to_msgpack(b'{"id": 1, "tags": ["a", "b"], "score": 2.5}')
# b'\x83\xa2id\x01\xa4tags\x92\xa1a\xa1b\xa5score\xcb@\x04\x00\x00\x00\x00\x00\x00'
msgpack.unpackb(packed)              # {'id': 1, 'tags': ['a', 'b'], 'score': 2.5}
fastsimdjson.msgpack_to_json(packed)  # b'{"id":1,"tags":["a","b"],"score":2.5}'
stream = fastsimdjson.json_to_msgpack(ndjson, ndjson=True, threads=8)  # one value per document
lines = fastsimdjson.msgpack_to_json(stream, stream=True, threads=8)   # NDJSON again
```

`json_to_msgpack(data, *, ndjson=False, block_size=1048576, threads=None)`
converts JSON (`bytes`, `bytearray`, `memoryview` or `str`, read in place) to
MessagePack `bytes` in the most compact encoding: the same bytes as
`msgpack.packb(json.loads(data))` (but for duplicate keys, which
`json.loads` drops, and integers beyond 64 bits). Integers take the smallest integer encoding
(integers beyond 64 bits become doubles), other numbers are doubles, and
strings, arrays and objects take the smallest header for their size; keys keep
their order, duplicates too. With `ndjson=True`, the input holds documents
separated by whitespace (NDJSON, JSON Lines), and each becomes one MessagePack
value, one after the other.

`msgpack_to_json(data, *, stream=False, validate_utf8=True,
binary_as_base64=True, ext_as_null=False, block_size=1048576, threads=None)`
converts MessagePack (any bytes-like object) to compact JSON `bytes`. Floats
get the shortest representation that gives them back, as Python's `repr`
(`1.0`, not `1`; `1e-05`; NaN and infinities as `null`), binary values as base64 strings, map keys that are not strings as
strings (`"1"`, `"true"`, `"null"`); extension values (timestamps too) raise
`ValueError`, or become `null` with `ext_as_null=True`. Strings must be valid
UTF-8 (unless `validate_utf8=False`). With `stream=True`, the input holds a
sequence of values (what `json_to_msgpack(..., ndjson=True)` or several
`msgpack.packb` calls give), and each becomes one line of NDJSON.

* Invalid input raises `ValueError`: `"TAPE_ERROR: ... (document 12)"` for
  JSON, `"truncated string (byte 1234)"` for MessagePack.
* With `threads=N` (`threads=0` or `None`: one per processor), NDJSON and
  streams are converted on N threads, in blocks of about `block_size` bytes.
  A JSON document must then fit on one line; when that fails, the input is
  converted again on one thread, so that errors are the same with any number
  of threads. Single documents use one thread.
* The functions release the GIL and run their threads outside Python, with or
  without free threading, and write their result straight into the `bytes`
  object (no copy). No Python object is made from the data.

They are built on
[simdjson_msgpack](https://github.com/simdjson/simdjson_msgpack), a C++ library
(vendored).

### Threads and free-threaded Python

The module is marked free-threading compatible (`Py_MOD_GIL_NOT_USED` on
Python 3.13 and newer), so importing it on a free-threaded build does not
re-enable the GIL. It is tested with many threads parsing, serializing and
sharing views and stream iterators on free-threaded Python 3.14 and 3.15.

`bytes` and `str` inputs are read in place. A `bytearray` or `memoryview`
could be changed by another thread while we read it, so it is copied first:
always by `loads_many`, `parse_many` and `map_many` (the iterator outlives
the call) and by `loads_arrow`, `json_to_msgpack` and `msgpack_to_json`
(which release the GIL), and only on a free-threaded build by `loads` and
`parse`. The exception: `loads_many`, `parse_many` and `map_many` read a
read-only, contiguous `memoryview` of `bytes`, or of a file mapped with
`mmap.ACCESS_READ`, in place (the file must not change meanwhile). Other
read-only views (`memoryview(bytearray).toreadonly()`, a read-only view of a
writable mmap) are copied, since what they show can still change. Likewise, on a free-threaded build, `dumps` and `dumpb` hold
references to the items of the containers they serialize.

### `release`

`release()` frees the simdjson parser and the string caches kept by the
calling thread. On free-threaded Python, it also has the idle worker threads
of the parallel streams (`threads=N`) free theirs; a worker busy with a
stream of another thread does so when it is done. Views returned by `parse`
remain valid.

## Build and test

Python 3.10 or newer, and a C++17 compiler (clang, GCC or MSVC). The simdjson
5.1.1 and simdutf 9.2.1 amalgamations, simdjson_arrow 0.1.5 (NDJSON to Arrow
and back: `simdjson_arrow.h` and `simdjson_arrow.cpp`) and simdjson_msgpack
0.1.1 (JSON to MessagePack and back: `simdjson_msgpack.h` and
`simdjson_msgpack.cpp`), both under the Apache-2.0 license, are already in
`vendor/`. simdjson_arrow includes zmij 1.2 (shortest float formatting, MIT
license), which simdjson_msgpack, `dumps` and `dumpb` also use
(`vendor/zmij.h`).
The tests of the Arrow functions (`tests/test_arrow.py`) run when pyarrow is
installed (and polars, for one of them); otherwise they are skipped.

pip:

```sh
python3 -m venv .venv
. .venv/bin/activate
python -m pip install -U pip setuptools
python -m pip install -e ".[test]"
pytest tests
```

uv:

```sh
uv venv
. .venv/bin/activate
uv pip install -e ".[test]"
pytest tests
```

Either one builds the extension and makes `import fastsimdjson` work in the
virtualenv. uv uses the `setuptools` build requirement from `pyproject.toml`,
so it does not need a separate setuptools install for this path.

To compile the extension in the tree instead, install setuptools and pytest
into the same virtualenv, then:

```sh
python -m pip install setuptools pytest
python setup.py build_ext --inplace
PYTHONPATH=src python -m pytest tests
```

```sh
uv pip install setuptools pytest
python setup.py build_ext --inplace
PYTHONPATH=src python -m pytest tests
```

Recent setuptools copies the `.so` next to `src/fastsimdjson.cpp`, which is
why `PYTHONPATH=src` is required for the in-place build.

`tests/test_loads.py` compares `loads` with `json.loads` on types and key
order; `tests/test_lazy.py` checks the views returned by `parse` the same way,
`tests/test_dumps.py` compares `dumps` with `json.dumps` and `dumpb` with
orjson (output and exceptions; the orjson comparisons are skipped when orjson
is not installed, and recorded orjson results are checked either way), and
`tests/test_stream.py` and `tests/test_files.py` cover streams and files, and
`tests/test_minify.py` covers `minify`. The suite
covers scalars, integers past 64 bits, UTF-8 strings at every length from 0 to
199, the key cache, random documents, rejected input, deep
nesting, padding at a page boundary, a saturated array count, reference
counts, and release of a parser that has grown past 64 MB. The corpus test
is skipped until simdjson-data is checked out beside the project:

```sh
git clone --depth 1 https://github.com/simdjson/simdjson-data.git
pytest tests
# or: JSONDIR=/path/to/jsonexamples pytest tests
```

The suite builds an ~80 MB document and a list of 16,777,221 integers, so
give it some RAM.

The benchmark scripts need simdjson-data and the `bench` extra (orjson,
jiter, msgspec, pysimdjson, cysimdjson). `bench.py` times `loads` against
`json.loads`, orjson and jiter, `bench_lazy.py` times `parse` against pysimdjson and
cysimdjson, `bench_dumps.py` times `dumps` against `json.dumps` and `dumpb`
against orjson and msgspec, `bench_many.py` times `loads_many`, and
`bench_minify.py` times `minify` against parsing and serializing again.

pip:

```sh
python -m pip install -e ".[bench]"
python bench.py
python bench_lazy.py
python bench_dumps.py
python bench_many.py
python bench_minify.py
```

uv:

```sh
uv pip install -e ".[bench]"
python bench.py
python bench_lazy.py
python bench_dumps.py
python bench_many.py
python bench_minify.py
```

## Benchmarks

Intel Xeon Gold 6548N (Emerald Rapids), one core unless stated otherwise,
Python 3.14.10 (free-threaded Python 3.14.10t for the streams on several
threads), fastsimdjson 2.0.1 (simdjson 5.1.1) built with GCC 16.2, the 22
files of [simdjson-data](https://github.com/simdjson/simdjson-data). The scripts are in
this repository and in
[the blog repository](https://github.com/lemire/Code-used-on-Daniel-Lemire-s-blog/tree/master/2026/09/pysimdjson).

### Whole documents

Each parser produces the whole document as Python objects. Speed is the
geometric mean over the 22 files (higher is better).

<picture>
  <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/simdjson/fastpysimdjson/main/doc/perf_loads_dark.png">
  <img src="https://raw.githubusercontent.com/simdjson/fastpysimdjson/main/doc/perf_loads.png" width="80%" alt="Parsing JSON in Python: fastsimdjson loads 0.77 GB/s, orjson 0.60, jiter 0.54, msgspec 0.53, jiter without its string cache 0.44, pysimdjson 0.44, cysimdjson 0.43, ujson 0.37, python-rapidjson 0.24, simplejson 0.23, json 0.22">
</picture>

| parser | GB/s | vs `json.loads` |
|---|---:|---:|
| json (standard library) | 0.22 | 1.00× |
| simplejson 4.3.0 | 0.23 | 1.04× |
| python-rapidjson 1.26 | 0.24 | 1.08× |
| ujson 6.1.0 | 0.37 | 1.65× |
| cysimdjson 26.27 | 0.43 | 1.95× |
| pysimdjson 8.0.0 | 0.44 | 1.99× |
| jiter 0.19.0 (no string cache) | 0.44 | 1.99× |
| msgspec 0.22.0 | 0.53 | 2.38× |
| jiter 0.19.0 (string cache, the default) | 0.54 | 2.44× |
| orjson 3.13.0 | 0.60 | 2.70× |
| **fastsimdjson `loads`** | **0.77** | **3.47×** |

fastsimdjson is the fastest on 21 of the 22 files. orjson is 13% faster on
`numbers.json`, an array of floating-point numbers. On numbers, both spend
most of their time creating Python floats, which costs the same in both:
simdjson parses the numbers of these files faster than orjson's parser
(yyjson), but that is a small part of the total. Part of the gain comes from
pausing the garbage collector while the objects are built. fastsimdjson's
lead over orjson is 1.28× with the collector enabled and 1.17× with it
disabled (geometric means); without the collector, orjson is also 4 to 5%
faster on `canada.json` and `mesh.json`. yyjson 4.0.6 is left out: it
returns wrong strings for non-ASCII text. jiter (the parser of pydantic)
caches strings by default; turning the cache off (`cache_mode=False`) makes it
slower here.

Parsing is no longer the bottleneck. simdjson alone parses these files at
3.1 GB/s. It accounts for about a third of the time of `loads`; the rest goes
into creating Python objects. Freeing those objects later costs about a sixth
of the total. Even if parsing took no time at all, `loads` would be less than
1.5 times faster.

### Parts of documents with `parse`

If you only need a few values, `parse` creates only those. Extracting the id
and the screen name of the 100 statuses of `twitter.json`:

<picture>
  <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/simdjson/fastpysimdjson/main/doc/perf_parse_dark.png">
  <img src="https://raw.githubusercontent.com/simdjson/fastpysimdjson/main/doc/perf_parse.png" width="80%" alt="Reading part of twitter.json: fastsimdjson parse 156 µs, pysimdjson 170, cysimdjson 235, msgspec 336, fastsimdjson loads 843, orjson 981, jiter 1148, jiter without its string cache 1580, json 3977">
</picture>

| method | µs |
|---|---:|
| `json.loads` | 3977 |
| jiter (no string cache) | 1580 |
| jiter (string cache) | 1148 |
| orjson | 981 |
| fastsimdjson `loads` | 843 |
| msgspec (typed `Struct`) | 336 |
| cysimdjson (lazy) | 235 |
| pysimdjson (lazy) | 170 |
| **fastsimdjson `parse`** | **156** |

Here `parse` is about 25 times faster than `json.loads` and 5.4 times faster
than `loads`. Most of its time is the simdjson parse itself.

### Typed decoding

`loads(data, type=T)` against msgspec (`msgspec.json.Decoder(T)` with the
same dataclasses, and with the equivalent `msgspec.Struct` classes), pydantic
(`TypeAdapter(...).validate_json`, with the equivalent models) and
`cattrs.structure(orjson.loads(data), T)`. Three cases:

* **twitter:** `twitter.json` as a dataclass of 100 statuses, six fields of
  each status and four of its user, the rest skipped.
* **citm:** `citm_catalog.json` typed in full (dicts of dataclasses, lists of
  nested dataclasses).
* **gh:** 64 MB of GH Archive events (NDJSON), six fields of each event
  (`loads_many(data, type=T)`, one call per line for the others).

MB/s, one core of an Intel Xeon Gold 6548N, Python 3.14.10, best of 3 runs
of 7 (`python bench_typed.py simdjson-data/jsonexamples gharchive.ndjson`):

<picture>
  <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/simdjson/fastpysimdjson/main/doc/perf_typed_dark.png">
  <img src="https://raw.githubusercontent.com/simdjson/fastpysimdjson/main/doc/perf_typed.png" width="80%" alt="JSON to dataclasses on one core, GB/s. twitter.json, a few fields: fastsimdjson 2.59, msgspec Struct 1.53, msgspec dataclasses 1.43, pydantic 0.66, orjson with cattrs 0.57. citm_catalog.json typed in full: fastsimdjson 1.00, msgspec Struct 0.64, msgspec dataclasses 0.47, pydantic 0.19, orjson with cattrs 0.19. GH Archive NDJSON, 6 fields: fastsimdjson 2.05, msgspec Struct 1.29, msgspec dataclasses 1.16, pydantic 0.52, orjson with cattrs 0.42">
</picture>

| | twitter | citm | gh |
|---|---:|---:|---:|
| **fastsimdjson `loads(type=)`** | **2594** | **996** | **2055** |
| msgspec (Struct) | 1530 | 636 | 1295 |
| msgspec (dataclass) | 1431 | 470 | 1158 |
| pydantic 2.14 | 662 | 192 | 517 |
| orjson + cattrs | 567 | 187 | 420 |
| fastsimdjson `loads` (dicts, untyped) | 932 | 1066 | 432 |

fastsimdjson is 1.6 to 1.7 times as fast as msgspec with Structs, 1.8 to 2.1
times as fast with the same dataclasses, and 3.9 to 5.2 times as fast as
pydantic. When most of the document is not needed (twitter, gh), typed
decoding is also 2.8 to 4.8 times as fast as building the dicts.

On free-threaded Python 3.14.10t, `loads_many(data, type=T, threads=N)` read
the 886 MB GH Archive file into dataclasses at 1.82 GB/s on one thread,
12.6 GB/s on 8, 41.8 GB/s on 32 and 49.8 GB/s on 64.

### `dumps` against `json.dumps`

<picture>
  <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/simdjson/fastpysimdjson/main/doc/perf_dumps_dark.png">
  <img src="https://raw.githubusercontent.com/simdjson/fastpysimdjson/main/doc/perf_dumps.png" width="80%" alt="Writing JSON as str: fastsimdjson dumps 4.26 times faster than json.dumps">
</picture>

`dumps` returns exactly the `str` of `json.dumps` and is 4.3 times faster
(geometric mean over the 22 files; from 2.7 times on `citm_catalog.json` to
10.4 times on `canada.json`).

### `dumpb` against orjson and msgspec

`dumpb`, `orjson.dumps` and `msgspec.json.encode` produce compact UTF-8
`bytes`; `dumpb` and orjson produce identical bytes on every file. With the
standard library, the same compact bytes come from
`json.dumps(obj, separators=(",", ":"), ensure_ascii=False).encode()`.
Each call encodes a fresh copy of the document (`bench_dumps.py`), so no
encoder gains from what it cached on the objects at a previous call. One
core of an Intel Xeon Gold 6548N, Python 3.14.10, GCC 16.2, fastsimdjson
2.0.1, orjson 3.13.0, msgspec 0.22.0. Microseconds:

<picture>
  <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/simdjson/fastpysimdjson/main/doc/perf_dumpb_dark.png">
  <img src="https://raw.githubusercontent.com/simdjson/fastpysimdjson/main/doc/perf_dumpb.png" width="80%" alt="Writing JSON as bytes, speed relative to json.dumps(...).encode(): fastsimdjson dumpb 8.03x, orjson 7.39x, msgspec 5.26x">
</picture>

| file | `json.dumps(...).encode()` | fastsimdjson `dumpb` | orjson | msgspec |
|---|---:|---:|---:|---:|
| twitter | 1859 | 259 | 313 | 385 |
| citm_catalog | 3129 | 642 | 675 | 724 |
| github_events | 213 | 22 | 20 | 32 |
| gsoc-2018 | 12129 | 953 | 1273 | 2100 |
| update-center | 2679 | 328 | 320 | 532 |
| canada | 37695 | 2781 | 2478 | 3986 |
| mesh | 8962 | 875 | 1055 | 1405 |
| numbers | 2478 | 201 | 201 | 334 |

Over the 22 files, `dumpb` is 9% faster than orjson (geometric mean), from
11% slower on `canada.json` (mostly floating-point numbers) and up to 6%
slower on the smallest files (`twitter_api_response.json`,
`github_events.json`) to 34% faster on `gsoc-2018.json`, whose strings are
mostly not ASCII, and on `semanticscholar-corpus.json`. `dumpb` is
1.5 times faster than msgspec (orjson 1.4 times) and 8 times faster than
`json`. The comparison was run on a processor
with AVX-512, which `dumpb` and orjson both use to escape strings; other x64
processors use SSE2 and ARM processors NEON.

### `minify`

`minify` only removes whitespace. The other columns parse the document and
serialize it again, which also validates it and rewrites numbers. Geometric
mean over the 22 files, one core of an Intel Xeon Gold 6548N, Python 3.14.10,
GCC 16.2, orjson 3.13.0 (`bench_minify.py`):

<picture>
  <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/simdjson/fastpysimdjson/main/doc/perf_minify_dark.png">
  <img src="https://raw.githubusercontent.com/simdjson/fastpysimdjson/main/doc/perf_minify.png" width="80%" alt="Minifying JSON in Python: fastsimdjson minify 18.58 GB/s, fastsimdjson dumpb(loads(s)) 0.54, orjson dumps(loads(s)) 0.44, json dumps(loads(s)) 0.11">
</picture>

| method | GB/s |
|---|---:|
| **fastsimdjson `minify`** | **18.58** |
| `loads` + `dumpb` | 0.54 |
| orjson `loads` + `dumps` | 0.44 |
| `json.loads` + `json.dumps` | 0.11 |

Per file, `minify` runs from 15.7 GB/s (`repeat.json`) to 20.3 GB/s
(`numbers.json`).

### Writing NDJSON with `dumpb_many`

The 593,945 events of two hours of [GH Archive](https://www.gharchive.org/)
(1.87 GB of NDJSON), written back as NDJSON from Python objects. With the
GIL (Python 3.14.10), one thread: `orjson.dumps` on each record, joined, runs
at 0.93 GB/s, `dumpb` on each record at 0.93 GB/s, and `dumpb_many` at
1.49 GB/s. On free-threaded Python 3.14.10t, GB/s (up to 32 threads on the
cores of one socket, 64 on both):

| threads | 1 | 8 | 32 | 64 |
|---|---:|---:|---:|---:|
| `dumpb_many(records, threads=N)` | 1.45 | 5.6 | 11.2 | 10.3 |

### NDJSON to Arrow and back

Two files: 4 million flat records (611 MB, 9 scalar fields), and 304,929 GH
Archive events (886 MB), read with a schema of 6 of their fields (two of them
structs; the rest of each event is skipped), or inferring the schema. Reading
gives a `pyarrow.Table` (`pyarrow.table(fastsimdjson.loads_arrow(data,
schema, threads=N))`); the time includes the import into pyarrow. Writing
starts from that table: fastsimdjson `dumpb_arrow`, polars `write_ndjson` and
DuckDB `COPY ... TO` (on tmpfs). Python 3.14.10, pyarrow 26.0, polars 2.1,
DuckDB 2.0, fastsimdjson 2.0.1, Intel Xeon Gold 6548N (up to 32 threads on
the 32 cores of one socket; 64 threads on both sockets).

<picture>
  <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/simdjson/fastpysimdjson/main/doc/perf_arrow_read_1_dark.png">
  <img src="https://raw.githubusercontent.com/simdjson/fastpysimdjson/main/doc/perf_arrow_read_1.png" width="80%" alt="Reading NDJSON into a pyarrow.Table on one thread: flat, fastsimdjson 0.79 GB/s, pyarrow 0.18, polars 0.27; GH Archive, fastsimdjson 1.98, pyarrow 0.46, polars 0.54; flat with an inferred schema, fastsimdjson 0.78, pyarrow 0.18, polars 0.09">
</picture>

<picture>
  <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/simdjson/fastpysimdjson/main/doc/perf_arrow_read_32_dark.png">
  <img src="https://raw.githubusercontent.com/simdjson/fastpysimdjson/main/doc/perf_arrow_read_32.png" width="80%" alt="Reading NDJSON into a pyarrow.Table on 32 threads: flat, fastsimdjson 14.7 GB/s, pyarrow 5.2, polars 4.0; GH Archive, fastsimdjson 28.6, pyarrow 12.3, polars 3.9; flat with an inferred schema, fastsimdjson 14.0, pyarrow 5.0, polars 0.12">
</picture>

<picture>
  <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/simdjson/fastpysimdjson/main/doc/perf_arrow_write_1_dark.png">
  <img src="https://raw.githubusercontent.com/simdjson/fastpysimdjson/main/doc/perf_arrow_write_1.png" width="80%" alt="Writing a pyarrow.Table as NDJSON on one thread: flat, fastsimdjson 0.92 GB/s, polars 0.45, DuckDB 0.32; GH Archive, fastsimdjson 1.07, polars 0.46, DuckDB 0.32">
</picture>

<picture>
  <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/simdjson/fastpysimdjson/main/doc/perf_arrow_write_32_dark.png">
  <img src="https://raw.githubusercontent.com/simdjson/fastpysimdjson/main/doc/perf_arrow_write_32.png" width="80%" alt="Writing a pyarrow.Table as NDJSON on 32 threads: flat, fastsimdjson 9.5 GB/s, polars 1.5, DuckDB 4.9; GH Archive, fastsimdjson 4.6, polars 1.0, DuckDB 1.2">
</picture>

All the numbers, in GB/s:

| | threads | 1 | 8 | 32 | 64 |
|---|---|---:|---:|---:|---:|
| flat, read | fastsimdjson `loads_arrow` | **0.79** | **5.7** | **14.7** | **17.5** |
| | pyarrow `read_json` | 0.18 | 1.4 | 5.2 | 7.6 |
| | polars `read_ndjson` | 0.27 | 1.9 | 4.0 | 3.1 |
| flat, read, inferring the schema | fastsimdjson `loads_arrow` | **0.78** | **5.6** | **14.0** | **12.6** |
| | pyarrow `read_json` | 0.18 | 1.3 | 5.0 | 6.6 |
| | polars `read_ndjson` | 0.09 | 0.12 | 0.12 | 0.12 |
| GH Archive, read | fastsimdjson `loads_arrow` | **1.98** | **12.6** | **28.6** | **24.5** |
| | pyarrow `read_json` | 0.46 | 3.5 | 12.3 | 14.2 |
| | polars `read_ndjson` | 0.54 | 3.5 | 3.9 | 3.0 |
| flat, write | fastsimdjson `dumpb_arrow` | **0.92** | **3.6** | **9.5** | **11.3** |
| | polars | 0.45 | 1.1 | 1.5 | 1.4 |
| | DuckDB | 0.32 | 2.3 | 4.9 | 3.6 |
| GH Archive, write | fastsimdjson `dumpb_arrow` | **1.07** | **3.0** | **4.6** | **4.3** |
| | polars | 0.46 | 0.51 | 1.0 | 0.64 |
| | DuckDB | 0.32 | 1.2 | 1.2 | 0.99 |

`load_arrow_file` on the same files, from the page cache, reads at 0.70 and
1.5 GB/s on one thread, 9.1 and 12.6 GB/s on 32 threads. The benchmark is
`THREADS=N python bench_arrow.py flat.ndjson:flat gharchive.ndjson:gh
flat.ndjson:infer`; `path:infer` times any NDJSON file without a schema. The
flat file comes from
[simdjson_arrow's `benchmarks/gendata.py`](https://github.com/simdjson/simdjson_arrow/blob/main/benchmarks/gendata.py)
and the events from GH Archive
(`curl -sSfL https://data.gharchive.org/2024-03-13-12.json.gz | gunzip > gharchive.ndjson`).
The charts come from `python doc/make_charts.py arrow`.

### JSON to MessagePack and back

From JSON bytes to MessagePack bytes, and back, against the usual ways in
Python, which decode into Python objects, then encode them:
`ormsgpack.packb(orjson.loads(s))`, `msgspec.msgpack.encode(msgspec.json.decode(s))`
and so on. The 22 files of simdjson-data, geometric means of the speeds, in
GB/s of JSON (read when writing MessagePack, written when writing JSON).
Python 3.14.10, msgpack 1.2, orjson 3.13, ormsgpack 1.13, msgspec 0.22, Intel
Xeon Gold 6548N, one core.

<picture>
  <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/simdjson/fastpysimdjson/main/doc/perf_msgpack_encode_dark.png">
  <img src="https://raw.githubusercontent.com/simdjson/fastpysimdjson/main/doc/perf_msgpack_encode.png" width="80%" alt="JSON to MessagePack in Python, geometric mean over the 22 files of simdjson-data: fastsimdjson json_to_msgpack 1.67 GB/s, orjson with ormsgpack 0.46, msgspec 0.42, fastsimdjson loads with msgpack 0.34, orjson with msgpack 0.32, json with msgpack 0.17">
</picture>

<picture>
  <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/simdjson/fastpysimdjson/main/doc/perf_msgpack_decode_dark.png">
  <img src="https://raw.githubusercontent.com/simdjson/fastpysimdjson/main/doc/perf_msgpack_decode.png" width="80%" alt="MessagePack to JSON in Python, geometric mean over the 22 files of simdjson-data: fastsimdjson msgpack_to_json 1.43 GB/s, ormsgpack with orjson 0.34, msgspec 0.32, msgpack with fastsimdjson dumpb 0.26, msgpack with orjson 0.26, msgpack with json 0.10">
</picture>

Streams: the 304,929 events of an hour of GH Archive (886 MB of NDJSON), with
`json_to_msgpack(data, ndjson=True, threads=N)` and
`msgpack_to_json(packed, stream=True, threads=N)`, against orjson with
ormsgpack (each document) and `msgpack.Unpacker` with orjson (each value), on
one thread (one thread per core of one socket up to 32 threads, then both
sockets):

<picture>
  <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/simdjson/fastpysimdjson/main/doc/perf_msgpack_ndjson_dark.png">
  <img src="https://raw.githubusercontent.com/simdjson/fastpysimdjson/main/doc/perf_msgpack_ndjson.png" width="80%" alt="GH Archive NDJSON to MessagePack: fastsimdjson 22.2 GB/s on 64 threads, 16.1 on 32, 8.7 on 8, 1.32 on one thread, orjson with ormsgpack 0.36 on one thread; MessagePack to NDJSON: fastsimdjson 6.0 GB/s on 8 threads, 1.51 on one thread, msgpack with orjson 0.25">
</picture>

| threads | 1 | 2 | 4 | 8 | 16 | 32 | 64 |
|---|---:|---:|---:|---:|---:|---:|---:|
| NDJSON to MessagePack | 1.32 | 2.45 | 4.8 | 8.7 | 14.1 | 16.1 | 22.2 |
| MessagePack to NDJSON | 1.51 | 2.27 | 4.4 | 6.0 | 6.0 | 6.0 | 5.9 |

MessagePack to NDJSON stops scaling at about eight threads: one thread must
find where the values end (MessagePack has no line ends) before the others
can convert them. The benchmark is `python bench_msgpack.py` (the files of
`JSONDIR`, by default `simdjson-data/jsonexamples`) and
`python bench_msgpack.py --ndjson gharchive.ndjson 1 8 32`; the charts come
from `python doc/make_charts.py msgpack`.

### Streams with `loads_many`

20 MB of NDJSON (5268 objects and arrays, one per line), made from the same
files; best of three runs:

<picture>
  <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/simdjson/fastpysimdjson/main/doc/perf_ndjson_dark.png">
  <img src="https://raw.githubusercontent.com/simdjson/fastpysimdjson/main/doc/perf_ndjson.png" width="80%" alt="Reading NDJSON: fastsimdjson parse_many 1.14 GB/s, loads_many 0.40, msgspec decode_lines 0.36, fastsimdjson loads per line 0.34, orjson per line 0.27, msgspec decode per line 0.27, jiter per line 0.24, jiter without its string cache per line 0.21, json per line 0.12">
</picture>

| method | ms | GB/s |
|---|---:|---:|
| `json.loads` on each line | 163 | 0.12 |
| jiter on each line (no string cache) | 94 | 0.21 |
| jiter on each line (string cache) | 84 | 0.24 |
| msgspec `decode` on each line | 74 | 0.27 |
| orjson on each line | 74 | 0.27 |
| fastsimdjson `loads` on each line | 58 | 0.34 |
| msgspec `decode_lines` | 55 | 0.36 |
| fastsimdjson `loads_many` | 49 | 0.40 |
| fastsimdjson `parse_many` (views only) | 17 | 1.14 |

`loads_many` is 13% faster than msgspec's `decode_lines`. `parse_many` only creates the views; reading values from them adds to
its time.

### Streams on several threads

300 MB of NDJSON (146,624 records: tweets, GitHub events, citm performances,
gsoc and update-center entries), free-threaded Python 3.14.10t, collector
disabled (see below), Intel Xeon Gold 6548N (64 cores); GB/s:

| threads | 1 | 2 | 4 | 8 | 16 | 32 | 64 |
|---|---:|---:|---:|---:|---:|---:|---:|
| `map_many(len, data, threads=N)` | 0.91 | 1.42 | 2.33 | 4.17 | 6.16 | 10.0 | 17.6 |
| `parse_many(data, threads=N)` | 1.70 | 3.02 | 4.15 | 5.52 | 10.5 | 16.0 | **21.1** |
| `loads_many(data, threads=N)` | 0.76 | 1.55 | 2.60 | 4.37 | 7.57 | 10.9 | 10.8 |

`map_many` is 19 times faster on 64 threads than on one, and `parse_many` 12
times. `loads_many` builds every document for the calling thread and is 14
times faster. On 64 threads, these numbers vary by up to a third from one
run to the next. With the collector enabled,
Python collects after the call returns: it examines the millions of objects
just created, which takes about 0.1 s here, with or without threads.

## Limitations

* The simdjson parser and the key and string caches are thread-local.
  `release()` frees the parser and the cached strings retained by the calling
  thread and by the worker threads of the parallel streams, and they are
  freed when a Python thread exits. A parser that grows past 64 MB is freed
  on its own at the end of that call; its caches stay.
* On free-threaded Python, the worker threads of the parallel streams
  (`threads=N`) are started at the first call that needs them and kept until
  the interpreter exits: starting threads costs about as much as the parsing.
  Idle workers wait without using the processor. Each one keeps a parser
  (about a megabyte) and its caches until `release()`. Free-threaded CPython
  also keeps the memory of freed objects for reuse, in the thread that
  allocated it, rather than returning it to the system (the standard `json`
  module shows the same): after a large threaded call, the process stays
  larger, and the next calls reuse that memory.
* Subinterpreters are not supported: the module keeps per-process state (its
  types, and per-thread parsers and string caches).
