Immutable documents · Rust · Python · browser

Keep the documents.
Rebuild the view.

CouchDB-inspired map/reduce over an immutable document log. A Rust engine keeps the facts durable; every index and live view is a replaceable cache. Write the computation in AssemblyScript and run the same WebAssembly in Python or a browser worker.

Install

python -m pip install foldlitedb==0.1.0

Installing a matching native wheel needs neither Rust nor Node. To build from a checkout instead, install Rust/Cargo 1.90+ and a platform C compiler and linker, then run from the repository root:

python -m venv .venv
. .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install .
python examples/portable_metrics.py

On Windows, activate with .venv\Scripts\activate in an MSVC build environment. Node is needed only to author or rebuild views. To build a wheel, run python -m pip wheel . --no-deps --wheel-dir dist, then install the resulting .whl on a matching platform. Installing a built wheel needs neither Rust nor Node. See the release guide for PyPI Trusted Publisher setup and publication.

A native metric in a few lines

The bundled metrics module maintains count, sum, minimum, maximum, and mean by metric. Start with a fresh metrics.fold log:

from foldlitedb import Database, View

metrics = View.bundled("metrics")
with Database("metrics.fold", metrics) as db:
    documents = db.add_many([
        {"metric": "loss", "value": 4},
        {"metric": "loss", "value": 8},
    ])
    assert list(db.rows())[0].result == {
        "count": 2, "sum": 12, "min": 4, "max": 8, "mean": 6,
    }

Rust runs both map and reduce in a bounded Wasm interpreter. An accepted batch is appended and synced to the document log. A derived checkpoint lives separately in metrics.fold.cache; one active database owns the log through metrics.fold.lock. Database and View are the primary names; PortableDatabase and PortableView are aliases.

Append freely. Name when useful.

Each unnamed add(payload) creates a new observation. Supply a source-scoped ID when identical retries should count once. Documents are immutable; conflicting reuse raises DocumentConflict.

with Database("metrics.fold", metrics) as db:
    saved = db.add({"metric": "loss", "value": 3}, id="measurement-3")
    assert db.add(saved.payload, id="measurement-3") == saved
    documents += (saved,)

External sources can supply their own IDs and monotonically increasing positions. To represent an update, append another observation and let an ordered view fold its meaning.

Read once, follow changes

rows() follows bounded snapshot pages. query() also returns a cursor for later changes:

with Database("metrics.fold", metrics) as db:
    snapshot = db.query()
    db.add({"metric": "loss", "value": 1})
    updates = db.changes(since=snapshot.cursor)
    assert updates.rows[0].result["mean"] == 4
    db.checkpoint()

Validation, mapping, and reduction finish before a batch is committed. A view error leaves the current batch uncommitted. Checkpoints are optional accelerators, written on request or clean close.

For a multi-metric consumer, collect every snapshot and change page. The persistent consumer example also replaces its application state from a fresh snapshot when a cursor expires.

A cache can always be reconstructed

Keep the document log and exact view module; the cache can be discarded. Reopening verifies the complete log, reuses a valid checkpoint, and executes Wasm only for its uncached suffix. Missing or incompatible checkpoints rebuild from the original batches. Opening with a different module rebuilds its derived view from the same documents.

with Database("metrics.fold", metrics) as db:
    expected = [(r.key, r.state, r.result) for r in db.rows()]
    db.rebuild()
    assert [(r.key, r.state, r.result) for r in db.rows()] == expected

Rebuild starts a new cursor generation. Its cost is a linear log read plus map/reduce work for uncached batches; no unmeasured throughput is promised. Current indexes and view history grow in host memory. Back up the log: a disposable cache does not make the only source copy disposable.

The same module, entirely in a browser

Open the static playground, import JSON or JSONL, and watch groups update locally. A dedicated worker executes the same compiled module as Python; JavaScript owns browser grouping and in-memory storage. Export resolved documents before reloading or closing.

The browser example includes resource copying and a complete static page. Browser persistence, log/history compaction, and multiple native views per database remain future work.

Keep exploring

Run the telemetry collector for 2,500 readings, safe source retries, and cache-loss recovery, or the catalogue example for an ordered item lifecycle. Use preview(payload) to inspect a view's map output when a document produces no rows.