CouchDB-inspired views, from Python to the browser

Write a view once.
Run it in either environment.

A portable view contains both its map and its reducer. AssemblyScript compiles them into one WebAssembly module, executed by the Rust engine or a browser worker.

A native database from Python

Build prerequisites are Python 3.11+, Rust/Cargo 1.90+, and a platform C compiler and linker. From the repository root:

python -m pip install .
python examples/portable_metrics.py

The Rust engine and precompiled view modules are included. Node is needed only for authoring or rebuilding AssemblyScript. To produce a wheel locally, run python -m pip wheel . --no-deps --wheel-dir dist. Installing the resulting wheel on a matching platform needs no compiler. See environment setup for virtual-environment and Windows notes. Start the following example with fresh database files:

from foldlitedb import Database, View

view = View.bundled("metrics")
with Database("metrics.fold", view) as db:
    documents = db.add_many([
        {"metric": "loss", "value": 4},
        {"metric": "loss", "value": 8},
    ])
    assert list(db.rows())[0].result["mean"] == 6
    snapshot = db.query()
    db.add({"metric": "loss", "value": 3})
    assert db.changes(since=snapshot.cursor).rows[0].result["mean"] == 5
    retained = tuple(db.documents())

Rust owns the append-only document log, immutable identity, derived checkpoints, group states, snapshots, and changes. Its bounded Wasm interpreter executes both mapping and reduction. The log is independent of the module; a checkpoint is a disposable cache. rows() follows paginated snapshots; explicit query() and changes() support resumable reads. The Python reference covers every portable method.

Two bundled views

Metrics accepts payloads with a string metric and finite numeric value. Valid documents emit one number under their metric key; malformed shapes, missing fields, nulls, and nonnumeric values emit nothing. States contain {count,sum,min,max}; finalized results add mean. Empty states use null minimum, maximum, and mean. This view supports merge. Invalid contributions, states, and numeric overflow fail.

Catalogue accepts {category:string,id:string,title:string,deleted?:boolean}. Each category folds item IDs to their latest titles. An event with {category,id,deleted:true} removes that item. Finalization returns {count,items:{id:title}}. Event order matters, so this view does not support merge. Malformed documents emit nothing; item IDs are limited to 65,535 UTF-16 code units.

Author a complete view

Use Node.js 20+ and install the pinned SDK dependencies from the repository root:

npm ci --prefix views

For a custom module, save this complete source as views/assembly/examples/metric-totals.ts. It groups numeric values by metric and returns a running total. The SDK handles UTF-8 buffers, finite JSON, allocation, dispatch, and error envelopes.

import { JSON, View, Emission, object, numeric, number, dispatch } from "../sdk";
export { abi_version, input_ptr, input_capacity, output_ptr } from "../sdk";

class MetricTotals extends View {
  constructor() { super("metric-totals", false, "1", "1"); }

  map(document: JSON.Value): Emission[] {
    const doc = object(document);
    if (doc == null) return [];
    const metric = doc.get("metric");
    const value = doc.get("value");
    if (metric == null || metric.type != JSON.Types.String || !numeric(value)) {
      return [];
    }
    return [new Emission(metric, value!)];
  }

  identity(): JSON.Value { return JSON.Value.from<f64>(0); }

  add(state: JSON.Value, values: JSON.Arr): JSON.Value {
    if (!numeric(state)) return this.fail("Invalid total state");
    let total = number(state);
    for (let i = 0; i < values.length; i++) {
      const value = values.at(i);
      if (!numeric(value)) return this.fail("Expected a numeric contribution");
      total += number(value);
    }
    if (!isFinite(total)) return this.fail("Total overflowed");
    return JSON.Value.from(total);
  }

  finalize(state: JSON.Value): JSON.Value {
    if (!numeric(state)) return this.fail("Invalid total state");
    return state;
  }
}

const view = new MetricTotals();
export function execute(length: i32): i32 { return dispatch(view, length); }

From the repository root, compile the source and generate its digest manifest with one command:

npm run build:view --prefix views -- assembly/examples/metric-totals.ts --out-dir .build

Both path arguments are relative to views/, regardless of the caller's working directory; absolute paths also work. The output directory is created if needed. The source filename determines the outputs: views/.build/metric-totals.wasm and views/.build/metric-totals.json. Use letters, digits, underscores, dots, or hyphens in the source filename. The manifest records the compiled descriptor and SHA-256; its descriptor name may differ from the filename. Browser connect({name}) uses the artifact filename stem.

The command applies the same pinned compiler settings as the bundled build and ignores ambient JSON_* settings. It validates the ABI and descriptor before replacing successful outputs; compilation or validation errors leave existing artifacts intact. Choose a separate output directory such as .build; SDK, dependency, script, and bundled asset directories are protected. The ordinary npm run build --prefix views still builds only the two bundled examples.

From the repository root, load the same bytes from Python:

import json
from pathlib import Path
from foldlitedb import Database, View

manifest = json.loads(Path("views/.build/metric-totals.json").read_text())
custom = View("views/.build/metric-totals.wasm", sha256=manifest["sha256"])
with Database(":memory:", custom) as db:
    db.add_many([{"metric": "loss", "value": 4}, {"metric": "loss", "value": 8}])
    assert list(db.rows())[0].result == 12

The map receives payloads only, not host IDs or positions. Include domain metadata inside the payload if your computation needs it. Keys, states, and contributions can be structured JSON. State is passed explicitly to every update; requests must not depend on mutable module globals, clocks, randomness, or undeclared external data. Hosts instantiate the module afresh for operations.

add must preserve contribution order and produce equivalent state when an input sequence is split into successive batches. Implement merge and set mergeable=true only when compatible independent states can be combined correctly. The example leaves merge disabled. Bump the view version for changed behavior and the state version for incompatible formats; rebuild the module and manifest together. A new compiler can also change the module digest.

Use the module on a static website

The playground imports JSON arrays or JSONL, appends measurements, streams simulated updates, exports resolved records, and verifies replay. All document processing stays in its worker. The same module bytes power the Python example; JavaScript owns browser grouping and storage. The Rust document-log engine is not part of the browser build.

To create your own page, copy the installed package's browser files and bundled modules. Run this Python from the repository root:

from pathlib import Path
from foldlitedb import portable_assets

target = Path("browser-demo/portable")
target.mkdir(parents=True, exist_ok=True)
for resource in portable_assets().iterdir():
    if resource.is_file():
        (target / resource.name).write_bytes(resource.read_bytes())

Save this as browser-demo/index.html. Browser methods are asynchronous; addMany returns a batch-result object, while Python's add_many returns a tuple of documents.

<!doctype html>
<meta charset="utf-8">
<title>Foldlite DB in the browser</title>
<pre id="output">Loading…</pre>
<script type="module">
import { BrowserDatabase } from './portable/client.js';

const db = await BrowserDatabase.connect({name: 'metrics'});
try {
  await db.addMany([
    {metric: 'loss', value: 4},
    {metric: 'loss', value: 8},
  ]);
  const snapshot = await db.query();
  await db.add({metric: 'loss', value: 3});
  const updates = await db.changes(snapshot.cursor);
  const retained = await db.export(); // Save these records outside the worker.
  const replay = await db.replay();
  document.querySelector('#output').textContent = JSON.stringify({
    rows: await db.rows(), updates, documents: retained.length, replay,
  }, null, 2);
} finally {
  db.close();
}
</script>

python -m http.server 8000 --directory browser-demo

Open http://localhost:8000; opening the file directly is insufficient for module workers and fetch. Production deployments can serve these files over HTTPS. Copy the entire resource directory, including client.js, worker.js, host.js, and errors.js, so their module imports resolve.

For the custom totals view, copy views/.build/metric-totals.wasm and metric-totals.json into browser-demo/portable/, then change the connection to:

const db = await BrowserDatabase.connect({
  name: 'metric-totals',
  base: new URL('./portable/', import.meta.url),
});

The browser verifies the manifest's SHA-256 and descriptor before execution. db.info contains initial stats. Compare (await db.stats()).module_sha256 with Python's View.sha256 to confirm the exact artifact.

Every method below returns a Promise. A resolved document is {id,payload,position,source,sequence}; a batch result is {documents,inserted,replayed,sequence,cursor}. inserted and replayed are counts, and documents contains the resolved input records in order.

db.add(payload, {id?, source?, position?} = {}) → resolved document
db.addMany(payloads, {source?} = {}) → batch result
db.ingest(records) → batch result
db.preview(payload) → [{key,value}, ...]
db.export() → resolved document[]
db.stats() → {documents,sequence,cursor,module_sha256,descriptor,
                checkpoints,storage:"memory",memory_bytes}
db.replay() → {equivalent,documents,groups,module_sha256}

add accepts an optional identity; unnamed appends create new observations. addMany commits one bounded batch. ingest retains supplied IDs, sources, and positions; identical re-imports count as replays. Exported sequence fields describe progress in the current worker and are regenerated on import. Preserve id, payload, position, and source; compare keys, states, and results across databases rather than sequence fields or cursors.

preview maps one payload without appending documents, advancing progress, or running the reducer. Use it to diagnose an empty view: await db.preview({metric:'loss',value:4}) returns [{key:'loss',value:4}], while an incompatible payload may return []. Python exposes the same diagnostic as db.preview(payload). A successful preview checks mapping only; an actual append can still fail reduction or host limits.

query({limit,cursor,after,key}) returns {rows,cursor,after,complete}; continue with both returned cursor and after. Each row contains {key,state,result,sequence}. rows({limit,key}) collects all snapshot pages into an array. changes(cursor,{limit}) returns {rows,cursor,complete}; continue with its returned cursor until complete. replay() uses the original committed batches to compare keys, states, and results, reporting the boolean equivalent without changing current cursors.

Browser failures reject with DatabaseError or its subclasses DocumentConflict, SourceOrderError, and CursorExpired, exported from client.js. Each has a stable code and name matching its class name, plus a descriptive message. Use a class or code instead of parsing message text:

import { DocumentConflict } from './portable/client.js';

try {
  await db.add({metric: 'loss', value: 4}, {id: 'observation-1'});
} catch (error) {
  if (error instanceof DocumentConflict) {
    console.error(error.code, error.message);
  } else {
    throw error;
  }
}

The computation contract and limits

The v1 ABI uses UTF-8 JSON with a 256 KiB request limit and a separate 256 KiB response limit, both including envelopes. Modules have no host imports and at most 16 MiB of linear memory. The native host imposes a fuel budget. The browser client runs a dedicated worker with a 15-second default deadline, configurable through timeoutMs from 1 through 60,000 milliseconds. Expiry terminates the worker and loses its in-memory state.

Both hosts limit an ingestion batch to 1024 documents and 4096 emissions, and snapshot pages to 1024 entries. Encoded map and per-group add requests can reach the ABI byte limit earlier. The browser additionally caps retained documents at 20,000 and 16 MiB of encoded records, distinct groups at 4096, and history at 100,000 entries and 32 MiB of encoded rows. Its public request envelope is capped at 4 MiB. These are component limits, not a bound on the whole browser process. A failing batch does not commit a prefix.

Portable JSON numbers use IEEE-754 double precision, as in JavaScript. Python rejects integer inputs that cannot be represented exactly as doubles; use strings for exact large identifiers. Positions and cursor sequences are nonnegative JavaScript-safe integers. Floating-point sums preserve ordered inputs; arbitrary merge trees can change rounding. See numerical reproducibility.

Retain documents; regenerate views

Keep the document log and exact compiled module. A persistent metrics.fold log has separate metrics.fold.cache and metrics.fold.lock siblings. One active owner holds the log. Accepted immutable batches are synced before acknowledgement; clean close or checkpoint() writes an atomic derived snapshot.

Reopen verifies the complete log, reuses a matching cache prefix, and executes Wasm for only the uncached suffix. Missing, corrupt, or different-module caches rebuild from the original batches. To change computation, open the same log with the new module; documents remain unchanged. A human version label cannot override a changed digest. Keep the previous module if its previous result must remain reproducible.

rebuild() explicitly recomputes the current view, checkpoints it, and starts a new cursor generation. Reconstruction is a linear log read plus map/reduce work for uncached batches; current indexes and history grow in host memory. Cache acceleration avoids computation, rather than the full validation read, and has no unmeasured throughput guarantee.

with Database("metrics.fold", view) as live:
    expected = [(row.key, row.state, row.result) for row in live.rows()]
    live.rebuild()
    assert [(row.key, row.state, row.result) for row in live.rows()] == expected

To copy resolved documents into a separate log, preserve their original order:

with Database("copy.fold", view) as rebuilt:
    rebuilt.ingest(retained)
    assert list(rebuilt.rows())[0].result["mean"] == 5

For larger streams, iterate db.ingest_many(records, batch_size=100); each yielded result commits one bounded batch. db.documents(limit=100) exports resolved records at a pinned snapshot. Browser exports include an additional engine sequence field; Python accepts them after constructing Document objects:

import json
from pathlib import Path
from foldlitedb import Document, Database, View

view = View.bundled("metrics")
records = json.loads(Path("exported-documents.json").read_text(), parse_int=float)
documents = [Document(r["id"], r["payload"], int(r["position"]), r["source"]) for r in records]
with Database("from-browser.fold", view) as db:
    for result in db.ingest_many(documents):
        print(result.inserted, result.replayed)

The final snippet expects an exported JSON array from the metrics view. Browser payload numbers use IEEE-754 doubles; parse_int=float preserves those semantics even when a large double is serialized without a decimal point. Safe source positions are converted back to Python integers when constructing documents. Preserve input order; preserve batch boundaries too if exact engine sequence values matter. Compare computation outputs rather than database-local cursors. A new database or rebuilt cache receives a new generation, so obtain new cursors even when results match. A valid checkpoint preserves cursors across an ordinary reopen.

Native documents persist in the log; a cache can always be reconstructed. Back up the log or an authoritative external stream: deleting the only source copy loses data. Browser state is in memory and is discarded on reload or close(). Export before closing and ingest those records into a new worker to restore it. Persistent browser storage, log/history compaction, and multiple native views per database remain future extensions. See the document and cache model.