Reference · development API
Database uses one Rust engine for document acceptance, source order, durable commits, and replay. View identifies a portable Wasm computation or a versioned Python callback projection.
View(module, *, sha256=None)
View.bundled(name="metrics") → View
portable_assets() → importlib.resources Traversable
Import these names from foldlitedb. module accepts Wasm bytes or a filesystem path. A View stores immutable module bytes and their sha256; an optional digest must match. Modules range from 1 byte through 16 MiB. The native database validates exports, imports, ABI, and descriptor when opening.
bundled accepts "metrics" or "catalogue" and verifies the manifest digest. Metrics returns {count,sum,min,max,mean}; catalogue returns {count,items}. See accepted payloads. portable_assets() exposes modules, manifests, and browser hosts; copy files with each resource's read_bytes() method.
PortableView is an exact alias of View. It does not select another engine.
View(name, *, version="1", map=..., reducer=..., references=None)
OrderedFold(initial, step, *, version, finalize=None)
RustReducer(name, *, runtime=None, positioned=False)
db.register(view) → view version
Callbacks receive resolved Document objects, including the host-assigned ID and source position. A map yields ordered (key, contribution) pairs. The reducer provides identity, add, and finalize; OrderedFold applies a Python step in accepted ingestion order. A callback batch can emit at most 4096 contributions. Keys, contributions, state, and results must be finite portable JSON. Callback code runs as trusted Python, without Wasm's execution or memory isolation, and must be deterministic and free of database mutations.
Every append API uses native acceptance and deduplication. Maps, reducers, and finalizers stage their complete batch before its source documents are durably committed. Conflicting input, a failing callback, or invalid output commits nothing. Identical retries do not map again. Python view operations are synchronized, and returned JSON is copied.
from foldlitedb import Database, Document, OrderedFold, View
totals = View(
"totals", version="1",
map=lambda document: [("total", document.payload["value"])],
reducer=OrderedFold(0, lambda state, value: state + value, version="1"),
)
with Database("callbacks.fold", totals) as callback_db:
callback_db.add({"value": 2}, id="first")
observation = Document("second", {"value": 3})
assert callback_db.ingest([observation]).inserted == 1
assert callback_db.ingest([observation]).replayed == 1
assert next(callback_db.rows()).result == 5
with Database("callbacks.fold", totals) as callback_db:
assert callback_db.query().rows[0].result == 5
callback_db.rebuild()
assert callback_db.query("totals").rows[0].result == 5
Callback cache coverage and descriptors live in the disposable native checkpoint. Reopen restores matching groups and replays any uncached original batches. Missing caches and changed descriptors reconstruct from retained documents; rebuild() also recomputes registered callback views. Callback code and captured reference changes require a new view/reducer version or changed references. Retain the Python computation and its inputs when preserving reproducibility.
Passing a callback view to Database makes it the primary query()/rows() view. Additional callbacks registered with register are queried by name. Snapshot pages use descriptor-bound cursors; historical pages reconstruct from the accepted batches and can cost a replay. Primary callback views explicitly reject changes() and preview(); use a fresh query snapshot or call the mapper directly on a resolved document. Portable Wasm views retain their change feeds and preview operations.
foldlitedb.projection.materialize_jsonl_view is an external-source accelerator: the external stream remains authoritative. It commits its consumed cursor and derived state together, binding both to the source generation and view/reducer/reference descriptor. A descriptor change rereads the external source instead of reusing its previous consumed watermark. Historical external projection snapshots are unavailable; obtain a fresh snapshot. This helper does not retain the external document history in its compact cache.
Document(id, payload, position=None, source="default")
Import Document from foldlitedb to construct records for ingest or ingest_many. Construction alone writes nothing. id is a stable, nonempty string naming one observation within its source; payload is its finite JSON value. The same ID in another source names a different observation.
Supply position when the upstream source already has ordered offsets. New positions must advance that source, with gaps allowed. If the source has stable IDs but no offsets, omit position: ingestion assigns one automatically. An identical named retry with position=None reuses the original position, even after newer records arrive. Input objects are not mutated; use documents() to export their assigned positions. Records returned by add, add_many, or documents always have resolved integer positions.
The fields are id, payload, position, and source. Returned payloads are independent copies; changing a copy cannot update a stored document. Reusing an ID or position with conflicting content raises DocumentConflict. Append a new named observation to express a change.
Source names beginning with _ are reserved for metadata. They are retained, exported and replayed, and advance source progress, but do not contribute to views in native or browser execution.
Database(path, view=None, *, generation=None, reset=False)
Database.migrate(source, destination)
A Rust-native database executing one primary Wasm or Python callback view. view accepts a View, Wasm bytes, or a module path; omit it for bundled metrics. path accepts a filesystem path or ":memory:". Parent directories must already exist. A persistent path is an append-only document log, with .cache and .lock siblings. Exactly one active owner may open it.
Use a context manager or close(). Clean close checkpoints persistent derived state and releases ownership. Closing is idempotent; later operations raise DatabaseError. PortableDatabase is an exact alias.
Reopen validates the complete log and reuses a compatible cache, then replays any uncached suffix. A missing or invalid cache rebuilds. Different module bytes automatically rebuild the view from the same documents. Reconstruction creates a new cursor generation; a valid checkpoint preserves generation across reopen. Old external-database files are not this log format.
New databases use FOLDLOG2 frames with persisted predecessor hashes. Existing FOLDLOG1 logs remain readable and exportable but reject new appends. Close their owner, then call Database.migrate("old.fold", "new.fold") to verify and copy their original batches into a new log. The source is unchanged; an existing destination is never replaced. Verify the new log with the intended view before retiring the source backup. Stats report log.format and log.read_only.
The upgrade guide walks through backup, source inspection, migration, verification, cutover, rollback, and application compatibility checks.
Primary filenames ending in .cache or .lock, or containing .cache., are reserved for sidecars. Conflicting source logs or symbolic links at auxiliary paths cause an error. Unrecognized cache bytes are preserved under a unique .cache.unrecognized.* filename when replaced by a valid checkpoint.
reset=True is explicitly destructive. Native ownership locks remain held throughout reset; an active owner prevents it. Invalid modules are rejected before reset changes data. Reset I/O errors are reported and may leave an uncertain outcome. generation is an optional identity for an external source generation; a mismatch raises GenerationMismatch.
from foldlitedb import Database, View
with Database(":memory:", View.bundled("metrics")) as db:
original = db.add({"metric": "loss", "value": 4}, id="a")
assert db.add(original.payload, id="a") == original
snapshot = db.query()
db.add({"metric": "loss", "value": 8})
assert db.changes(since=snapshot.cursor).rows[0].result["mean"] == 6
retained = tuple(db.documents())
assert db.stats()["documents"] == 2
with Database(":memory:") as replay:
assert replay.ingest(retained).inserted == 2
assert replay.ingest(retained).replayed == 2
db.add(payload, *, id=None, source="default", position=None) → Document
db.add_many(payloads, *, source="default") → tuple[Document, ...]
db.ingest(documents) → IngestResult
db.ingest_many(documents, *, batch_size=100) → iterator[IngestResult]
add appends one finite JSON payload and returns its resolved Document(id, payload, position, source). Omitted IDs are generated; omitted positions start at zero and advance per source. An identical named add or replay at an existing source position reuses the original document. Conflicting reuse fails; previously unseen positions must advance that source. IDs and sources are nonempty strings of at most 4096 UTF-8 bytes; positions range from zero through 2**53 - 1.
add_many eagerly consumes one iterable of payloads and commits atomically. Its returned records preserve input order. Every member is one payload, including an entire array; empty input returns (). Retrying unnamed payloads appends new observations. Documents are immutable; append a new observation to express a change.
ingest takes Document instances with stable identities, payloads, and optional upstream positions. Its result contains inserted, replayed, and sequence. Identical replay performs no map work.
ingest_many is lazy: you must iterate its results to write anything. Each yielded result commits a separate batch of 1–1024 records, defaulting to 100. A later error preserves earlier committed batches; the complete import is not one atomic transaction.
An atomic batch has at most 1024 documents and 4096 emissions. The complete map request, including its envelope and newly inserted payloads, must fit 256 KiB; per-group add requests, states, and results must fit their own buffers. The log frame limit is 8 MiB. Reduce batch size for larger payloads. Validation, map, reducer, and limit errors leave the current batch uncommitted; successful persistent batches are synced before acknowledgement.
Finite JSON uses IEEE-754 doubles and at most 64 nesting levels. Python and the native engine reject integer inputs not exactly representable as doubles, such as 2**53 + 1, rather than rounding them; 2**53 + 2 is accepted. Accepted integers above 2**53 - 1 are returned as floats. Nonfinite values and invalid Unicode scalar strings fail. Use strings for arbitrary-precision identifiers.
If your telemetry export has offsets, retain them along with the source and stable reading IDs. Fully iterating the import below commits 25 batches; resending an old batch counts no readings twice:
from foldlitedb import Database, Document
source_records = [
{"id": f"gateway-07:{offset}", "offset": offset,
"metric": "temperature.c", "value": 20 + (offset % 20) * 0.25}
for offset in range(2500)
]
records = [Document(r["id"], {"metric": r["metric"], "value": r["value"]},
position=r["offset"], source="gateway-07")
for r in source_records]
with Database(":memory:") as imported:
results = list(imported.ingest_many(records, batch_size=100))
assert sum(result.inserted for result in results) == 2500
retry = imported.ingest(records[100:200])
assert retry.inserted == 0 and retry.replayed == 100
assert list(imported.rows())[0].result["count"] == 2500
If upstream provides IDs but no offsets, use named records with automatic positions instead. Do not invent new offsets from each export's row numbers:
named = [
Document("probe-a:reading-1", {"metric": "temperature.c", "value": 20}, source="probe-a"),
Document("probe-a:reading-2", {"metric": "temperature.c", "value": 21}, source="probe-a"),
]
with Database(":memory:") as automatic:
assert sum(r.inserted for r in automatic.ingest_many(named)) == 2
automatic.add({"metric": "temperature.c", "value": 22}, id="probe-a:reading-3", source="probe-a")
assert automatic.ingest(named).replayed == 2
assert [record.position for record in automatic.documents()] == [0, 1, 2]
For JSONL, construct each Document while iterating the file and pass that iterator to ingest_many. Keep source order, and retain resolved records when exporting. The complete telemetry example verifies summaries against a Python oracle, follows live changes, and reopens after deleting the cache.
db.preview(payload) → list[{"key": JSON, "value": JSON}]
preview runs only the current view's map function without appending a document or changing source progress, rows, or history. It does not run the reducer or guarantee that a later ingestion batch fits its limits. Use it to inspect unexpected zero-emission payloads:
with Database(":memory:") as inspection:
assert inspection.preview({"metric": "temperature.c", "value": 21}) == [
{"key": "temperature.c", "value": 21},
]
assert inspection.preview({"kind": "heartbeat"}) == []
assert inspection.stats()["documents"] == 0
Zero emissions are intentional filtering, not a rejection: ingestion still retains a valid document and advances its source progress. The bundled metrics view ignores shapes without a string metric and finite numeric value. Validate your application schema before ingestion if every retained document should contribute a reading.
db.rows(*, key=<omitted>, start_key=<omitted>, end_key=<omitted>,
prefix=<omitted>, inclusive_start=True, inclusive_end=True, limit=100) → iterator[Row]
db.query(*, key=<omitted>, start_key=<omitted>, end_key=<omitted>,
prefix=<omitted>, inclusive_start=True, inclusive_end=True,
limit=100, cursor=None, after=None) → Snapshot
db.changes(*, since, limit=100) → ChangePage
rows lazily follows snapshot pages, pinning the committed prefix on first iteration. limit is a page size from 1 through 1024, not a total row limit. Omit all selectors for every group or supply any JSON key, including null, for exact filtering. No operation lock spans a yield into application code.
A Row has key, reducer state, finalized result, and committed sequence. A Snapshot has rows, cursor, after, and complete. Continue with both returned cursor and after token, preserving every selector and inclusive flag. Page size may change. Changed selectors, another snapshot, and legacy query tokens fail explicitly. New ingestion does not change the pinned snapshot.
with Database(":memory:") as db:
db.add_many([
{"metric": "a", "value": 2},
{"metric": "b", "value": 8},
])
page = db.query(limit=1)
snapshot_cursor = page.cursor
rows = list(page.rows)
while not page.complete:
page = db.query(limit=1, cursor=page.cursor, after=page.after)
rows.extend(page.rows)
assert [row.key for row in rows] == ["a", "b"]
db.add({"metric": "a", "value": 4})
changes = db.changes(since=snapshot_cursor, limit=1)
assert changes.rows[0].result["mean"] == 3
while not changes.complete:
changes = db.changes(since=changes.cursor, limit=1)
Use exactly one selector mode: an exact key, a range with either or both of start_key/end_key, or a prefix. Combining modes raises DatabaseError. Range bounds are inclusive by default; set inclusive_start=False or inclusive_end=False to exclude an endpoint. These options must be booleans. Omitted bounds are open; an explicit None is a JSON null bound. If the start sorts after the end, the result is empty. Traversal is ascending.
A string prefix matches strings beginning with its characters. An array prefix matches arrays beginning with the same complete elements, compared by JSON structure rather than text. It includes an array equal to the prefix. Nested elements must match completely: ["sales", [2026]] does not prefix-match ["sales", [2026, 11]]. Empty string and array prefixes match all strings and arrays respectively. Other prefix types are invalid. All views, including named Python callbacks, use these selectors; for example db.query("sales", prefix=["alice"]).
Key order is null < false < true < numbers < strings < arrays < objects. Numbers compare numerically, so 2 precedes 10. Strings compare by Unicode codepoint, independently of locale; this differs from CouchDB's ICU ordering. Arrays compare their elements recursively, then shorter arrays first when one is a complete prefix. Objects compare codepoint-sorted member names and each corresponding value recursively, then shorter equal member sequences first. Member insertion order does not affect equality or ordering.
from foldlitedb import OrderedFold, View
sales = View(
"sales", version="1",
map=lambda document: [(
[document.payload["customer"], document.payload["day"]],
document.payload["amount"],
)],
reducer=OrderedFold(0, lambda total, amount: total + amount, version="1"),
)
with Database(":memory:", sales) as db:
db.add_many([
{"customer": "alice", "day": "2026-11-01", "amount": 10},
{"customer": "alice", "day": "2026-11-02", "amount": 20},
{"customer": "bob", "day": "2026-11-02", "amount": 30},
])
assert [row.result for row in db.rows(prefix=["alice"], limit=1)] == [10, 20]
selected = db.query(start_key=["alice", "2026-11-01"],
end_key=["alice", "2026-11-02"], inclusive_start=False)
assert [(row.key, row.result) for row in selected.rows] == [
(["alice", "2026-11-02"], 20),
]
Range queries select existing per-key state and results; they do not re-reduce the selected keys, return raw map emissions, or implement CouchDB group_level. Include the dimensions you need in emitted keys. The native host seeks a derived ordered index; browser and callback hosts sort their in-memory groups. The document log and ingestion sequence order are unchanged.
changes returns committed group postimages after a snapshot or change cursor. since=0 reads retained history from the start. A page has rows, continuation cursor, and complete; continue with that cursor. Repeated changes to a key in one batch coalesce. A page can split group updates from one batch. Zero-emission documents still advance progress.
Cursors bind to the database generation and exact module. Checkpointed reopen can reuse them; rebuild, cache reconstruction, or module replacement expires them. A new database or browser worker cannot reuse another's cursor. There is no history compaction API.
Save a consumer's cursor together with the application state it describes. Change rows contain whole resulting group states, so replace each key's result. If the cursor expires, collect a fresh stable snapshot and replace the entire application state. Replaying from since=0 onto existing totals would double count.
from pathlib import Path
from tempfile import TemporaryDirectory
from foldlitedb import CursorExpired
def collect_snapshot(db):
page = db.query(limit=1)
state, cursor = {}, page.cursor
while True:
state.update((row.key, row.result) for row in page.rows)
if page.complete:
return state, cursor
page = db.query(limit=1, cursor=page.cursor, after=page.after)
def poll_changes(db, state, cursor):
try:
while True:
page = db.changes(since=cursor, limit=1)
state.update((row.key, row.result) for row in page.rows)
cursor = page.cursor
if page.complete:
return state, cursor
except CursorExpired:
return collect_snapshot(db) # New dict replaces all old application state.
with TemporaryDirectory() as directory:
path = Path(directory) / "consumer.fold"
with Database(path) as db:
db.add_many([{"metric": "cpu.percent", "value": 20},
{"metric": "temperature.c", "value": 21}])
state, cursor = collect_snapshot(db)
db.add({"metric": "temperature.c", "value": 23})
state, cursor = poll_changes(db, state, cursor)
assert state["temperature.c"]["mean"] == 22
with Database(path) as db: # Clean close kept this cursor valid.
assert db.changes(since=cursor).rows == ()
Path(str(path) + ".cache").unlink()
with Database(path) as db:
previous_state = state
state, cursor = poll_changes(db, state, cursor)
assert state is not previous_state
assert state == {row.key: row.result for row in db.rows()}
The runnable telemetry collector also persists its consumer checkpoint and proves that cache reconstruction preserves every resolved source record and summary.
db.checkpoint() → dict
db.rebuild() → dict
db.documents(*, limit=100) → iterator[Document]
db.stats() → dict
db.close()
checkpoint atomically persists the current derived snapshot and returns stats plus checkpointed: true. Clean close also checkpoints. The cache is optional and is not rewritten after every append. Encoded snapshots are capped at 1 GiB. Checkpoint failures are reported, including during close; the synced source log remains intact and close releases ownership even on failure. rebuild replays original log batches through the current module, starts a new cursor generation, writes a checkpoint, and returns stats plus rebuilt: true. Documents remain unchanged.
with Database("maintenance.fold") as db:
db.add_many([{"metric": "loss", "value": 4}, {"metric": "loss", "value": 8}])
expected = [(row.key, row.state, row.result) for row in db.rows()]
assert db.checkpoint()["checkpointed"] is True
assert db.rebuild()["rebuilt"] is True
assert [(row.key, row.state, row.result) for row in db.rows()] == expected
documents exports resolved records in original ingestion order at a snapshot pinned on first iteration. Page size is 1–1024. Back up these records or the log and retain exact view artifacts; the cache can be regenerated. Re-import through ingest or ingest_many.
stats includes documents, groups, sequence, cursor, generation, module_sha256, descriptor, and source checkpoints. storage is "append-only-log" or "memory". The cache object reports status, bytes, and log_offset; replay reports processed batches and documents; log reports bytes, batches, recovered_tail_bytes, and writer_poisoned, which is true after an I/O error until the log is reopened.
Engine sequence counts inserted documents and changed group states, rather than documents alone. Reopening always verifies the complete log, even with a valid cache. Current retained documents, indexes, and view history grow in host memory; there is no compaction or bounded total-memory guarantee.
cache.status describes startup or the most recent checkpoint operation, not whether the in-memory view is usable. For example, status="missing", bytes=0, and replay.documents=2513 is healthy after rebuilding from a missing cache. The status stays "missing" until checkpoint() or clean close writes a cache. cache.bytes is the cache file size inspected or last written, not the memory used by the view; cache.log_offset is the reused or checkpointed log prefix.
| Status | Meaning |
|---|---|
memory | An in-memory database has no cache file. |
missing | No cache was found on open; the log was replayed. |
invalid | A cache could not be validated or did not match the log; the log was replayed. |
module-mismatch | The cached view used different module bytes; the log was replayed through the current module. |
loaded | A valid cached prefix was reused, with any remaining log suffix replayed. |
checkpointed | A new persistent cache was written by checkpoint() or rebuild(). |
DatabaseError extends ValueError. Validation and Wasm failures raise it; conflicting replay, source ordering, and invalid cursors use DocumentConflict, SourceOrderError, and CursorExpired where identified. A missing compiled extension raises an actionable RuntimeError.
Opening an already-owned log fails. Complete log corruption fails rather than skipping records; only an incomplete final frame is recoverable as an interrupted append. A cache failure can be repaired by reconstruction. If an I/O error makes an append's acknowledgement uncertain, the writer is poisoned: later appends and checkpoints fail, reads still serve the last acknowledged state, and stats()["log"]["writer_poisoned"] is true. Close (which reports the failed checkpoint but releases the log), reopen, and retry using explicit resolved identities; an append that did reach the log becomes an identical replay. Keep backups of source data.
Rust applications can depend on the foldlitedb-portable-engine crate directly. Engine::open(path, module_bytes) opens a log or ":memory:", and Engine::execute takes the same JSON requests as the Python binding (ingest, query, changes, documents, get, stats, checkpoint, rebuild, close). The crate root exports its limits so callers can split work before submitting it:
| Constant | Value | Bounds |
|---|---|---|
ABI_BUFFER_BYTES | 262,144 | Each Wasm request and, separately, each response, including envelopes. A batch's one map request holds all newly inserted payloads; a per-group add request holds the previous state and that group's new values. |
MAX_BATCH_DOCUMENTS | 1024 | Documents in one atomic ingestion batch. |
MAX_BATCH_EMISSIONS | 4096 | Map contributions produced by one batch. |
MAX_LOG_FRAME_BYTES | 8 MiB | One encoded log frame (one committed batch). |
MAX_JSON_DEPTH | 64 | Nesting of payloads, keys, states, and results. |
MAX_SAFE_INTEGER | 253 − 1 | Positions and sequences. Integers above it that a double represents exactly are accepted but read back as floats, so keep integer-typed domain values within it. |
Limit errors are host-neutral: they name the operation, the encoded size, and the limit, for example Wasm map request is 300123 bytes, which exceeds the 262144-byte ABI limit (ABI_BUFFER_BYTES). Ingestion rejects any payload integer a double cannot represent exactly, naming the value, before mapping or writing.
Engine::writer_poisoned() is a cheap, read-only health check that is true once an I/O error has made a durable append uncertain; Engine::is_writable() is true while the engine is open and not poisoned. To recover, execute close or drop the engine (both release the lock even if the final checkpoint fails), call Engine::open again, which verifies the log and discards an interrupted tail, and retry with explicit document ids or source positions.