Upgrade guide · 0.3.2

Upgrade to Foldlite DB 0.3.2

Version 0.3.2 upgrades the native Wasm interpreter for faster execution. The Python API, document log, cache format, bundled modules, and query continuations are compatible with 0.3.1. Existing 0.2.0 logs need no migration; older 0.1.x logs require the verified copy migration below.

The consolidated changelog covers every published release. The 0.2.0 release includes distribution hashes and validation receipts.

Upgrade from 0.3.1

Close the database, preserve its log and exact view artifacts, and install foldlitedb==0.3.2. Open the same log with the same view and compare results. Compatible caches retain their generation and cursors; no log migration or module rebuild is required. Rollback to 0.3.1 uses the same files after closing the new owner.

Native execution now uses Wasmi 2.0 stable fuel accounting. Custom reducer fuel values count Wasm operations differently, so applications overriding that budget should validate representative requests again. Exhausted instances still require replacement. Python callbacks and browser execution use their existing computation paths.

Upgrade from 0.2.0

  1. Close the database and preserve a backup of the document log and exact view artifacts, as for any package upgrade.
  2. Install foldlitedb==0.3.2 in a separate environment and open the same log with the same view. FOLDLOG2 is unchanged. Existing compatible caches restore their generation and reconstruct the query ordering index without rerunning checkpointed Wasm.
  3. Start fresh query traversals. 0.2.x after query tokens are rejected explicitly because their ordering and filter binding differ. Saved snapshot/change cursors remain valid when the compatible cache survives; cache reconstruction, module replacement, or explicit rebuild still expires them. Document pagination is unchanged.
  4. Check application assumptions about row order. Numbers now sort numerically, strings by Unicode codepoint, arrays by recursive element order, and objects by their ordered members. Query ordering changes do not reorder documents, reducer contributions, or change-feed sequences. Compare results by key if an older test assumed serialized-JSON ordering.

Use the same selector and inclusive flags on every query page; only limit may change. See the range and prefix reference. Browser deployments must update the complete host assets together and obtain new worker cursors. The portable module ABI and bundled Wasm bytes are unchanged.

For rollback to 0.2.0, close the new owner and use a fresh 0.2.0 query traversal; do not reuse 0.3.1 query tokens. The unchanged log format remains readable. Preserve the original backup and computation, and account for any writes made during the upgrade.

Before you start

The remaining steps upgrade native FOLDLOG1 document logs from 0.1.x to FOLDLOG2. New 0.2.0 and later databases already use the new format. Browser databases have no native log to migrate; see browser upgrades. Compact external projection caches have a different source of truth; see external projections.

Use Python 3.11 or newer. Preserve the exact Wasm module, or Python callback code, versions, references, and other reproducible inputs needed by your view. Migration copies facts and batch boundaries; it does not change your computation or repair already incorrect application data. Plan new snapshot cursors for the migrated database.

1. Stop writers and back up the source

  1. Stop producers, background imports, and consumers that hold the database open. Close its owner cleanly before copying files.
  2. Copy the original document log to a separate backup location. Keep the view artifacts and reference inputs with that backup. Confirm the copied log exists and has the expected size or checksum.
  3. Keep the original filename in place for comparison. Choose a different destination, such as metrics-v2.fold.

If the old primary filename uses the newly reserved sidecar namespace, make an independent closed-log copy under a permitted name such as metrics-original.fold and use that copy as the source in steps 3–5. Keep the untouched original as well. This lets the new reader inspect and compare it without bypassing filename protection.

The log contains the retained documents. Its .cache sibling is disposable; copying a cache alone is not a backup. Do not delete a live owner's .lock file. Do not use reset=True to upgrade: reset explicitly discards source documents.

2. Install 0.3.2 in a separate environment

python -m venv .venv-upgrade
. .venv-upgrade/bin/activate
python -m pip install --only-binary=:all: foldlitedb==0.3.2
python -c "from importlib.metadata import version; print(version('foldlitedb'))"

On Windows Command Prompt, activate with .venv-upgrade\Scripts\activate.bat. Matching wheels need no compiler. For other source-build environments, follow the installation prerequisites. Keep the previous environment available until cutover is complete.

3. Inspect the closed log with its intended view

The Python snippets in steps 3–5 run in order in the same session or script, from the directory containing your log. This example uses bundled metrics. For a custom portable view, replace View.bundled("metrics") with your preserved module, for example View("metrics-original.wasm"). For ordinary callback views, pass your preserved callback definition.

from pathlib import Path
from foldlitedb import Database, View

source = Path("metrics.fold")
if not source.is_file():
    raise FileNotFoundError(source)  # Avoid creating an empty database by mistake.

view = View.bundled("metrics")
with Database(source, view) as original:
    log = original.stats()["log"]
    print({"format": log["format"], "read_only": log["read_only"]})

A legacy log reports format: 1 and read_only: True. Reads and export work; new appends fail with migration guidance. If it reports format 2, skip the copy migration: it already uses the current format. Opening may recover an interrupted final frame, which is why the raw closed-log backup comes first.

4. Copy into a new log

destination = Path("metrics-v2.fold")
if destination.exists():
    raise FileExistsError(destination)

Database.migrate(source, destination)

Both paths must be persistent files in existing parent directories. The destination must not exist and must differ from the source. Primary filenames ending in .cache or .lock, or containing .cache., are reserved case-insensitively; metrics-v2.fold is suitable.

Migration verifies the source, preserves document IDs, sources, positions, order, and original batch boundaries, and leaves source bytes unchanged. It refuses an active owner or an existing destination. If copying fails, it removes its partial destination when possible and reports cleanup failures; inspect any remaining destination before retrying. It does not copy old cursor generations or transplant caches. Ordinary opens reject reserved legacy primary names; explicit migration can copy such an old source into a permitted destination.

5. Compare documents, results, and rebuilt results

With writers still stopped, compare resolved documents and view output under the same computation. The verifier streams through bounded pages and excludes database-local cursors and row sequence metadata from the comparison.

from itertools import zip_longest

def require_equal(left, right, label):
    missing = object()
    for number, (a, b) in enumerate(zip_longest(left, right, fillvalue=missing), 1):
        if a != b:
            raise RuntimeError(f"{label} differs at entry {number}")

def view_values(database):
    for row in database.rows():
        yield row.key, row.state, row.result

with Database(source, view) as original, Database(destination, view) as migrated:
    require_equal(original.documents(), migrated.documents(), "Documents")
    require_equal(view_values(original), view_values(migrated), "View")
    migrated.rebuild()
    require_equal(view_values(original), view_values(migrated), "Rebuilt view")
    print("Verified", migrated.stats()["documents"], "documents")
    print("Destination format:", migrated.stats()["log"]["format"])

Expected output names the same document count and destination format 2. A mismatch stops the upgrade; keep both files and investigate the view artifacts or source data before resuming writes. A cache/checkpoint error reports accelerator failure and does not undo already committed log data.

6. Switch the application and replace cursors

  1. Configure the application to use metrics-v2.fold with its verified view and the 0.3.2 environment. Avoid renaming or mixing old sidecars into the new database.
  2. Recreate application state from a fresh snapshot. Old cursors belong to the original database generation; portable change-feed consumers must resume from the new snapshot's cursor.
  3. Resume one database owner and normal producers. Stable source IDs and positions still support identical retries.
  4. Check the application’s expected results and writer health. Keep the original log, backup, and previous environment until the cutover is accepted.

Application compatibility checks

AreaCurrent behavior and action
Callback readsA primary callback supplies query()/rows(). Additional registered callbacks use query(name). Snapshot pages are descriptor-bound; historical pages can replay original batches.
Callback live feeds and previewPrimary callback changes()/preview() explicitly fail. Use fresh query snapshots, or inspect the mapper with a resolved Document. Portable Wasm views retain both operations.
Callback code changesUpdate the view/reducer version or references when behavior or captured inputs change. Ordinary callback caches then reconstruct from documents. Callbacks remain trusted deterministic Python.
Metadata sourcesSource names beginning _ are retained metadata and do not map in either host. If older browser data used such names for observations, resolve that source policy before cutover; copying a log does not rename identities.
NumbersUnrepresentable integer inputs are rejected rather than rounded, a rule enforced natively since 0.1.4. Use strings for exact large identifiers.
Authoring toolsUse the documented Node runtime range to rebuild modules. Installing a matching Python wheel needs no Node runtime.

External projection caches

materialize_jsonl_view keeps compact cursor/state documents; the external stream is authoritative and its raw history is not retained in that cache. Do not treat that file as a replacement for the external source.

Preserve the external stream, generation identity, callback code, and references. Close the old cache owner, select a fresh permitted cache filename, and run the helper against the verified external source with 0.3.2. This regenerates state and consumed progress together. Copy migration is available if retaining a legacy cache is necessary, but it does not turn compact metadata into a source archive. A changed descriptor or source generation must reconstruct from the external source; historical external snapshots are unavailable.

Browser upgrades

Export resolved documents before reloading or closing the old worker. Deploy the complete new portable_assets directory, including modules, manifests, and client/worker/host/error files; do not mix old hosts with new manifests or Wasm. Start a new worker, ingest the exported records with their original IDs, sources, positions, and order, and compare keys, states, and results. Obtain new cursors. Browser state remains in memory and has no native log migration step.

Rollback

Before the new log accepts writes, rollback means returning the application to the untouched original log and its matching previous environment/view. Older binaries do not understand FOLDLOG2; an in-place downgrade is unsupported.

After new writes, the original file no longer contains all accepted observations. Preserve both logs and plan replay/reconciliation of the new observations from your authoritative source before switching back. Simply pointing at the old filename would omit those writes. Never discard the converted log as part of an unverified rollback.

If a step fails

Message or symptomNext step
Legacy log is read-onlyComplete copy migration and configure the new filename.
Already open or cannot be lockedStop and close every owner. Do not remove the ownership lock to bypass it.
Destination existsChoose a fresh filename or inspect the existing migrated copy. Do not reset or overwrite it to retry.
Migration source has an incomplete tailKeep the raw backup. Work on a copy, open/close it normally with the intended view to recover an interrupted tail, then retry migration. Complete-frame corruption must fail; do not edit or remove frames to force acceptance.
Unsupported or corrupt headerConfirm that this is a Foldlite native document log and that the backup is intact. Copy migration does not convert another database's format.
Checkpoint or close failsKeep the document log. Fix the reported cache-path, disk-space, or size issue; a failed accelerator save does not roll back durable appends. See documents and caches.