Upgrade guide · 0.4.1

Upgrade to Foldlite DB 0.4.1

Version 0.4.1 ships reproducible in-repo numeric reducers, hardened AssemblyScript portable views, incremental callback checkpoints, and stricter query/get validation while keeping the FOLDLOG2 document log format unchanged. Existing 0.2.0 and 0.3.x logs need no log migration; older 0.1.x logs require the verified copy migration below.

The consolidated changelog covers every published release. The 0.2.0 release includes distribution hashes and validation receipts.

Upgrade from 0.3.1, 0.3.2, or 0.3.3

Close the database, preserve its log and exact view artifacts, and install foldlitedb==0.4.1. Open the same log with the same view and compare results. Custom portable Wasm views whose module bytes are unchanged retain their checkpoints, generation, and cursors without replaying checkpointed batches. Bundled portable views (View.bundled("metrics"), View.bundled("catalogue")) and the callback document module were rebuilt with hardened JSON/Unicode/decimal parsing, so opening an older bundled database automatically reconstructs its disposable .cache from the unchanged log and starts a fresh cursor generation.

Check two tightened Python call signatures before cutover: the optional positional argument on db.query(view_name, ...) and db.rows(view_name, ...) now exclusively names a registered callback view (pass key=... explicitly for exact key selection), and db.get(id=None, *, position=None, source="default") requires either a nonempty string id or a safe integer position rather than both or neither. Native execution continues to use Wasmi 2.0 fuel accounting from 0.3.2+. Rollback to 0.3.x uses the same FOLDLOG2 log after closing the 0.4.1 owner and starting fresh cursors if the disposable cache was rebuilt.

Upgrade from 0.2.0

  1. Close the database and preserve a backup of the document log and exact view artifacts, as for any package upgrade.
  2. Install foldlitedb==0.4.1 in a separate environment and open the same log with the same view. FOLDLOG2 is unchanged. Unchanged custom Wasm modules restore their compatible checkpoint and reconstruct the query ordering index without rerunning checkpointed Wasm; bundled views automatically rebuild their disposable .cache under the hardened 0.4.1 bundled modules.
  3. Start fresh query traversals. 0.2.x after query tokens are rejected explicitly because their ordering and filter binding differ. Saved snapshot/change cursors remain valid when a compatible custom module cache survives; cache reconstruction, bundled module updates, module replacement, or explicit rebuild expires them cleanly so consumers refresh from a new snapshot. Document pagination is unchanged.
  4. Check application assumptions about row order and query calls. Numbers sort numerically, strings by Unicode codepoint, arrays by recursive element order, and objects by their ordered members; pass exact key filters via key=... rather than positional arguments. Query ordering changes do not reorder documents, reducer contributions, or change-feed sequences.

Use the same selector and inclusive flags on every query page; only limit may change. See the range and prefix reference. Browser deployments must update the complete host and module assets together and obtain new worker cursors. The portable module ABI v1 is unchanged.

For rollback to 0.2.0, close the new owner, restore the 0.2.0 module if using bundled views, and use a fresh 0.2.0 query traversal; do not reuse 0.3.x/0.4.x query tokens. The unchanged log format remains readable. Preserve the original backup and computation, and account for any writes made during the upgrade.

Before you start

The remaining steps upgrade native FOLDLOG1 document logs from 0.1.x to FOLDLOG2. New 0.2.0 and later databases already use the new format. Browser databases have no native log to migrate; see browser upgrades. Compact external projection caches have a different source of truth; see external projections.

Use Python 3.11 or newer. Preserve the exact Wasm module, or Python callback code, versions, references, and other reproducible inputs needed by your view. Migration copies facts and batch boundaries; it does not change your computation or repair already incorrect application data. Plan new snapshot cursors for the migrated database.

1. Stop writers and back up the source

  1. Stop producers, background imports, and consumers that hold the database open. Close its owner cleanly before copying files.
  2. Copy the original document log to a separate backup location. Keep the view artifacts and reference inputs with that backup. Confirm the copied log exists and has the expected size or checksum.
  3. Keep the original filename in place for comparison. Choose a different destination, such as metrics-v2.fold.

If the old primary filename uses the newly reserved sidecar namespace, make an independent closed-log copy under a permitted name such as metrics-original.fold and use that copy as the source in steps 3–5. Keep the untouched original as well. This lets the new reader inspect and compare it without bypassing filename protection.

The log contains the retained documents. Its .cache sibling is disposable; copying a cache alone is not a backup. Do not delete a live owner's .lock file. Do not use reset=True to upgrade: reset explicitly discards source documents.

2. Install 0.4.1 in a separate environment

python -m venv .venv-upgrade
. .venv-upgrade/bin/activate
python -m pip install --only-binary=:all: foldlitedb==0.4.1
python -c "from importlib.metadata import version; print(version('foldlitedb'))"

On Windows Command Prompt, activate with .venv-upgrade\Scripts\activate.bat. Matching wheels need no compiler. For other source-build environments, follow the installation prerequisites. Keep the previous environment available until cutover is complete.

3. Inspect the closed log with its intended view

The Python snippets in steps 3–5 run in order in the same session or script, from the directory containing your log. This example uses bundled metrics. For a custom portable view, replace View.bundled("metrics") with your preserved module, for example View("metrics-original.wasm"). For ordinary callback views, pass your preserved callback definition.

from pathlib import Path
from foldlitedb import Database, View

source = Path("metrics.fold")
if not source.is_file():
    raise FileNotFoundError(source)  # Avoid creating an empty database by mistake.

view = View.bundled("metrics")
with Database(source, view) as original:
    log = original.stats()["log"]
    print({"format": log["format"], "read_only": log["read_only"]})

A legacy log reports format: 1 and read_only: True. Reads and export work; new appends fail with migration guidance. If it reports format 2, skip the copy migration: it already uses the current format. Opening may recover an interrupted final frame, which is why the raw closed-log backup comes first.

4. Copy into a new log

destination = Path("metrics-v2.fold")
if destination.exists():
    raise FileExistsError(destination)

Database.migrate(source, destination)

Both paths must be persistent files in existing parent directories. The destination must not exist and must differ from the source. Primary filenames ending in .cache or .lock, or containing .cache., are reserved case-insensitively; metrics-v2.fold is suitable.

Migration verifies the source, preserves document IDs, sources, positions, order, and original batch boundaries, and leaves source bytes unchanged. It refuses an active owner or an existing destination. If copying fails, it removes its partial destination when possible and reports cleanup failures; inspect any remaining destination before retrying. It does not copy old cursor generations or transplant caches. Ordinary opens reject reserved legacy primary names; explicit migration can copy such an old source into a permitted destination.

5. Compare documents, results, and rebuilt results

With writers still stopped, compare resolved documents and view output under the same computation. The verifier streams through bounded pages and excludes database-local cursors and row sequence metadata from the comparison.

from itertools import zip_longest

def require_equal(left, right, label):
    missing = object()
    for number, (a, b) in enumerate(zip_longest(left, right, fillvalue=missing), 1):
        if a != b:
            raise RuntimeError(f"{label} differs at entry {number}")

def view_values(database):
    for row in database.rows():
        yield row.key, row.state, row.result

with Database(source, view) as original, Database(destination, view) as migrated:
    require_equal(original.documents(), migrated.documents(), "Documents")
    require_equal(view_values(original), view_values(migrated), "View")
    migrated.rebuild()
    require_equal(view_values(original), view_values(migrated), "Rebuilt view")
    print("Verified", migrated.stats()["documents"], "documents")
    print("Destination format:", migrated.stats()["log"]["format"])

Expected output names the same document count and destination format 2. A mismatch stops the upgrade; keep both files and investigate the view artifacts or source data before resuming writes. A cache/checkpoint error reports accelerator failure and does not undo already committed log data.

6. Switch the application and replace cursors

  1. Configure the application to use metrics-v2.fold with its verified view and the 0.4.1 environment. Avoid renaming or mixing old sidecars into the new database.
  2. Recreate application state from a fresh snapshot. Old cursors belong to the original database generation; portable change-feed consumers must resume from the new snapshot's cursor.
  3. Resume one database owner and normal producers. Stable source IDs and positions still support identical retries.
  4. Check the application’s expected results and writer health. Keep the original log, backup, and previous environment until the cutover is accepted.

Application compatibility checks

AreaCurrent behavior and action
Callback readsA primary callback supplies query()/rows(). Additional registered callbacks use query(name); pass exact key filters via key=.... Snapshot pages are descriptor-bound; historical pages can replay original batches.
Callback live feeds and previewPrimary callback changes()/preview() explicitly fail. Use fresh query snapshots, or inspect the mapper with a resolved Document. Portable Wasm views retain both operations.
Callback code changesUpdate the view/reducer version or references when behavior or captured inputs change. Ordinary callback caches then reconstruct from documents. Callbacks remain trusted deterministic Python.
Metadata sourcesSource names beginning _ are retained metadata and do not map in either host. If older browser data used such names for observations, resolve that source policy before cutover; copying a log does not rename identities.
NumbersUnrepresentable integer inputs are rejected rather than rounded, a rule enforced natively since 0.1.4. Use strings for exact large identifiers.
Authoring toolsUse the documented Node runtime range to rebuild modules. Installing a matching Python wheel needs no Node runtime.

External projection caches

materialize_jsonl_view keeps compact cursor/state documents; the external stream is authoritative and its raw history is not retained in that cache. Do not treat that file as a replacement for the external source.

Preserve the external stream, generation identity, callback code, and references. Close the old cache owner, select a fresh permitted cache filename, and run the helper against the verified external source with 0.4.1. This regenerates state and consumed progress together. Copy migration is available if retaining a legacy cache is necessary, but it does not turn compact metadata into a source archive. A changed descriptor or source generation must reconstruct from the external source; historical external snapshots are unavailable.

Browser upgrades

Export resolved documents before reloading or closing the old worker. Deploy the complete new portable_assets directory, including modules, manifests, and client/worker/host/error files; do not mix old hosts with new manifests or Wasm. Start a new worker, ingest the exported records with their original IDs, sources, positions, and order, and compare keys, states, and results. Obtain new cursors. Browser state remains in memory and has no native log migration step.

Rollback

Before the new log accepts writes, rollback means returning the application to the untouched original log and its matching previous environment/view. Older binaries do not understand FOLDLOG2; an in-place downgrade is unsupported.

After new writes, the original file no longer contains all accepted observations. Preserve both logs and plan replay/reconciliation of the new observations from your authoritative source before switching back. Simply pointing at the old filename would omit those writes. Never discard the converted log as part of an unverified rollback.

If a step fails

Message or symptomNext step
Legacy log is read-onlyComplete copy migration and configure the new filename.
Already open or cannot be lockedStop and close every owner. Do not remove the ownership lock to bypass it.
Destination existsChoose a fresh filename or inspect the existing migrated copy. Do not reset or overwrite it to retry.
Migration source has an incomplete tailKeep the raw backup. FOLDLOG1 copy migration automatically skips an incomplete torn final frame while leaving the source untouched; for a FOLDLOG2 source with an interrupted final frame, open and close a working copy normally first to recover the tail, then retry migration. Complete-frame corruption must fail; do not edit or remove frames to force acceptance.
Unsupported or corrupt headerConfirm that this is a Foldlite native document log and that the backup is intact. Copy migration does not convert another database's format.
Checkpoint or close failsKeep the document log. Fix the reported cache-path, disk-space, or size issue; a failed accelerator save does not roll back durable appends. See documents and caches.