Upgrade guide · 0.4.2
Version 0.4.2 ships reproducible in-repo numeric reducers, hardened AssemblyScript portable views, incremental callback checkpoints, and stricter query/get validation while keeping the FOLDLOG2 document log format unchanged. Existing 0.2.0 and 0.3.x logs need no log migration; older 0.1.x logs require the verified copy migration below.
The consolidated changelog covers every published release. The 0.2.0 release includes distribution hashes and validation receipts.
Close the database, preserve its log and exact view artifacts, and install foldlitedb==0.4.2. Open the same log with the same view and compare results. Custom portable Wasm views whose module bytes are unchanged retain their checkpoints, generation, and cursors without replaying checkpointed batches. Bundled portable views (View.bundled("metrics"), View.bundled("catalogue")) and the callback document module were rebuilt with hardened JSON/Unicode/decimal parsing, so opening an older bundled database automatically reconstructs its disposable .cache from the unchanged log and starts a fresh cursor generation.
Check two tightened Python call signatures before cutover: the optional positional argument on db.query(view_name, ...) and db.rows(view_name, ...) now exclusively names a registered callback view (pass key=... explicitly for exact key selection), and db.get(id=None, *, position=None, source="default") requires either a nonempty string id or a safe integer position rather than both or neither. Native execution continues to use Wasmi 2.0 fuel accounting from 0.3.2+. Rollback to 0.3.x uses the same FOLDLOG2 log after closing the 0.4.2 owner and starting fresh cursors if the disposable cache was rebuilt.
foldlitedb==0.4.2 in a separate environment and open the same log with the same view. FOLDLOG2 is unchanged. Unchanged custom Wasm modules restore their compatible checkpoint and reconstruct the query ordering index without rerunning checkpointed Wasm; bundled views automatically rebuild their disposable .cache under the hardened 0.4.2 bundled modules.after query tokens are rejected explicitly because their ordering and filter binding differ. Saved snapshot/change cursors remain valid when a compatible custom module cache survives; cache reconstruction, bundled module updates, module replacement, or explicit rebuild expires them cleanly so consumers refresh from a new snapshot. Document pagination is unchanged.key=... rather than positional arguments. Query ordering changes do not reorder documents, reducer contributions, or change-feed sequences.Use the same selector and inclusive flags on every query page; only limit may change. See the range and prefix reference. Browser deployments must update the complete host and module assets together and obtain new worker cursors. The portable module ABI v1 is unchanged.
For rollback to 0.2.0, close the new owner, restore the 0.2.0 module if using bundled views, and use a fresh 0.2.0 query traversal; do not reuse 0.3.x/0.4.x query tokens. The unchanged log format remains readable. Preserve the original backup and computation, and account for any writes made during the upgrade.
The remaining steps upgrade native FOLDLOG1 document logs from 0.1.x to FOLDLOG2. New 0.2.0 and later databases already use the new format. Browser databases have no native log to migrate; see browser upgrades. Compact external projection caches have a different source of truth; see external projections.
Use Python 3.11 or newer. Preserve the exact Wasm module, or Python callback code, versions, references, and other reproducible inputs needed by your view. Migration copies facts and batch boundaries; it does not change your computation or repair already incorrect application data. Plan new snapshot cursors for the migrated database.
metrics-v2.fold.If the old primary filename uses the newly reserved sidecar namespace, make an independent closed-log copy under a permitted name such as metrics-original.fold and use that copy as the source in steps 3–5. Keep the untouched original as well. This lets the new reader inspect and compare it without bypassing filename protection.
The log contains the retained documents. Its .cache sibling is disposable; copying a cache alone is not a backup. Do not delete a live owner's .lock file. Do not use reset=True to upgrade: reset explicitly discards source documents.
python -m venv .venv-upgrade
. .venv-upgrade/bin/activate
python -m pip install --only-binary=:all: foldlitedb==0.4.2
python -c "from importlib.metadata import version; print(version('foldlitedb'))"
On Windows Command Prompt, activate with .venv-upgrade\Scripts\activate.bat. Matching wheels need no compiler. For other source-build environments, follow the installation prerequisites. Keep the previous environment available until cutover is complete.
The Python snippets in steps 3–5 run in order in the same session or script, from the directory containing your log. This example uses bundled metrics. For a custom portable view, replace View.bundled("metrics") with your preserved module, for example View("metrics-original.wasm"). For ordinary callback views, pass your preserved callback definition.
from pathlib import Path
from foldlitedb import Database, View
source = Path("metrics.fold")
if not source.is_file():
raise FileNotFoundError(source) # Avoid creating an empty database by mistake.
view = View.bundled("metrics")
with Database(source, view) as original:
log = original.stats()["log"]
print({"format": log["format"], "read_only": log["read_only"]})
A legacy log reports format: 1 and read_only: True. Reads and export work; new appends fail with migration guidance. If it reports format 2, skip the copy migration: it already uses the current format. Opening may recover an interrupted final frame, which is why the raw closed-log backup comes first.
destination = Path("metrics-v2.fold")
if destination.exists():
raise FileExistsError(destination)
Database.migrate(source, destination)
Both paths must be persistent files in existing parent directories. The destination must not exist and must differ from the source. Primary filenames ending in .cache or .lock, or containing .cache., are reserved case-insensitively; metrics-v2.fold is suitable.
Migration verifies the source, preserves document IDs, sources, positions, order, and original batch boundaries, and leaves source bytes unchanged. It refuses an active owner or an existing destination. If copying fails, it removes its partial destination when possible and reports cleanup failures; inspect any remaining destination before retrying. It does not copy old cursor generations or transplant caches. Ordinary opens reject reserved legacy primary names; explicit migration can copy such an old source into a permitted destination.
With writers still stopped, compare resolved documents and view output under the same computation. The verifier streams through bounded pages and excludes database-local cursors and row sequence metadata from the comparison.
from itertools import zip_longest
def require_equal(left, right, label):
missing = object()
for number, (a, b) in enumerate(zip_longest(left, right, fillvalue=missing), 1):
if a != b:
raise RuntimeError(f"{label} differs at entry {number}")
def view_values(database):
for row in database.rows():
yield row.key, row.state, row.result
with Database(source, view) as original, Database(destination, view) as migrated:
require_equal(original.documents(), migrated.documents(), "Documents")
require_equal(view_values(original), view_values(migrated), "View")
migrated.rebuild()
require_equal(view_values(original), view_values(migrated), "Rebuilt view")
print("Verified", migrated.stats()["documents"], "documents")
print("Destination format:", migrated.stats()["log"]["format"])
Expected output names the same document count and destination format 2. A mismatch stops the upgrade; keep both files and investigate the view artifacts or source data before resuming writes. A cache/checkpoint error reports accelerator failure and does not undo already committed log data.
metrics-v2.fold with its verified view and the 0.4.2 environment. Avoid renaming or mixing old sidecars into the new database.| Area | Current behavior and action |
|---|---|
| Callback reads | A primary callback supplies query()/rows(). Additional registered callbacks use query(name); pass exact key filters via key=.... Snapshot pages are descriptor-bound; historical pages can replay original batches. |
| Callback live feeds and preview | Primary callback changes()/preview() explicitly fail. Use fresh query snapshots, or inspect the mapper with a resolved Document. Portable Wasm views retain both operations. |
| Callback code changes | Update the view/reducer version or references when behavior or captured inputs change. Ordinary callback caches then reconstruct from documents. Callbacks remain trusted deterministic Python. |
| Metadata sources | Source names beginning _ are retained metadata and do not map in either host. If older browser data used such names for observations, resolve that source policy before cutover; copying a log does not rename identities. |
| Numbers | Unrepresentable integer inputs are rejected rather than rounded, a rule enforced natively since 0.1.4. Use strings for exact large identifiers. |
| Authoring tools | Use the documented Node runtime range to rebuild modules. Installing a matching Python wheel needs no Node runtime. |
materialize_jsonl_view keeps compact cursor/state documents; the external stream is authoritative and its raw history is not retained in that cache. Do not treat that file as a replacement for the external source.
Preserve the external stream, generation identity, callback code, and references. Close the old cache owner, select a fresh permitted cache filename, and run the helper against the verified external source with 0.4.2. This regenerates state and consumed progress together. Copy migration is available if retaining a legacy cache is necessary, but it does not turn compact metadata into a source archive. A changed descriptor or source generation must reconstruct from the external source; historical external snapshots are unavailable.
Export resolved documents before reloading or closing the old worker. Deploy the complete new portable_assets directory, including modules, manifests, and client/worker/host/error files; do not mix old hosts with new manifests or Wasm. Start a new worker, ingest the exported records with their original IDs, sources, positions, and order, and compare keys, states, and results. Obtain new cursors. Browser state remains in memory and has no native log migration step.
Before the new log accepts writes, rollback means returning the application to the untouched original log and its matching previous environment/view. Older binaries do not understand FOLDLOG2; an in-place downgrade is unsupported.
After new writes, the original file no longer contains all accepted observations. Preserve both logs and plan replay/reconciliation of the new observations from your authoritative source before switching back. Simply pointing at the old filename would omit those writes. Never discard the converted log as part of an unverified rollback.
| Message or symptom | Next step |
|---|---|
| Legacy log is read-only | Complete copy migration and configure the new filename. |
| Already open or cannot be locked | Stop and close every owner. Do not remove the ownership lock to bypass it. |
| Destination exists | Choose a fresh filename or inspect the existing migrated copy. Do not reset or overwrite it to retry. |
| Migration source has an incomplete tail | Keep the raw backup. FOLDLOG1 copy migration automatically skips an incomplete torn final frame while leaving the source untouched; for a FOLDLOG2 source with an interrupted final frame, open and close a working copy normally first to recover the tail, then retry migration. Complete-frame corruption must fail; do not edit or remove frames to force acceptance. |
| Unsupported or corrupt header | Confirm that this is a Foldlite native document log and that the backup is intact. Copy migration does not convert another database's format. |
| Checkpoint or close fails | Keep the document log. Fix the reported cache-path, disk-space, or size issue; a failed accelerator save does not roll back durable appends. See documents and caches. |