finding · findings/evolution/shared_data/manifest.md

manifest.json — the catalogue card for every slice of crypto data

A manifest is just a master list — like the index card catalogue in a library that tells you what books exist, where each one sits on the shelf, and whether it's in good condition. This page documents manifest.json: the master list of every chunk of crypto market data the research used to pull together, what each entry means, and the rules that kept that list trustworthy.

What is this?

• manifest.json is a JSON file — JSON is a simple, human-readable text format that computers read easily (think of it as a tidy outline of labels and values).
• It was meant to be the machine-readable index of every data cell that had been pulled into the shared data layer. A "cell" here is one specific slice of data defined by four coordinates: which market (substrate), which trading pair (symbol), which sensitivity setting (threshold), and which time window (slice).
• Think of it as the address system for the data: every cell gets a unique name, and the manifest records everything about it in one place.

Why it matters

Two separate quality checks — one that evaluates measurement metrics, one that evaluates candidate features — both needed to look things up programmatically (i.e., with code, not by hand). Specifically, they needed to discover:

• Which cells exist at all.
• Which cells apply to which symbol — some data starts later than others, so a slice from before a coin existed simply doesn't apply (the effective_start filter).
• How much data and how clean — the row count plus how much of it is actually filled in (the non-null fraction).
• Where the file lives on the bigblack server, so a script can open it.
• A fingerprint (SHA-256) so results can be reproduced and tampering caught.

The shape of the file (top level)

At the very top, the manifest carries a few bookkeeping facts about the whole catalogue, then two big lists: cells (the data chunks) and fetch_runs (a log of each time the catalogue was built or refreshed).

{
  "schema_version": 1,                      // bump when this schema changes
  "pre_spec_version": 1,                    // which pre-spec.md version produced this
  "pre_spec_file": "pre-spec.md",
  "generated_at": "2026-05-23T14:32:00Z",   // ISO timestamp of last fetch
  "host": "bigblack",                       // host that ran the fetcher
  "data_root_on_bigblack": "/home/nasimubd/shared_data/",
  "cells": [ ... ],                          // per-cell records (see below)
  "fetch_runs": [ ... ]                      // per-run log entries (see below)
}

A schema is just the agreed-upon shape/layout of the data — which fields exist and what type each is. schema_version is a number bumped whenever that layout changes, so old readers can tell they're looking at a newer format.

One entry per data chunk (the per-cell record)

Each cell in the cells list looks like this — a complete catalogue card for one slice of data:

{
  "cell_id": "crypto_BTCUSDT_100dbps_slice01",
  "substrate": "crypto",
  "symbol": "BTCUSDT",
  "threshold_dbps": 100,
  "slice_id": "slice01",
  "slice_start_iso": "2026-01-01T00:00:00Z",
  "slice_end_iso": "2026-02-28T23:59:59.999999Z",
  "applicable": true,
  "applicable_reason": "in range; effective_start satisfied",
  "bar_count": 84321,
  "non_null_frac_per_col": {
    "ofi": 1.0,
    "vwap": 1.0,
    "kyle_lambda_proxy": 1.0,
    "lookback_hurst": 0.7234,
    ...
  },
  "fetch_timestamp_ingestion": "2026-05-23T14:32:18Z",
  "parquet_path": "/home/nasimubd/shared_data/crypto/BTCUSDT_100dbps_slice01.parquet",
  "file_size_bytes": 4521734,
  "sha256": "abc123def456..."
}

The schema / fields, explained

A log of each build run (the per-fetch-run record)

Besides the data chunks, the manifest keeps a diary: one entry every time the fetcher ran, recording how it went.

{
  "run_id": "20260523T143200Z",
  "host": "bigblack",
  "started_at": "2026-05-23T14:32:00Z",
  "completed_at": "2026-05-23T14:50:18Z",
  "pre_spec_version": 1,
  "cells_attempted": 552,
  "cells_fetched": 550,
  "cells_skipped_already_fetched": 0,
  "cells_skipped_not_applicable": 2,
  "cells_failed": 0,
  "cpu_env": {
    "OPENBLAS_NUM_THREADS": "1",
    "OMP_NUM_THREADS": "1",
    "MKL_NUM_THREADS": "1",
    "NUMEXPR_NUM_THREADS": "1"
  },
  "loadavg_at_start": [6.81, 6.90, 7.27],
  "nproc": 32
}

• What it captures: a run ID and host, start/end times, and a tally — how many cells it tried (cells_attempted), succeeded (cells_fetched), skipped (already there, or not applicable), and failed.
• The cpu_env block pins each math library to a single thread — a precaution so parallel work stays predictable and doesn't fight over the machine.
• System snapshot: loadavg_at_start (how busy the machine was) and nproc (how many CPU cores it had, here 32).

How it's used

The point of a machine-readable catalogue is that a few lines of code can answer real questions instantly. Examples from the doc:

Python — list every cell for one symbol

import json
from pathlib import Path

manifest = json.loads(Path("findings/evolution/shared_data/manifest.json").read_text())
btc_cells = [c for c in manifest["cells"] if c["symbol"] == "BTCUSDT"]
print(f"BTCUSDT cells: {len(btc_cells)}")

Python — find every applicable cell for one check

btc_100dbps = [c for c in manifest["cells"]
               if c["substrate"] == "crypto"
               and c["symbol"] == "BTCUSDT"
               and c["threshold_dbps"] == 100
               and c["applicable"]]
parquet_paths = [c["parquet_path"] for c in btc_100dbps]
# Audit-side spike reads each path; the 24 slices give it K=24 per-pair values

Bash — quick counts with jq

jq is a small command-line tool for slicing JSON from the terminal.

# Total cells
jq '.cells | length' findings/evolution/shared_data/manifest.json

# By substrate
jq '.cells | group_by(.substrate) | map({substrate: .[0].substrate, count: length})' \
   findings/evolution/shared_data/manifest.json

# List non-applicable cells with reasons
jq '.cells[] | select(.applicable == false) | {cell_id, applicable_reason}' \
   findings/evolution/shared_data/manifest.json

The rules that kept the catalogue trustworthy

The manifest was designed with built-in guarantees — and the doc is honest about exactly how far each one goes:

These were accurate as of pre-spec v1. Stricter rules — a hard error on duplicate IDs, and a write-time cross-check of timestamps against pre-spec.md — were tracked as follow-up items to add before the first real run.

When the manifest changes

Where to look next

• pre-spec.md — what cells should exist (v2, crypto-only): the locked plan.
• orthogonality_probe.py — the active probe that replaced the fetcher (reads the database live; writes results/, not this manifest).
• README.md — the folder hub for the shared data layer.