The forex data-availability probe: "is the EURUSD data really there, and exactly what should we grab?"
A quick, look-but-don't-touch inventory check that confirmed our euro/dollar (EURUSD) forex price data actually exists on the research server, then pinned down the precise 42-item shopping list of data fields a future analysis should pull. It has since been shelved — forex is out of scope for now — but the homework is saved so nobody has to re-do it.
In plain English. Before building anything on top of a dataset, you first walk into the warehouse and check two things: is the box of goods actually on the shelf, and which specific items inside the box do we want. This probe did exactly that for foreign-exchange (forex) data — the euro-vs-dollar exchange rate. It confirmed the data is there in full (over 7 million records, no holes), then wrote down a tidy 42-item picking list and deliberately left out the things that would either contaminate the analysis or just waste space. Then the whole forex idea got parked, so this note is the saved receipt for whenever someone restarts it.
What is this?
A read-only probe — a check that only looks at data and never changes it — run against bigblack, the remote research server that holds all the price databases.
It inspected fxview_cache.forex_bars — a database table of pre-computed forex bars (a "bar" is one summarized slice of price action: its open, high, low and close, like a single candle on a price chart). "FXView" is the upstream data vendor the forex prices come from.
It was run during code review of pull request #501 on 2026-05-26, as two checklist items: A2 (does the data exist and is it complete?) and A3 (which columns do we actually want?).
Status: archived on 2026-05-30. Forex was removed from active scope "for now" — the shared data layer is currently crypto-only. This note is kept so that if forex is ever reintroduced, nobody has to re-run the availability check from scratch.
One term, defined once.dbps = "decimal basis points," the unit for how big a price move triggers a new bar. 1 dbps = 0.001%. So 5 dbps = 0.005% — a very small move, which makes sense for forex where the euro/dollar rate barely budges compared to crypto. A small threshold means many bars get created, giving a fine-grained view of price action.
Why it matters
Don't build on sand. If you wire up an analysis pipeline assuming a dataset exists and is complete, and it turns out to be missing or full of holes, every result downstream is garbage. This probe removes that risk up front.
Lock the shopping list once. The forex table is unusually rich — 201 columns (data fields). Grabbing all of them would be wasteful and, worse, some columns would secretly poison the analysis. Deciding exactly which 42 to take, and writing down why, means the decision is documented and repeatable.
Save the work for later. Because forex is parked, the value of this note is purely as a saved decision — the equivalent of leaving detailed notes for whoever picks the project back up.
What the probe found (A2: does the data exist?)
Two simple look-only queries were run. The first just listed the tables in the fxview_cache database and confirmed forex_bars is present (22 tables total). The second counted the actual EURUSD records at the 5 dbps setting:
7.24M
Total EURUSD bars on the server
5.50M
Bars inside the research window
2018→2026
Coverage span (Jan 2018 → May 2026)
5 dbps
Bar trigger size (0.005%)
Verdict: full, continuous coverage for EURUSD at 5 dbps — no gaps across all 24 time slices the analysis would use.
For context, each slice only needs roughly 12,000–90,000 bars to produce statistically reliable numbers. With ~230,000 bars per slice available, the data is comfortably above every minimum threshold — plenty to work with.
A bonus fix: the column names were wrong in the code
The fetching code had been guessing two column names. The probe revealed the real names, which happen to match the crypto table's conventions exactly — so no special renaming logic is needed:
What the code assumed
Actual name on the server
symbol_normalized
symbolsame as crypto
threshold_dbps
threshold_decimal_bpssame as crypto
What the probe decided (A3: which columns to take?)
The table has 201 columns — far richer than the crypto equivalent (~50 columns). The probe sorted every column into a category and applied one clear rule:
Include columns that are populated and are genuine, non-forward-looking feature candidates (raw ingredients for analysis). Exclude labels, not-yet-implemented columns, pre-derived rankings, bookkeeping metadata, and empty placeholders.
The single most important exclusion is the 32 "label" columns — these are forward-looking (they describe what happened in the future relative to each bar). Including them in an analysis that's meant to find patterns would be leakage: like grading your own exam with the answer key visible. They're excluded by design.
The 201 columns, sorted
Category
~Count
Kept?
Why
Core: time, OHLC, identifiers
8
subset
Drop redundant IDs, keep the essentials
Quote: bid / ask prices
8
subset
Keep bid_close, ask_close
Spread: gap between buy & sell price
9
subset
Keep close, mean, variance
Quote activity (counts, duration)
3
subset
Keep quote_count, duration_us
Session flags + label
5
subset
Keep session_label (London / NY etc.)
Intra-bar features (within one bar)
9
all 9
Genuine ingredients
Lookback features (recent-history window)
10
all 10
Genuine ingredients
Active microstructure (volatility, liquidity, etc.)
~40 active
top 10
Best volatility / liquidity / jump signals
Forward-looking labels
32
excluded
Would leak the future into the analysis
Deferred / always-empty features
7
excluded
NULL by design — no data in them
Pre-computed rankings (quintiles, etc.)
~40
excluded
Can be re-derived from base columns later
Governance / provenance metadata
~10
excluded
Bookkeeping, not analysis input
Gap-detection placeholders
2
excluded
Hardcoded empty
Key numbers
201
Total columns in the forex table
169
Non-label candidate columns
42
Columns actually selected (v1)
32
Forward-looking labels excluded
Why 42 — and not all 169?
Balance with crypto. The crypto list has 21 columns; 42 is about 2× that — enough to honor forex's richer data while keeping the two analyses comparable.
Storage sanity. The 42-column slice for EURUSD comes to roughly 1.8 GB raw, shrinking to about 250–500 MB on disk after compression. Taking all 169 columns would balloon it to ~7 GB — mostly wasted bytes on empty or re-derivable fields.
Faster downstream reads. A smaller, cleaner column set means analyses load faster, and anything left out (like quintile rankings) can be recomputed on demand from the base columns.
What it means for scope
Both checklist items passed. A2 ✅ (the data exists and is complete) and A3 ✅ (the 42-column list is locked, labels deliberately excluded).
No data-quality problems were found for EURUSD at this time — the forex data is clean.
A few technical notes were carried forward for whoever revives forex: forex bars reset weekly (Sunday 21:00 UTC) whereas crypto bars run continuously; queries must use a FINAL modifier to get the single deduplicated latest row per bar; and the "Binance spot-only" data rule doesn't apply here because forex comes from a different vendor (FXView).
Forex is parked. The shared data layer is crypto-only as of 2026-05-30. This finding is the saved groundwork for any future forex reintroduction, so the availability check never needs repeating.
Bottom line. The euro/dollar forex data is confirmed present and complete (5.5M clean bars in-window), and the exact 42-column "what to fetch" list is locked with future-leaking labels excluded by design — all saved on ice so that whenever forex comes back into scope, the team can pick up exactly where this left off without re-checking the warehouse.