Does the Backtest Tell the Truth?

What I did

Four names, equally weighted. On day 100 the feed simply stops sending D — which is what a real feed does when a name delists.

What happened

D stops being priced after 2024-04-09

as_of        holdings      gross
2024-04-09          4   0.001877
2024-04-14          4   0.001877   <- D has no price here
2024-04-19          4   0.001877
2024-04-24          4   0.001877

total notes in the whole report: 0

three live names at 1/4 each would give ~0.001877   <- this is what it earned
four live names would give                ~0.002503

Why it matters

A quarter of the book silently became non-earning cash. The alpha said hold 25% D; the engine held 25% nothing, and reported holdings = 4 anyway.

There is code to note a missing price — but it never fires. The price table is a pivot, so D is still a column after it dies, just full of nulls. The intersection that decides "did I have a price for this name" therefore still contains D, the note is skipped, and the null return quietly contributes zero to the sum.

This is the bias a point-in-time engine is sold to prevent, arriving through a side door. Compare two alphas where one happens to pick names that delist, and the comparison is distorted with no warning at all.

What I did

Ran a backtest. Changed the numbers inside the source CSV — same path, same project, same window. Re-polled and ran it again.

What happened

run 1  digest 0dc91cb420e4d8af  net 0.197002   A last close 148.23
       (the source file is edited; A now drifts three times as fast)
run 2  digest 0dc91cb420e4d8af  net 0.713240   A last close 324.94

qanat compare says: same_question=True
  "same inputs and same window, so any difference here is the engine"

Why it matters

The digest is described as "what this run was computed from". It hashes the job specs, the backtest block and the window. For a CSV source the spec contains the path, never the contents. So the one thing that actually changes from day to day — the data — is not in it.

Then compare takes the matching digest and tells the researcher the difference must be the engine. It sends them to debug a bug that does not exist, while the real cause is sitting in the data folder.

What I did

The whole journey through the API, nothing by hand: new project, connect a CSV source, add a normalize step, add momentum from the shelf, run, backtest.

What happened

[200] POST /api/sources (csv)            ok
[200] POST /api/steps  (normalize)       ok
[200] GET  /api/check                    {"ok": true, "errors": []}
[200] POST /api/alphas (momentum)        {"id":"alpha_mom","writes":"weights.mom"}

run the pipeline:
   alpha_mom  failed  ValueError: step 'alpha_mom' has no universe set

backtest it:
   HTTP 200   totals {}   periods 0   failures 37

Why it matters

Every alpha on the shelf calls ctx.universe() without checking. But universe is optional on the request and on the step, and validate() only checks that a universe you named exists — never that a step which needs one has one.

So the shortest path a new user can take, entirely inside the console, ends with a green check and a broken alpha. The API already refuses an alpha pointed at a table with no price column; this is the same class of check and it is missing.

And the backtest returns HTTP 200 for a run with no periods and 37 failures. The store records status: failed correctly, but the response says success with an empty body, so the console draws an empty report instead of an error.

What I did

USGS puts its timestamp at properties.time as epoch milliseconds. I flattened it into a top-level column, which is exactly what a normalize step does.

What happened

time_column() finds it:      'time'                the name matches, so qanat adopts it

store.max_time()          -> ConversionException: Unimplemented type for cast (BIGINT -> TIMESTAMP)
store.read(as_of=...)     -> ConversionException: Unimplemented type for cast (BIGINT -> TIMESTAMP)
open_pit() -- the replay  -> succeeded, and built a view that throws when read

Why it matters

Every as-of comparison does a bare CAST(col AS TIMESTAMP). DuckDB refuses that for an integer, so any table whose clock is an epoch number is unusable for replay — and epoch seconds or milliseconds is what USGS, GitHub, Slack, Binance and most public APIs send.

The worst part is the third line. open_pit succeeds and reports the table as shadowed, because the view is lazy. The failure arrives later, mid-replay, as a DuckDB cast error that says nothing about which table or why.

What I did

Polled the same holiday feed for two countries. Korea has no regional holidays, so counties is null in every row. The United States does.

What happened

poll 1 (KR): ok, 16 rows
   counties column typed: INTEGER          every value was null, so pandas guessed

poll 2 (US): FAIL  ConversionException: Unimplemented type for cast (VARCHAR[] -> INTEGER)
   table rows: still 16

Why it matters

The first response permanently decides the schema. A field that happens to be empty on day one becomes INTEGER, and the day it carries real data the feed stops landing — with an error about a cast, not about a schema.

This is not a corner case. Optional fields that are usually null are the normal shape of a public API, and adding a second country, a second exchange or a second ticker is the normal way a project grows. There is no way to recover except dropping the table by hand.