Chaos run · qanat 0.1.1 · main @ 39d4fe8

Breaking Qanat on Purpose

I broke qanat 25 different ways on purpose: crashes, races, bad input, and bad market data. 20 of them found a problem. Here is what I did, what happened, and what to change.

Part two · the console → Part three · the workflow →
Tests before I started 102 pass, 6 skip Test project qanat init --demo Store DuckDB file Network not used
58 fixed 6 partly fixed 9 still open

Everything below was fixed in v0.1.2, except where a badge says otherwise

Each finding carries its own status. Every fix was checked by re-running the script that found it, and the ones that mattered most are now tests — tests/test_audit_fixes.py holds one case per finding, so each of these mistakes fails loudly if it comes back. The full ledger, with the repro scripts, is in audit/.

One finding cannot be closed and is recorded as a limit rather than a fix: a step is arbitrary Python, so it can always read a source file off disk itself and reach data the as-of views hide. The three doors that were qanat's to close are closed.

The numbers

25 tests · 15 problems
25Tests I ran
20Something broke
5Nothing broke
15Problems found
2 very bad 5 bad 6 medium 2 small

Almost every problem comes from one of two things.

One. A backtest changes the project's real tables while it runs. So if the program crashes, or a second backtest starts, or a timed job wakes up in the middle, your tables are left holding old, cut-off data. Nothing tells you.

Two. qanat check looks at the shape of the file, but not at what is inside it. A table name and a file path are used exactly as written. Both end up somewhere that runs them.

Words I use below

short list
replay / backtest
Run the pipeline again over old dates, to see what the alpha would have earned.
store
The one database holding every table. A DuckDB file, or Postgres.
step
One .sql or .py script that reads tables and writes tables.
as-of date
The day a replay is pretending it is right now.
scheduler
The part of qanat serve that runs jobs on a clock.
SIGKILL
Killing a program with no warning. Like pulling the power cable.

All 25 tests

in the order I ran them
What I did Where Result Problem
Crashes
Killed the program 3 seconds into a replaybacktest, storebrokeF-03
fixed in 0.1.2
Cut qanat.yaml in half while it was being savedproject_iobrokeF-06
fixed in 0.1.2
A step that calls sys.exit(3)runnerbrokeF-09
fixed in 0.1.2
A step that raises KeyboardInterruptrunnerbrokeF-09
fixed in 0.1.2
A step that raises MemoryError or RecursionErrorrunnerfine—
Two things at the same time
Counted rows while a replay was runningstore, apibrokeF-04
partly fixed
Ran a timed job while a replay was runningscheduler, storebrokeF-04
partly fixed
Started two replays on one storebacktest, apibrokeF-05
fixed in 0.1.2
Two threads saving qanat.yamlproject_iobrokeF-06
fixed in 0.1.2
Read the project 400 times while it was being savedprojectbrokeF-06
fixed in 0.1.2
200 threads writing a run row at oncestorefine—
Bad input
A table name that contains SQLmodels, storebrokeF-01
fixed in 0.1.2
Ran qanat check on that same projectprojectbrokeF-01
fixed in 0.1.2
script: ../OUTSIDE/pwned.py from the consoleeditor, apibrokeF-02
fixed in 0.1.2
script: /tmp/abs.py from the consoleeditor, apibrokeF-02
fixed in 0.1.2
One bad edit, then normal edits after iteditor, apibrokeF-07
fixed in 0.1.2
SQL inside the /api/table URLapifine—
Moved store: outside the project foldereditorbrokeF-15
still open
Too much work
One year of dates with rebalance: 1sbacktestbrokeF-12
fixed in 0.1.2
A step that never finishes, started 6 timesschedulerbrokeF-13
still open
Bad data
A zero price and a negative pricebacktestbrokeF-08
still open
An alpha writing NaN and inf weightsrunner, apifine—
Retention set to 1s on a raw tableretentionbrokeF-10
fixed in 0.1.2
Two steps in one stage reading each otherproject, runnerbrokeF-11
fixed in 0.1.2
A replayed step writing a table with no date columnrunnerbrokeF-14
still open

The 15 problems

worst first

Each one has the same four parts: what I did, what happened, why it matters, and how to fix it.

Very bad F-01 fixed in 0.1.2 models.py · store.py

A table name can be SQL, and qanat check says it is fine

What I did

I put SQL code inside a table name in qanat.yaml, then ran the step.

to: ['features.z" AS SELECT 1 AS q; DROP TABLE features__momentum; CREATE OR REPLACE TABLE "zzz']

What happened

$ qanat check
  ✓ demo: 5 stages, 12 tables, 12 jobs · contract holds

$ qanat run inj
  run status: failed | ParserException: syntax error at or near "SELECT"
  features.momentum survived: False

The check passed. The step then deleted a real table. The console only shows a red job and a confusing SQL error. It never says a table was deleted.

Why it matters

The code that checks a table name (_qualified()) only counts the dots. It never looks at the rest of the name. That name goes straight into SQL in store.py.

The console API accepts this name with 200 OK. So does the MCP tool save_step. That is the real risk: qanat is built so an agent writes these files. A model does not need bad intent to write a strange name.

How to fix it

  • Check the table name with the same rule Stage.id already uses: ^[a-z][a-z0-9_]*$. Put it in _qualified(), so it covers sources, steps, when:, retention: and time_columns: together.
  • Also double any " inside Store._q(), as a second wall.
  • Add a test that gives validate() a bad name and expects an error.
Very bad F-02 fixed in 0.1.2 editor.py · api.py · mcp.py

A step's script: can point outside the project, and then it is run

What I did

I made a step whose script path goes up and out of the project folder. Then I tried an absolute path too.

What happened

POST /api/steps  {"script":"../OUTSIDE/pwned.py", ...}
  [200] {"ok":true}
  file written outside the project: True

POST /api/steps  {"script":"/tmp/qanat_abs_pwned.py", ...}
  [400] step 'evil2' reads 'raw.bars', which nothing produces
  file written anyway: True

Why it matters

Two problems in one call.

First, save_step() does root / step.script and never checks that the result stays inside the project. So the console can write a .py file anywhere on the disk. Later, _load_module() imports and runs that file.

Second, the file is written before the check runs. So even a request that fails with 400 still leaves a file behind.

How far can someone reach? qanat serve uses --host 127.0.0.1 by default, which is the right choice and keeps this on your own machine. But no write endpoint asks for a password, and docker-compose.yml sets QANAT_HOST: 0.0.0.0 and opens port 8420. In Docker, anyone who can reach the port gets the editor.

How to fix it

  • In validate(), require (root / step.script).resolve() to sit inside root.resolve(). Refuse absolute paths.
  • In save_step(), write the stub file only after apply_and_save() succeeds.
  • Make qanat serve refuse a non-local --host unless you pass --token. Let the Docker entrypoint create one.
Bad F-03 fixed in 0.1.2 backtest.py · store.py

If the program dies during a replay, your tables are left empty

What I did

I started a one-year replay, waited 3 seconds, then killed the process with SIGKILL. Then I opened the store again and counted rows.

What happened

rows before: 3360

AFTER THE CRASH
  raw.daily_prices         rows=3360
  normalized.prices        rows=0
  weights.momentum         rows=0
  leftover pit views: 2
  backtests still marked 'running': 1

Why it matters

A replay rewrites the real tables at every as-of date. The only thing that puts them back is the finally: block at the end of run_backtest. A SIGKILL, an out-of-memory kill, or a closed laptop never reaches that block.

Opening the store again does not notice and does not repair. You come back to an empty pipeline, a backtest that has been "running" for three days, and leftover views. The cure is one plain qanat run — cheap and correct. But nothing tells you that.

How to fix it

  • Best: let a replay work on copies in a separate schema, not on the real tables. This also fixes F-04 and most of F-05.
  • Cheaper: write a marker in _qanat_state when open_pit starts, and clear it in close_pit. If Store.__init__ sees the marker, mark the derived tables as stale, close the stuck backtest as interrupted, and drop the qanat_pit schema.
  • At startup, close any run still marked running. No process owns it any more.
Bad F-04 partly fixed store.py · scheduler.py

While a replay runs, the console shows 0 rows and timed jobs read old data

What I did

I counted rows over and over while a replay ran. Then I started a normal timed job in the middle of one.

What happened

before: normalized.prices = 3360
during: normalized.prices  -> [0, 0, 0, 0, 0, 3360, 3360]

search_path before:        (empty)
search_path DURING replay: qanat_pit,main
timed step 'risk' during replay -> status ok, rows 0

Why it matters

open_pit() changes search_path on the one shared connection, and every pass empties the derived tables. Everything else in the process sees both changes.

So for the whole length of a backtest, the console shows an empty pipeline. The api file promises "if a table says 12,043 rows, that is a count(*)" — and it is, of a table the replay just emptied.

At the same time, a timed step can start, quietly read the old as-of data, write 0 rows, log ok, and wake up everything after it.

How to fix it

  • Replay on copies (see F-03). That removes both halves at once.
  • Until then: make Scheduler.fire() refuse while a replay holds the store, and log "skipped — a replay is running".
  • Set the as-of search_path on a cursor owned by the replay, not on self.con.
  • Add replaying: true to /api/graph, so the console can say the counts are not current.
Bad F-05 fixed in 0.1.2 api.py · scheduler.py

Two replays can run at once, and both give an empty answer

What I did

I started two replays on the same store at the same time.

What happened

replay-0: periods=0  net=None  failures=0
replay-1: periods=0  net=None  failures=0
(the same window alone gives 81 periods)

Why it matters

POST /api/backtest protects itself with AppState._replay and returns a clean 409 to a second caller. But Scheduler._score() — the live scoring pass — calls run_backtest directly and takes no lock. With backtest.live: true, which is a normal setting, a timed pass and a console run can overlap.

They share one Store.as_of, one set of views, and one progress slot. So each keeps resetting the other's clock. No error is raised and no failure is recorded. They just return nothing — the worst shape a wrong answer can take.

How to fix it

  • Move the lock into run_backtest(), or onto the Store, so the API, the scheduler, the CLI and MCP all go through the same door.
  • Key progress by run_id instead of using one slot.
  • Raise BacktestError instead of waiting, so the live pass logs "skipped" and tries again on the next tick.
Bad F-06 fixed in 0.1.2 project_io.py · scheduler.py

qanat.yaml is saved unsafely, from two threads, with no backup

What I did

I had two threads save the project at the same time. Then I read the project 400 times while another thread kept saving it.

What happened

two threads saving at once
  final name:      demo          (the console's rename is gone)
  final live_from: 2025-01-01
  -> last writer wins, no lock, no merge

400 reads while another thread saves
  broken reads: 167  {'ValidationError'}

Why it matters

save_project() is one plain path.write_text(text). That empties the file first, then writes. There is no temp file, no lock, and no old copy kept.

Two parts of qanat serve write this file: the API on every edit, and Scheduler.note_frontier() from the scheduler thread.

Almost half of my reads failed. Each of those is something real: state.reload() after an edit, qanat check in a second terminal, or an agent reading the project over MCP. And a crash during the write leaves half a file, with nothing to go back to.

How to fix it

  • Write to qanat.yaml.tmp in the same folder, fsync, then os.replace(). That is safe on Linux, macOS and Windows.
  • Keep one RLock in project_io that both save_project and load use.
  • Keep the last good file as qanat.yaml.bak, and let load() say so when the main file will not parse.
  • note_frontier() only writes one field. Consider keeping live_from in _qanat_state instead, so the scheduler never touches the file.
Bad F-07 fixed in 0.1.2 editor.py · api.py

One rejected edit breaks the console until you restart it

What I did

I sent one bad edit (a step reading a table that does not exist). The API said no. Then I did normal work.

What happened

[200] check before                 {"ok":true,"errors":[]}
[400] bad step                     step 'bogus' reads unknown stage 'nope'

[200] check AFTER                  {"ok":false,"errors":["step 'bogus' reads unknown stage 'nope'"]}
[400] a normal, valid edit         step 'bogus' reads unknown stage 'nope'
[400] deleting an unrelated step   step 'bogus' reads unknown stage 'nope'

does qanat.yaml contain 'bogus'?   False

Why it matters

save_step() calls upsert_step(project, step) first, and checks the project after. When the check fails, the change to the in-memory project stays. The file on disk is still clean, so the console and the file now disagree.

After that, the console reports a broken pipeline that does not exist, and every later edit fails with an error about a step you cannot find in the file. The only way out is restarting the server. One typo in a form takes the editor down.

How to fix it

  • In apply_and_save(), work on project.model_copy(deep=True). Change the copy, check the copy, and only keep it after save_project succeeds.
  • Add a test that sends a rejected edit and then a valid one. A test with only one request will never see this.
Medium F-08 still open backtest.py

A price of zero becomes an infinite return, with no warning

What I did

I set one entry price to 0, then to -1, and scored one period.

What happened

entry price 0.0   gross=inf      net=inf       notes=[]
entry price -1.0  gross=-226.09  net=-226.10  notes=[]

Why it matters

_price_frame() drops empty values and nothing else. Then score_period() computes p1 / p0 - 1. Many real feeds print 0 for a halt, a delisting, or a bad tick.

Both numbers go into totals_of() and into the equity curve. notes stays empty, so the report gives the reader nothing to notice. The negative one is worse than the infinite one: −22,609% for one period still looks like a number, so a bad run reads as a bad alpha instead of bad data.

How to fix it

  • In _price_frame(), drop prices that are zero or below, and count them.
  • Put that count in result.notes and in conditions. A line like "312 bad prices dropped from normalized.prices" saves an afternoon.
  • In score_period(), skip a period whose gross is not a finite number, and say why.
Medium F-09 fixed in 0.1.2 runner.py

sys.exit() in a step escapes the runner and leaves a job "running"

What I did

I wrote four steps, each one failing in a different way, and ran them.

What happened

sys.exit(3)         -> ESCAPED the runner: SystemExit 3
KeyboardInterrupt   -> ESCAPED
MemoryError         -> caught, status=failed   (correct)
RecursionError      -> caught, status=failed   (correct)

runs still marked 'running': 1

Why it matters

run_step() catches Exception. But SystemExit and KeyboardInterrupt come from BaseException, so they pass right by store.end_run().

The run row then stays running forever, and the console shows that job spinning forever. This is not a strange input: sys.exit() is common in scripts, and pressing Ctrl-C during qanat run does exactly the same thing.

How to fix it

  • Put the body of run_step and run_source in try/finally so end_run always runs. Or catch BaseException, record interrupted, and raise again.
  • Do the same in Scheduler._execute. It already frees the worker slot correctly — only the run row leaks.
Medium F-10 fixed in 0.1.2 retention.py

Retention deletes raw data, which the contract says is never edited

What I did

I set retention to 1s on a raw table and ran it.

What happened

policy '1s' on raw.daily_prices: 3360 -> 8 rows (removed 3352)
parse_duration('0d') -> 0:00:00   (accepted)

Why it matters

Rule 1 of the stage contract says: "raw is landed and never edited". Retention is not a step, so the rule does not cover it. It deletes rows with no question and no undo.

1s and 1d are one key apart. PUT /api/retention takes either one, and the scheduler applies it within 60 seconds. Raw is the one thing a replay cannot rebuild, because every other table is computed from it. So this is the only delete in the whole system that you truly cannot undo.

How to fix it

  • Refuse retention on a raw stage unless the project sets something like allow_raw_retention: true. Deleting the archive should be a clear choice.
  • Require a sensible minimum, such as 1 hour, and refuse 0.
  • Add qanat retention --dry-run. Make POST /api/retention/run return how many rows it would delete, unless you pass confirm=true.
Medium F-11 fixed in 0.1.2 project.py · runner.py

A loop between two steps passes the contract check

What I did

In one features stage I made mk_a read features.b, and mk_b read features.a. Each one needs the other first.

What happened

check ok?  True
errors:    []
run order: ['mk_a', 'mk_b', 'alpha_x']   # but mk_a needs mk_b to run first

Why it matters

runner.order() has this comment: "a cycle; qanat check is what reports it". But validate() has no loop detection at all. The stage rules catch loops that cross stages. A features stage is allowed to chain, so two steps inside one can point at each other.

order() gives up and adds the rest in file order. The run then fails at mk_a with LookupError: upstream table(s) not there yet. That is a runtime error about a missing table, for a graph the file should never have been allowed to describe. Two such steps with when: would also wake each other in a loop.

How to fix it

  • Sort the graph in validate() and name the loop: "mk_a → mk_b → mk_a".
  • Make order() raise an error on leftover steps, instead of returning an order it knows is wrong.
  • Refuse a when: chain that closes a loop, for the same reason.
Medium F-12 fixed in 0.1.2 backtest.py

Nothing limits how big a replay can be

What I did

I asked for one year of dates with rebalance: 1s.

What happened

2020-01-01..2020-01-02 every 1s:      86,401 dates in  0.1s, memory 105 MB
2020-01-01..2021-01-01 every 1s:  31,622,401 dates in 49.0s, memory 762 MB

Why it matters

That is only the list of dates. After that, every single date runs the whole pipeline again.

dates() is called before start_backtest(), so there is not even a run row to look at. The request just hangs while holding _replay, and every other backtest gets a 409 in the meantime.

rebalance is a free text field in the console form and in the MCP schema. So this is a likely typo, not an attack.

How to fix it

  • Count the dates with math first, and refuse above a limit. Say the number: "2020-01-01..2021-01-01 every 1s is 31,622,401 rebalances. Use a bigger gap, or pass --force."
  • Compare the gap with the price data. A gap smaller than the data itself cannot mean anything.
  • Record the run before the slow work starts, so a long replay is visible and can be stopped.
Medium F-13 still open scheduler.py · api.py

A job has no time limit and no stop button

What I did

I made a step that sleeps for ten minutes, and started it six times.

What happened

in-flight: {'tone'}
quiet(2s)? False
events: 'still running -- this tick was skipped'  x4
runs still marked 'running': 1

Why it matters

The skip logic is correct and the log is honest. The missing piece is a way out. The job holds its worker until the process dies.

The default is workers: 4. So four slow or stuck jobs stop the whole scheduler, and the only sign is warning messages nobody is reading. A REST source against a slow server can get there by itself: the 30-second httpx timeout covers a full hang, but not a slow trickle.

How to fix it

  • Add a per-job timeout: with a project default. When it runs out, end the run as timeout and free the worker.
  • Add DELETE /api/jobs/{id}/run and a stop button on the job panel.
  • Show "3 of 4 workers busy" in the /api/graph health block, so you can see it without reading the log.
Small F-14 still open runner.py

The lookahead check skips any table with no date column

What I did

I made a replayed step write a table with only symbol and score — no date column at all.

What happened

step written at as_of=2024-01-01 with no date column -> ok
time column found: None

Why it matters

_check_lookahead() returns early when store.time_column(ref) is None. So a table with no date column is simply not checked.

The check is described as catching "a step that built a timestamp itself, or read around the views with raw SQL". But a table with no date column skips the as-of views and the check at the same time — which is exactly what such a step would produce. I call it small because nothing is quietly wrong. It is a promise that is narrower than it sounds.

How to fix it

  • Warn in qanat check when a replayed step writes a table with no date column, and point at time_columns: as the answer.
  • Say it in the report's conditions: "3 tables here have no date column and were not lookahead-checked."
Small F-15 still open editor.py

store: can be moved anywhere, in one call

What I did

I sent PUT /api/store with an absolute path outside the project.

What happened

[200] {"store":"/tmp/qanat_elsewhere.duckdb","warnings":[]}

Why it matters

Same family as F-02, much smaller. It creates a folder, not a file that gets run. In practice the console just loses its data and shows an empty project. Still worth the same path check, and worth a confirmation — moving the store is not a normal settings change.

How to fix it

  • A file store must resolve inside the project folder, or be a clear postgresql:// address.
  • Return what the new store holds — "0 tables, 0 rows" — so a mistake is obvious right away.

What did not break

5 tests, nothing wrong

These matter too. They are the parts you do not need to touch.

Ideas beyond the fixes

8 features

Replay on copies

The most valuable change here. Let a replay work on copies in its own schema, not on the real tables. Then a replay stops being destructive: no crash damage, no empty console, no timed job reading old data, and two replays stop fighting.

fixes F-03, F-04, most of F-05

qanat doctor

One command, also run at startup. It finds stuck "running" rows, leftover views, tables emptied by a dead replay, and a qanat.yaml that will not parse. Report it, then offer to repair it.

makes F-03 and F-06 recoverable

One writer for the project file

Safe temp-and-rename, one shared lock, a .bak copy, and a version number on /api/project so an old console tab is refused instead of quietly overwriting somebody else's edit.

fixes F-06

Job control

Time limits per job, a stop endpoint, a stop button in the console, and worker use in the health block. Right now a stuck job is invisible unless you read the event log.

fixes F-13

Make check look inside

It checks shape well and content not at all. Add: table name pattern, script path inside the project, loop detection, tables with no date column, retention on raw. Each one is a few lines, and each one is a whole group of bugs that reach runtime today.

fixes F-01, F-02, F-10, F-11, F-14

Data quality in the report

End every replay with a short data block: bad prices dropped, symbols with no price, dates skipped and why. A backtest that quietly swallows bad ticks is the exact failure this tool exists to stop.

fixes F-08

Limits on a replay

Count the dates before building them, and refuse a silly grid with the number in the message. Save the run before the slow work, so it can be watched and stopped.

fixes F-12

A token when the console is not local

Local by default is right. Refuse a non-local --host without --token, and let the Docker entrypoint create one. Today the compose file opens an editor that can write and run files, with no password.

helps F-02, F-15
Order I would fix them in.
  1. F-01 and F-02. Both are input checks in validate() and save_step(). About an afternoon. These two let a bad string become a running one.
  2. F-07. Three lines: check a deep copy. It removes a whole group of console outages.
  3. Replay on copies. This is the real design change, and it pays for F-03, F-04 and F-05 at once.

The other two parts

separate runs

Part two · Clicking Every Button in Qanat

Everything above is the engine. The console is its own surface, so I drove it in a real browser: every button, the run form, the pipeline graph, the tab order, and what the page does when the server hangs and when it dies.

Twenty more problems. The one that connects to this report: I typed -500 into the commission box, and the form let me. The alpha then showed +145.64% in the strategy book. The backend does refuse a negative fee in qanat.yaml — but not in the per-run override the form sends.

Read part two →

Part three · Does the Backtest Tell the Truth?

The real job: connect a data source, build a graph, write an alpha, backtest it — tested against price data whose correct answer I worked out first. The money arithmetic matches to thirteen decimal places. But an alpha can still reach the future four different ways, and a loop between two steps multiplies your features by ten on every run while qanat check says the project is fine.

Read part three →