---
name: systematic-debugging
description: Debug methodically by forming and testing hypotheses instead of guessing — reproduce, isolate, find root cause, fix, and verify. Use when a bug is non-obvious, a test fails for unclear reasons, results look wrong but don't crash, or you've already tried one or two fixes that didn't work. Especially for "silent" quant bugs where numbers are wrong but nothing errors.
---

# Systematic debugging

A discipline for finding the *root cause* rather than patching symptoms. The cardinal rule: **change one thing at a time, and verify each step against evidence.** Random edits that "might fix it" usually mask the real bug or create new ones.

## The loop

1. **Reproduce reliably.** Get a deterministic, minimal repro. Fix the random seed. Shrink the input until the bug still happens with the smallest possible case. If you can't reproduce it, you can't fix it — instrument until you can.
2. **Observe, don't assume.** Read the actual error/traceback fully. Print/inspect real values (shapes, dtypes, index, NaN counts, head/tail) at the boundary. Don't trust your mental model of what the data "should" be.
3. **Form a hypothesis.** State a specific, falsifiable claim: "the merge drops rows because the date keys have mismatched dtypes." A hypothesis predicts what you'll see if it's true.
4. **Test the cheapest hypothesis first.** Run the one experiment that confirms or kills it. One variable at a time.
5. **Binary-search the failure.** Bisect: in the data pipeline, the code history (what changed?), or the input range. Halve the search space each step.
6. **Fix the root cause, not the symptom.** Once found, fix the actual cause. If you're tempted to add `.fillna(0)` or `try/except: pass`, ask whether that hides the real problem.
7. **Verify the fix.** Confirm the repro now passes AND you understand *why*. Add a regression test (pair with [[property-based-testing]]) so it can't silently return.

## When you're stuck

- **Re-read the inputs.** Most "impossible" bugs are wrong/misaligned input data, not wrong logic.
- **Rubber-duck the assumptions.** List everything you *believe* is true, then verify each one — the bug lives in a belief that's actually false.
- **Bisect history.** If it worked before: what changed? `git log`/`git diff` the suspect files (note: this repo is not a git repo per the env, so diff against backups/notes).
- **Reduce, don't add.** Strip the program down to the minimal failing core rather than piling on logging everywhere.
- **Check the boundaries.** Off-by-one, empty input, single element, all-equal, NaN/inf, timezone, int vs float, index alignment.

## Quant-specific silent-bug hunt (numbers wrong, no crash)

These don't raise — they just give you a wrong backtest. Check explicitly:

- **Look-ahead bias.** Is the signal at date *t* using only data available at *t*? Check every `shift`, `rolling`, `merge_asof`, and resample boundary.
- **Index/date misalignment.** pandas aligns on index during arithmetic — a silent reindex can inject NaN or shift values by one day. Print `.index` on both operands.
- **Survivorship.** Are delisted names dropped from history? That inflates returns.
- **NaN handling.** Does a NaN become 0 somewhere (e.g. `.fillna(0)` before a sum/weight)? That's often a hidden bug, not a fix.
- **Double-counting / unit errors.** Returns vs log-returns, pct vs decimal, daily vs annualized, gross vs net.
- **Join cardinality.** Did a join silently fan out (1→many) or drop rows? Compare row counts before/after.

## Anti-patterns (stop if you catch yourself doing these)

- Changing several things at once, then not knowing which fixed it.
- Adding broad `try/except` to make the error "go away."
- Re-running hoping for a different result without changing anything.
- Trusting a comment/variable name over the actual runtime value.
- Declaring it fixed without a repro that now passes.
