flight
Your traceback tells you where Python died. flight tells you why — a crash you can open, ask "why is this value what it is?", and re-run. The black box for Python.
Why it exists
The real debugging loop is: add prints → try to reproduce → fail to reproduce → add more prints → wait for it to happen again. A traceback tells you where a program died, almost never why. flight records what actually happened, so the bug report writes itself — and it's cheap enough to leave on in production.
How it works
One path in, one file out, many ways to read it.
your program
│ sys.monitoring (PEP 669) — native Rust callbacks, no Python frame
▼
in-process · hot path · Rust
┌──────────────────────────────────────────────────────────┐
│ is-this-mine? cache → lock-free ring buffer → clock │
└───────────────────────────┬──────────────────────────────┘
│ uncaught exception · or capture()
▼
object graph (aliasing ↔) + frames + source + exception chain
▼
crash.flight msgpack + zstd · versioned · shareable
┌─────────────────┼──────────────────────────┐
▼ ▼ ▼
flight inspect browser viewer (WASM) why · diff · fix · what-if
- Record cheap enough to leave on. CPython's
sys.monitoringcalls straight into native Rust — no Python callback frame, no FFI hop. The hot path takes no lock. - On a crash, write a black box — not a trace. The object graph is serialized
identity-first, so the same object in two frames is one, marked
↔; every local, the exception chain and the source come too. - Read it anywhere. The
.flightis the spine: the CLI, the offline WASM viewer, and every analysis only ever speak to the file — never to a live process.
What you can do with a .flight
See the crash
Frames, locals, object graph and aliasing (↔) — the whole last moment.
Backward slice
Ask why is this value what it is? and get the chain of writes back to the origin.
Compare two runs
The first point a good run and a bad run diverged — the cause boundary.
Find the commit
Which commit introduced the bug — by fingerprint, or by replaying against each commit.
The boundary
The exact value at which a recorded input flips the failure on and off.
A proven patch
Propose a fix and verify it by replaying the recorded tape with it applied.
Rewrite the past
Change a past value, re-execute over the same recorded world, see the counterfactual.
Fleet mode
A dashboard aggregating thousands of black boxes, with regression detection.
What it is (and isn't)
It is a scoped, post-mortem recorder with a first-class viewer, evolving toward
time-travel debugging. It is not an APM, a live debugger (that's pdb), or a
profiler.
Get started
A prebuilt wheel, no Rust toolchain needed. Install, leave it on, let a crash write its own report.
1 · Install
$ pip install pyflight
Wheels ship for Linux, macOS (Intel + Apple Silicon) and Windows. The native core is built against CPython's stable ABI (abi3), so one wheel per platform covers Python 3.12, 3.13 and every later 3.x. Optional extras:
$ pip install 'pyflight[viewer]' # the Textual TUI viewer
$ pip install 'pyflight[crypto]' # at-rest encryption of a .flight
2 · Leave it on
Three lines. On an uncaught exception, a .flight is written automatically.
import flight
flight.install() # cheap enough to leave on
... # run your program
3 · Or wrap a script without editing it
$ python -m flight run yourscript.py
...
ZeroDivisionError: division by zero
[flight] recorded flight-57275.flight
4 · Read the black box
$ python -m flight inspect flight-57275.flight
exception : ZeroDivisionError: division by zero
frames : 4 (crash first)
#0 compute_average (crash.py:26)
numbers = list[0] ↔ # empty! this is the bug
#1 summarize (crash.py:32)
data = list[0] ↔ # the SAME empty list, aliased in
The ↔ marks an object that is the same across frames. The black box
diagnosed the bug — and it never had to run twice.
.flight opens with nothing
installed in the offline browser viewer — drop the file
and read the crash right there.From source (contributors)
$ python -m venv .venv && . .venv/bin/activate
$ pip install maturin pytest textual
$ maturin develop --release # compiles the Rust core, installs editable
Documentation
The complete surface: the Python API, the CLI, configuration, and the
.flight format.
Python API
| Call | What it does |
|---|---|
flight.install(**cfg) | Start recording via sys.monitoring + install the crash excepthook. Returns the active Config. |
flight.uninstall() | Restore the interpreter to its previous state. |
flight.capture(path=None) | Write a .flight right now — e.g. inside an except block for a handled error. |
flight.stats() | {'total_events', 'threads', 'codes', 'ring_capacity'} for the live recorder. |
flight.read(path) | Open a .flight; query .crash(), .frames, .recording(), the object graph and aliasing. |
with flight.record() as rec: | Scope time-travel: record every state write as a MUTATION; rec.watch(obj) tracks writes into a container. |
with flight.deterministic(path): | Record all non-determinism (time/random/uuid/I/O/threads) so the run replays bit-for-bit. |
flight.replay(path, fn) | Re-run fn feeding the recorded values back; raises ReplayDivergence if control flow differs. |
flight.time_travel(path) | A reverse-debugger cursor: step_back(), find_first("running > 100"), state(). |
flight.minimize(path, fn) | ddmin over the tape → the minimal set of recorded values your bug actually needs. |
flight.why(path, var=...) | A dynamic backward slice: how did this value come to be? |
flight.what_if(path, fn, Override(...)) | Edit a past value and re-execute forward over the deterministic tape (Python 3.13+). |
flight.correlate(service=...) / flight.link(path) | Stamp the W3C trace context / link an upstream black box for cross-service crashes. |
@flight.adapter("pkg.Type") | Register an adapter so your big types are summarized (shape/dtype/preview), not dumped. |
Command line
| Command | What it answers |
|---|---|
flight run script.py | Run a script with recording on; write a .flight on crash. |
flight inspect f.flight | The crash: frames, locals, object graph, aliasing. |
flight view f.flight | The interactive Textual TUI (needs the [viewer] extra). |
flight timeline f.flight | A scope recording's mutation timeline (--var / --who). |
flight why f.flight --var x | The backward slice for a value. |
flight diff a b | The first point two runs diverged (--html for a shareable page). |
flight bisect … | Which commit introduced the bug (passive corpus or active replay). |
flight generalize f.flight | The boundary at which a recorded value flips the failure. |
flight fix f.flight | Propose and verify a patch over the recorded tape. |
flight explain f.flight | Heuristic root cause (--llm / --prompt). |
flight repro f.flight [--pytest] | A self-verifying reproduction script / regression test. |
flight debug f.flight | Reverse debugger: a DAP server (VS Code / PyCharm) or --find on the CLI. |
flight fingerprint f.flight | A stable dedup id by frame + state. |
flight trace ./crashes | The cross-service crash graph (by trace id). |
flight encrypt / decrypt | Seal a .flight at rest with AES-256-GCM ([crypto] extra). |
flight ci .flight | A Markdown root-cause comment for a red CI. |
flight serve ./store | Fleet mode: a collector + index + dashboard. |
pytest --flight | A .flight for every failing test (the plugin). |
Configuration
Pass to flight.install(...) or build a flight.Config:
| Key | Meaning |
|---|---|
ring_capacity | Events kept in the per-thread ring buffer. |
output_dir | Where a .flight is written. |
record_lines | Per-line detail (default off — call/return/exception granularity). |
record_returns | Keep PY_RETURN events (on by default; off halves events on call-heavy code). |
deny_prefixes / force_include | The policy that keeps stdlib and site-packages out of the recording. |
capture_deadline_ms / capture_max_bytes | The crash-capture time and size budget (a giant object can't hang or blow up capture). |
max_str / max_container / max_depth / repr_limit | Per-value limits for the object-graph serializer. |
scrub_patterns | Names whose values are redacted before any byte is written (P5). |
overhead_slo / daemon / correlation | Production knobs: the SLO governor, the crash-surviving supervisor, trace correlation. |
The .flight format
Versioned, append-only and truncation-tolerant: a header, typed blocks (msgpack + zstd)
and an optional footer index. A .flight holds the process environment (META), the
event ring, the exception chain, every frame with its locals, the identity-preserving object
graph, and the source of every file involved — so the values still make sense on another
machine. New readers read old files; old readers skip unknown blocks.
The five inviolables
- P1 — Primum non nocere. The recorder never crashes the program it records.
- P2 — Honest, bounded overhead. <5% target; an SLO governor can defend a ceiling.
- P3 — The format is the spine. Engine and viewer only speak through it.
- P4 — Every phase is useful on its own.
- P5 — Privacy by design. Redaction of sensitive fields is a day-one feature.
Examples
Copy-paste starting points for each capability.
Capture a handled error
import flight
flight.install()
try:
risky()
except Exception:
flight.capture(path="handled.flight") # a black box without crashing
raise
Time-travel a scope
with flight.record() as rec:
cache = {}
rec.watch(cache, name="cache") # track writes into this container
running = 0
for it in [5, 3, 8]:
running = running + it # a local rebind, recorded as a MUTATION
cache[it] = running # a container write, recorded too
$ python -m flight timeline --var running flight-scope-*.flight
history of local 'running' (4 writes):
#3 tt.py:8 running = 0
#11 tt.py:10 running = 16 # ← how it evolved, step by step
Deterministic replay
import flight, time, random
def work():
return time.time(), random.random()
with flight.deterministic("run.flight"):
original = work()
assert flight.replay("run.flight", work) == original # identical, though time moved on
Ask why a value is what it is
$ python -m flight why crash.flight --var numbers
numbers ([]) — how this value came to be:
#0 compute_average crash.py:26 parameter — comes from the caller (aliased below)
↔ the SAME object as 'data' in summarize (frame #1)
↰ is datasets['evening'] in main (frame #2) (dict[3])
⇒ root: 'datasets' was dict[3] in main (frame #2)
Compare two runs
$ python -m flight diff run_ok.flight run_fail.flight
comparing nondets: diverged at step 7 (12 steps compared)
random.random answered differently
left : random.random [f] 0.8313
right: random.random [f] 0.1174
A patch that proves itself
$ python -m flight fix crash.flight
+ if not numbers:
+ return 0.0
verification over the recorded tape:
✓ the crash no longer reproduces
✓ no boundary divergence (time/random/IO identical)
⇒ FIX VERIFIED
What-if: rewrite the past
# The recorded run crashed because `data` was empty. What if it weren't?
wi = flight.what_if("run.flight", compute, flight.Override("data", [2, 4], line=42))
print(wi.render())
# before: raised ZeroDivisionError: division by zero
# after: returned 2067.0 → the change alters the outcome.
A black box for every failing test
$ pytest --flight
FAILED test_orders.py::test_refund - IndexError: list index out of range
------------------------------- Flight recording -------------------------------
black box: .flight/test_orders.py_test_refund.flight
A .flight per HTTP 500
from flight._web import FlightWSGI, FlightASGI
app = FlightWSGI(app) # Flask / Django / Pyramid …
app = FlightASGI(app) # FastAPI / Starlette / Quart …
Learn flight
A path from your first recording to production time-travel. Each lesson stands on its own — do them in order, or jump to the one you need.
Record your first flight
Goal: turn a crash into a file, without changing how you run anything.
Add flight.install() at the top of your program, or run it with
python -m flight run script.py. Trigger the bug once. On the uncaught exception,
flight writes a .flight next to you and prints its name. That file is the whole
last moment of the process — you never have to reproduce the bug again.
Read a black box
Goal: find the bug in the recording, not in your head.
Run flight inspect crash.flight. Read the frames crash-first, look at each frame's
locals, and watch for the ↔ marker — it means the object is the same
across frames (aliasing), which is where surprising bugs hide. Prefer a picture? Drop the file
into the browser viewer.
Watch state evolve (scope time-travel)
Goal: see how a variable got its value, write by write.
Wrap the suspicious code in with flight.record() as rec: and
rec.watch(...) the containers you care about. Every write is recorded with its
exact line. Afterwards, flight timeline --var x shows the full history of
x; --who cache shows who wrote into a container and when.
Make a flaky bug deterministic
Goal: reproduce a "fails 1% of the time" bug every single time.
Run inside with flight.deterministic("run.flight"):. flight records the clock,
randomness, uuids, file/pipe/socket reads and the thread lock order. flight.replay(path,
fn) then re-runs your function feeding those values back, bit-for-bit — and raises
ReplayDivergence at the exact step if control flow ever differs.
Step backward through time
Goal: put a breakpoint in the past.
flight debug scope.flight --find "running > 100" jumps to the write where a value
first crossed a threshold. Or start the DAP server (flight debug scope.flight) and
VS Code / PyCharm show real Step Back and Reverse buttons driven by the
recording — no live process.
Compare & shrink
Goal: isolate the cause to the smallest possible thing.
flight diff ok.flight fail.flight points at the first place two runs diverged.
flight.minimize(path, fn) then runs delta debugging over the tape until only the
load-bearing values remain — "your bug needs only these 3 of the 500 recorded values".
Ask "why" and "what if"
Goal: reason about causes and counterfactuals.
flight why --var x builds the backward slice of writes and aliasings that produced
a value. flight.what_if(...) overwrites a past value and re-executes over the same
recorded world, telling you whether the outcome changes, diverges, or never reached the point
(Python 3.13+).
Leave it on in production
Goal: a recorder you trust under load.
flight.install(overhead_slo=0.03, daemon=True) caps overhead as an SLO and keeps a
supervisor that promotes a checkpoint to a black box even on SIGKILL/OOM.
flight.correlate(service=...) stamps the W3C trace context so
flight trace can stitch a cross-service crash together.
Prove and verify a fix
Goal: close the loop — from crash to a patch you can trust.
flight fix crash.flight proposes a unified diff and then re-executes the
deterministic tape with it applied: crash gone and no divergence → VERIFIED.
Turn it into a permanent guard with flight repro --pytest.
Black box viewer
Drop a .flight file — or two, to compare. It's parsed in your browser
by the Rust reader compiled to WebAssembly. Nothing is uploaded, nothing installed.
The creator
Gabriel Lima
Builds systems from scratch to understand them to the metal — from CUDA kernels and a micro-PyTorch to a Raft implementation and this recorder.
Why flight was built this way
flight is a from-scratch project with a deliberate split: a lock-free ring buffer and the
.flight writer live in Rust (via PyO3), fed by CPython's
sys.monitoring, while the reading experience is treated as a first-class half of the
product. It grew phase by phase — foundation, the full black box, a TUI viewer, scope
time-travel, deterministic replay, a reverse debugger, comparison & delta debugging, an
intelligence layer, a production-grade black box, a whole ecosystem, and finally what-if
debugging and a fleet dashboard — each one useful on its own.
The bets behind it
- Instrumentation can be cheap.
sys.monitoring(PEP 669) finally makes it affordable to leave a recorder on, because a callback can disable itself at a cold location. - Debugging is 50% engine, 50% reading. So a viewer isn't an afterthought — it's a planned half of the work, and it runs offline in your browser.
- The file is the viral vector. A shareable
.flightmeans "open this and you'll see everything" — which is exactly what this page is for.