Metadata-Version: 2.4
Name: pyxtrace
Version: 4.0.0
Summary: Catch Python performance regressions in CI using deterministic operation counts instead of flaky wall-clock timing.
Author-email: Abhineet Saha <abhineetsaha.2004@gmail.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/AbhineetSaha/pyxTrace
Project-URL: Repository, https://github.com/AbhineetSaha/pyxTrace
Project-URL: Issues, https://github.com/AbhineetSaha/pyxTrace/issues
Keywords: profiling,performance,regression,regression-testing,ci,benchmark,tracing,profiler,regression-gate
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Education
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Software Development :: Debuggers
Classifier: Topic :: System :: Monitoring
Classifier: Topic :: Utilities
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: rich>=13.0
Provides-Extra: dev
Requires-Dist: ruff; extra == "dev"
Requires-Dist: mypy; extra == "dev"
Requires-Dist: pytest>=7; extra == "dev"
Dynamic: license-file

<div align="center">

<picture>
  <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/AbhineetSaha/pyxTrace/main/docs/logo/logo-dark.svg">
  <img alt="pyxtrace" src="https://raw.githubusercontent.com/AbhineetSaha/pyxTrace/main/docs/logo/logo-light.svg" width="380">
</picture>

<p><em>Catch Python performance regressions in CI — deterministically,<br/>so your benchmark gate stops crying wolf.</em></p>

<p>
  <a href="https://pypi.org/project/pyxtrace/"><img alt="PyPI" src="https://img.shields.io/pypi/v/pyxtrace?style=flat-square&logo=python"></a>
  <a href="https://github.com/AbhineetSaha/pyxTrace/blob/main/LICENSE"><img alt="License" src="https://img.shields.io/github/license/AbhineetSaha/pyxTrace?style=flat-square"></a>
  <a href="https://github.com/AbhineetSaha/pyxTrace/actions/workflows/ci.yml"><img alt="CI" src="https://img.shields.io/github/actions/workflow/status/AbhineetSaha/pyxTrace/ci.yml?style=flat-square&label=CI"></a>
</p>

<img src="https://raw.githubusercontent.com/AbhineetSaha/pyxTrace/main/docs/demo.gif" alt="An ordinary pytest test, a change on a branch, and pyxtrace diff main HEAD tying the regression to the changed function" width="820">

<p><a href="https://abhineetsaha.github.io/pyxTrace/">Website</a></p>

</div>

---

## Why

Wall-clock benchmarks are noisy enough to make a pull-request gate unreliable.
Run the same unchanged code seven times and time it, then count the operations
it performed on those same runs (`python benchmarks/overhead.py`):

```
wall-clock       : 2.06ms  2.06ms  2.20ms  2.04ms  2.06ms  2.05ms  2.17ms   → 3.1% spread
operation counts : 3196    3196    3196    3196    3196    3196    3196     → 0.0% spread
```

That 3.1% is on an idle local machine, which is the *best* case — a shared CI
runner with a noisy neighbour is worse, and smaller workloads are worse still
(sub-millisecond ones swing well past 15%). Meanwhile the counts are not
"stable," they are identical.

So on wall-clock you either set the threshold tight and get flaky failures, or
set it wide and miss real regressions. Most teams end up deleting the job.

PyxTrace counts **operations** — function calls and lines executed — instead of
measuring time. Those counts are byte-for-byte reproducible, so a 10% threshold
actually means something. Sampling profilers cannot do this: sampling is
non-deterministic by construction.

## Install

```bash
pip install pyxtrace
```

One dependency (`rich`). Python 3.10+.

## Use

### With your test suite: nothing to set up

```bash
pip install pyxtrace
pyxtrace diff main HEAD                # or `pyxtrace diff main` for uncommitted work
```

With no runs recorded, `diff` uses your pytest suite as the workload. It checks
out both commits into temporary git worktrees (your checkout is never touched),
runs the head's tests against both versions of the code, and counts each test
on its own. Only tests that passed on both sides are compared, so a test for a
feature the base does not have yet is left out rather than reported. Tests that
fail on the head are named, not silently dropped; pyxTrace is not your test
runner, so run pytest as its own CI job as well (or pass
`--fail-on-test-failure`).

Each regression is then tied to the diff. A function the change edited is
marked `← changed in this diff`; one that only got busier names the changed
functions on its call path. Changed functions the gate could not compare are
listed too, split by why: no test runs them, or only tests new in this change
do, so there is no baseline yet:

```
⚠  src/shop/orders.py::process_order  +120 calls to other functions  (same code, more work)
   ← on the call path of changed: src/shop/orders.py::handle_request
   process_order() runs 120x and makes 1 more call(s) to fetch_customer() each time

✗ FAIL — 3 function(s) regressed

1 of 2 changed function(s) were compared
⚠  1 changed function(s) no test runs, so the gate cannot see them:
     src/shop/orders.py::refund
```

`--require-covered` turns that last warning into a failure.

Only each test's own body is counted, not fixtures or imports. Each test file
runs in its own pytest process, several at once, so a test's counts never depend
on which other files ran before it; pytest-xdist itself is not used.

On the head, only the test files that can see the change are run: those whose
tests ran a changed function on the base (or failed there). A change outside
any function (an import, a constant, a class attribute), a method added to or
removed from a class (an override changes what unchanged calls run), or a
change to a non-Python file can reach any test, so it runs every file, and
says why. `--all-tests`
always runs every file.

A stress test can cost more traced time than the rest of the suite together
(on sqlglot, one test is 82% of all counted lines). A test that makes more than
5,000,000 calls (`--max-test-calls`) stops being counted and runs on untraced;
it is named in the report and not compared, so a regression that only it would
show is missed, never silently. The largest test on jinja makes 324,000 calls,
on rich 55,000.

### With a workload script

Record a run per commit, then compare commits:

```bash
pyxtrace examples/orders.py            # → .pyxtrace/<commit sha>.pyxt
# ... commit your change ...
pyxtrace examples/orders.py
pyxtrace diff HEAD~1 HEAD              # any git ref, or a .pyxt path
```

`diff` exits **1** when something regressed, so it drops straight into CI.

Before committing, compare your uncommitted work against any commit. A run with
uncommitted changes is filed separately, so it never overwrites the commit's
clean run:

```bash
pyxtrace examples/orders.py            # → .pyxtrace/<sha>-dirty.pyxt
pyxtrace diff main                     # main vs. the checkout as it is now
```

### What a caught regression looks like

The bundled example has a batched customer lookup. Set `PYXTRACE_NPLUSONE=1` and
it fetches each order's customer individually instead:

```
⚠  examples/orders.py::process_order  +120 calls to other functions  (same code, more work)
   process_order() runs 120x and makes 1 more call(s) to fetch_customer() each time
   Attributed to:
     examples/orders.py::fetch_customer   0 → 120 calls   new

⚠  examples/orders.py::Cursor.execute  +5,950% operations  (6 → 363)

⚠  examples/orders.py::fetch_customer  new operations  (0 → 240)
   Attributed to:
     examples/orders.py::Cursor.execute   0 → 120 calls   new

✗ FAIL — 3 function(s) regressed
```

`process_order` ranks first: it is where the extra work was introduced.
`Cursor.execute` and `fetch_customer` merely run more often as a consequence.

Every line there is arithmetic over exact counts, and there is no confidence
score because there is nothing to be uncertain about. Note also what it does
**not** say. "Runs 120x and makes 1 more call each time" is a measurement;
calling it an N+1 would be a conclusion, and one input size cannot support it.
The same shape is produced by a function that simply does more work per call.
Reading it is your job, and you have the counts to do it with.

### A single run

```bash
pyxtrace examples/orders.py
```

```
                 pyxTrace – examples/orders.py
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━━┓
┃ function                            ┃ calls ┃ lines ┃ own ms ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━━┩
│ examples/orders.py::serialize       │   120 │   480 │    0.2 │
│ examples/orders.py::process_order   │   120 │   360 │    0.2 │
│ examples/orders.py::handle_request  │     1 │   245 │    0.1 │
│ examples/orders.py::fetch_orders    │     1 │   122 │    0.0 │
└─────────────────────────────────────┴───────┴───────┴────────┘
```

`calls` and `lines` are deterministic. `own ms` is timing and varies run to run —
it is shown for orientation and is never what `diff` gates on.

## In CI

With a test suite, the gate is one step after installing your project's test
dependencies:

```yaml
- uses: actions/checkout@v4
  with: { fetch-depth: 0 }                  # the base commit has to be present
- run: pip install pyxtrace pytest -e .
- uses: actions/cache@v4                    # the base side, counted once per test file
  with:
    path: .pyxtrace/tests
    key: pyxtrace-${{ runner.os }}-${{ github.event.pull_request.base.sha }}
    restore-keys: pyxtrace-${{ runner.os }}-
- run: pyxtrace diff ${{ github.event.pull_request.base.sha }} HEAD --stability-runs 2
```

The base side's results are kept in `.pyxtrace/tests/`, one entry per test file,
named by a hash of everything the counts depend on: the base commit, that test
file, every conftest and test helper, the Python version, the installed
packages and pyxTrace itself. A second diff against the same base runs only the
head; a pull request that edits one test file counts only that file again on the
base. An entry can never be reused for anything it does not match, so restoring
an older cache is safe.

`--stability-runs 2` runs the head's tests twice and leaves out any test whose
counts differ between the two, so a test driven by the clock cannot fail the
gate. It lists those tests, and costs one more run of the head.

With a workload script, run files are keyed by commit sha, so a PR gate records
the base commit and the head and diffs them by name:

```yaml
- uses: actions/checkout@v4
  with: { fetch-depth: 0 }                  # the base commit has to be present
- run: pip install pyxtrace
- run: pyxtrace benchmarks/workload.py      # HEAD → .pyxtrace/<sha>.pyxt
- run: |
    git checkout -q ${{ github.event.pull_request.base.sha }}
    pyxtrace benchmarks/workload.py         # base → .pyxtrace/<sha>.pyxt
    git checkout -q -
- run: pyxtrace diff ${{ github.event.pull_request.base.sha }} HEAD --threshold 10
```

Because the counts are exact, the base run can be recorded once and cached
(`actions/cache` on `.pyxtrace/`) instead of re-run per PR: the same commit on
the same Python version always produces the same bytes. Put the Python version
in the cache key — see *Limits*.

## Options

| Command | What it does |
|---|---|
| `pyxtrace run SCRIPT` | Profile → `.pyxtrace/<commit sha>.pyxt` (`run` is optional; `-o` overrides) |
| `pyxtrace diff BASE [HEAD]` | Compare two commits (or `.pyxt` paths); exit 1 on regression. Without `HEAD`, compares against the current checkout. With no runs recorded, runs the pytest suite on both, starting from where `HEAD` branched off `BASE` |
| `pyxtrace diff BASE...HEAD` / `BASE..HEAD` | As in git: from where the two diverged, or from `BASE` itself |
| `--require-covered` | With the test suite: fail when a changed function is run by no test |
| `--fail-on-test-failure` | With the test suite: fail when a test fails on the head |
| `--stability-runs N` | With the test suite: run the head's tests N times and leave out, and name, tests whose counts differ (default 1) |
| `--all-tests` | With the test suite: run every test file on the head, not only those that ran the changed code |
| `--max-test-calls N` | With the test suite: stop counting a test after N calls and name it instead of comparing it (default 5,000,000; 0 for no limit) |
| `--file-timeout SECONDS` | With the test suite: stop a test file still running after this long and name it instead of comparing its tests (default 600; 0 for no limit) |
| `-j N`, `--jobs N` | With the test suite: test files run at once, each in its own process (default: one per CPU) |
| `--no-merge-base` | Compare against the tip of `BASE` even when `HEAD` does not contain all of it |
| `--threshold N` | Percent growth that fails the gate (default 10) |
| `--min-ops N` | Ignore growth below N operations (default 10), so a 2→3 line change is not an alarm |
| `--root DIR` | Profile this directory instead of the git checkout the script is in, for an installed or out-of-tree package |
| `--version` | Print the installed version |

`diff` exits **0** on a pass, **1** on a regression, and **2** on a usage error
or an unreadable run file. `run` exits with the script's own status; a script
that fails records nothing, since a partial count is not a baseline.

## What keeps the counts exact

Determinism is the whole product, so these are handled rather than documented
away:

- **String hashing.** Python randomises it per process, which reorders every
  `set` and `dict` of strings and changes how many lines a loop over one runs.
  The `pyxtrace` command re-launches itself with `PYTHONHASHSEED=0`. The Python
  API cannot, and warns instead.
- **Threads.** Every thread counts into its own counters, merged at the end, so
  no update is lost even on free-threaded Python. Threads the script leaves
  running are waited for exactly as `python app.py` waits for them. Daemon
  threads still running at exit are reported, since their counts are partial.
- **Randomness and garbage collection in tests.** When `diff` runs your test
  suite, `random` is seeded the same way before every test, and garbage
  collection is held off while a test runs, so it cannot fire at a point that
  depends on earlier tests. On jinja that took the tests whose counts moved
  between identical runs from 5 of 911 to none. Still yours to fix: other
  random generators (NumPy, Faker), and work driven by the clock or by timer
  threads. On rich, 1 to 3 of 948 tests still move between runs by under
  0.1%, because log lines carry the time and a live display refreshes on a
  timer. `--stability-runs 2` finds such tests and leaves them out.
- **Names.** Functions are keyed by qualified name (`Cursor.execute`), so two
  classes' `__init__` never merge into one entry. Python 3.10 has no qualified
  names on code objects, so pyxTrace reads them from the source instead.

## Overhead

Measured on `fib(22)`, a deliberately call-heavy worst case
(`python benchmarks/overhead.py`):

| | `fib(22)` | Real code |
|---|---|---|
| **PyxTrace (default profiling path)** | **~49x** | **~9x** |
| `cProfile` (stdlib, for reference) | ~8x | ~3.4x |

`fib(22)` is a deliberate worst case: it does almost nothing between calls, so
the per-event cost has nothing to amortise against. The real-code column is
sqlglot parsing 838 SQL statements, which is the number to plan around for a
workload script.

With your test suite, on Python 3.12+ the counting uses `sys.monitoring`
(PEP 669), which switches code outside the root off for good instead of
filtering it on every call. Each side of the diff, measured on whole suites:

| Suite | pytest alone | settrace (3.10, 3.11) | sys.monitoring (3.12+) |
|---|---|---|---|
| jinja (911 tests) | 2.5 s | 14.2 s | 10.8 s |
| rich (948 tests) | 5.2 s | 12.5 s | 9.8 s |

The two tracers count calls differently (settrace counts a generator resuming
as a call), so a run file records which one made it and runs from different
tracers are never compared. `PYXTRACE_TRACER=settrace` forces the slower one,
for example when another profiler already holds `sys.monitoring`.

## Limits

Stated plainly, because a profiler that hides these is worse than none:

- **Work inside C is invisible.** PyxTrace counts Python operations. A
  regression that lives in a C extension, a regex, NumPy, a database driver, or
  any I/O wait produces no extra Python operations and will not be caught. A
  measured example: moving the same workload onto a backtracking regex made it
  230,000x slower with byte-identical counts, and the gate passed. This is the
  mirror image of what sampling profilers cannot do, and it is the first thing
  to check before adopting PyxTrace. It is strongest on pure-Python CPU-bound
  code: parsers, template engines, serializers, business logic.
- **Subprocesses are not traced.** Threads and `asyncio` are: coroutines run on
  the event loop's thread, and worker threads are covered via
  `threading.settrace`. A thread started before tracing begins is not.
- **The workload has to be deterministic itself.** Unseeded `random`, the
  clock, or network responses that steer control flow will move the counts.
  Seed them in the benchmark script.
- **Only code under one root directory** is profiled — the git checkout the
  script is in by default (the script's own directory outside git), or
  `--root DIR`. Anything outside it is skipped, and so is any virtualenv or
  `site-packages` inside it. A run that captured nothing but the entry script
  says so.
- **Counts are exact per Python minor version, not across them.** The
  interpreter decides which lines emit events, and that changes between
  releases: 3.12 inlined comprehensions, so upgrading from 3.11 moves line
  counts with no code change. Record both sides of a diff on the same Python;
  each run file records its interpreter and `diff` warns when they differ.
- **Operation counts are a proxy for cost, not a measure of it.** An algorithmic
  change can cut operations and still be slower. Gate on counts, then confirm
  with a real timer.
- **Not for production.** This is a deterministic tracer for CI. To profile a
  live process, use [py-spy](https://github.com/benfred/py-spy).

## When to use something else

| You want | Use |
|---|---|
| Profile a running production process | [py-spy](https://github.com/benfred/py-spy) |
| Find where memory is allocated | [memray](https://github.com/bloomberg/memray) |
| A readable one-off wall-clock profile | [pyinstrument](https://github.com/joerick/pyinstrument) |
| A timeline / flame-graph UI | [VizTracer](https://github.com/gaogaotiantian/viztracer) |
| **Know whether this PR made things slower** | **PyxTrace** |

## Contributing

Contributions are welcome, and the most useful one right now is trying pyxtrace
on a real project and [reporting what happened](https://github.com/AbhineetSaha/pyxTrace/issues/new?template=field_report.yml).

```bash
git clone https://github.com/AbhineetSaha/pyxTrace.git
cd pyxTrace
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"

pytest -q                        # tests
python benchmarks/overhead.py    # overhead + determinism gate
```

[CONTRIBUTING.md](https://github.com/AbhineetSaha/pyxTrace/blob/main/CONTRIBUTING.md) explains how the code is organised and the
few rules that keep the counts exact. Look for
[good first issues](https://github.com/AbhineetSaha/pyxTrace/labels/good%20first%20issue)
to get started. This project follows the [Contributor Covenant](https://github.com/AbhineetSaha/pyxTrace/blob/main/CODE_OF_CONDUCT.md).

## License

MIT — see [`LICENSE`](LICENSE).
