Metadata-Version: 2.4
Name: xilinx-test-runner
Version: 0.1.0
Summary: Test runner for FPGA verification: build the bitstream, keep the variants, run tests on the board, compare against the baseline.
Author: Xilinx Test Runner contributors
License: MIT
Project-URL: Homepage, https://github.com/shchuchkin-pkims/xilinx-test-runner
Project-URL: Documentation, https://github.com/shchuchkin-pkims/xilinx-test-runner/tree/main/docs
Project-URL: Issues, https://github.com/shchuchkin-pkims/xilinx-test-runner/issues
Keywords: fpga,verification,vivado,xilinx,regression,test runner,rtl,hardware in the loop
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console :: Curses
Classifier: Environment :: X11 Applications :: Qt
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Software Development :: Testing
Classifier: Topic :: Scientific/Engineering :: Electronic Design Automation (EDA)
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: tomli>=1.2.0; python_version < "3.11"
Provides-Extra: gui
Requires-Dist: PyQt6>=6.2; extra == "gui"
Provides-Extra: yaml
Requires-Dist: PyYAML>=5.4; extra == "yaml"
Dynamic: license-file

<div align="center">

<img src="assets/logo.svg" width="96" alt="">

# Xilinx Test Runner

**A test runner for FPGA verification work: build the bitstream, keep the old
ones, run the tests on the board, compare against the baseline.**

One tool for the whole loop, with a console, a terminal interface and a window
over the same run.

[![tests](https://github.com/shchuchkin-pkims/xilinx-test-runner/actions/workflows/tests.yml/badge.svg)](https://github.com/shchuchkin-pkims/xilinx-test-runner/actions/workflows/tests.yml)
[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
[![Python 3.10+](https://img.shields.io/badge/python-3.10%2B-blue.svg)](https://www.python.org/)
[![Dependencies: standard library](https://img.shields.io/badge/dependencies-standard%20library-green.svg)](#requirements)
[![Docs: English and Russian](https://img.shields.io/badge/docs-english%20%7C%20русский-lightgrey.svg)](docs/)

</div>

![The tests tab](docs/images/window-tests.png)

<sub>The window follows the desktop theme. The same run on a dark one:
<a href="docs/images/window-dark.png">window-dark.png</a>.</sub>

---

## What it does

Verifying a design on an FPGA prototype is the same loop every day: get the
current sources, build a bitstream, run tests in simulation and on the board,
compare the behaviour against the documentation, fix something, then prove that
nothing which used to work has broken.

Every step of that already has a tool. What is usually missing is the thing
that holds them together and remembers what happened. This is that thing.

It is not a simulator, not a replacement for UVM and not a continuous
integration system. It sits in front of those and keeps the results.

## Installation

Python 3.10 or newer. Nothing else is required; the window needs Qt.

```bash
git clone https://github.com/shchuchkin-pkims/xilinx-test-runner
cd xilinx-test-runner
./bin/xtr --help              # runs straight from the checkout

pip install .                 # or install it properly
pip install ".[gui]"          # with PyQt6, for the window
```

Every release also carries a wheel and a source archive on its
[releases page](https://github.com/shchuchkin-pkims/xilinx-test-runner/releases).

`python3 -m xtr` works anywhere the `xtr` command does.

## Quick start

A demo project is included. It builds a fake bitstream with a shell script, so
it runs with no vendor tools installed at all:

```bash
cd examples/demo_project
xtr list                      # what is here
xtr run                       # run it in the console
xtr artifacts build           # "synthesise" and store a variant
xtr run --accept              # accept these results as the baseline
xtr --gui                     # or the window
```

For your own project:

```bash
cd /path/to/project
xtr init                      # writes xtr.toml and an example test
xtr doctor                    # is everything where the config says it is
xtr run
```

`xtr --gui` and `xtr --tui` also open with no project at all, and can point
themselves at one from the File menu.

## How a test is written

A test is **any executable file**. Its exit code is the verdict, and a few
prefixed lines carry everything else. A script that has never heard of this
runner works as it is.

```bash
#!/bin/bash
# TEST: nack_timing
# GROUP: Hardware
# DESC: NACK must start inside the window the standard allows
# NEEDS: board                     # one board: never two of these at once
# REQUIRES: bitstream up to date   # skip me if the build is stale
# RETRY: 1                         # flaky connection; the report will say so
# TARGET: lab_pc                   # run me on the machine with the board
# COVERS: ISO7816-3 8.3

command -v xsdb >/dev/null || { echo "SUMMARY: no debugger here"; exit 2; }

echo "PROGRESS: 30 programming the device"
etu=$(measure_etu)

echo "METRIC: nack_window $etu etu tol=0.5"
echo "CHECK: nack inside the window $( [ "$etu" -le 12 ] && echo pass || echo fail )"

[ "$etu" -le 12 ] || { echo "SUMMARY: NACK at $etu etu, limit is 11.5"; exit 1; }
echo "SUMMARY: NACK at $etu etu, inside the window"
```

### Five verdicts, because two are not enough

| Exit code | Verdict | Meaning |
|---|---|---|
| `0` | `PASS` | checked, behaves as expected |
| `1` | `FAIL` | checked, the design is wrong |
| `2` | `SKIP` | could not check: no board, no licence, no tool |
| `3` | `ERROR` | the test setup broke, not the design |
| `4` | `KNOWN` | a known open defect, reproducing exactly as recorded |

A missing cable must not paint the run red, and an open defect must not hide
under green. `KNOWN` has its own counter and does not fail the run.

### Lines a test can print

| Line | Purpose |
|---|---|
| `SUMMARY: ...` | the one line that lands in the results table |
| `PROGRESS: 40 message` | moves the progress bar |
| `TRACE: ...` | verbose output, hidden until asked for, always kept |
| `METRIC: etu 66673 cycles tol=50` | a number compared between runs |
| `CHECK: name pass` | a named verdict inside a test |
| `ARTIFACT: /path/file` | a file to keep with the result |
| `COVERS: ISO7816-3 8.3` | which requirement this test checks |

The full contract is in [docs/writing-tests.md](docs/writing-tests.md).

## What the runner adds

**Bitstreams are objects, not a file that happens to be there.** Every build is
stored as a named variant together with a hash of the sources it was built
from. Switching between variants is a file copy, so comparing a fix against the
previous build takes seconds instead of another synthesis run. Because the
provenance is a hash of the content, copying a variant around cannot make an
old build look current. A check based on file dates can, and that mistake is
how results from two different designs end up mixed in one report.

**Regression means comparison, not a demand for perfection.** Tests report
measurements and named checks; the runner keeps the accepted values and reports
what moved. A test that used to fail and still fails in exactly the same way is
reported as a known defect rather than noise. A test that starts passing is
pointed out rather than silently absorbed.

**The board is usually on another machine.** A test can declare where it runs,
and the runner drives it over SSH. Stopping a run kills the whole remote
process tree, which is what keeps a cancelled run from leaving a debugger
attached and the cable locked for the next hour.

**Random stimulus stays reproducible.** Each run has a seed, and each test gets
a stable seed derived from it. Both are recorded, so a failure found with
random data can be replayed with `--seed`.

**Reports carry their context.** Tool versions, source revision, active
bitstream, machine name, seed. A report without those cannot be interpreted a
month later.

Also included: dependencies between tests, setup tests that stop a section
cleanly, retries that are always visible in the report, exclusive resources
that hold even between separate runs of the program, a configuration matrix
with a comparison report, and reports in text, JSON, JUnit XML and HTML.

## Configuring a project

One file in the project root. TOML is read by the standard library; YAML works
too if PyYAML is installed.

```toml
[project]
name  = "uart_7816 on EBAZ4205"
tests = "tests"

[tools.vivado]
path = "/opt/Xilinx/2025.1/Vivado/bin/vivado"   # or omit it and let it be found

[targets.lab_pc]                                 # the board is on another machine
type     = "ssh"
host     = "lab@10.0.0.2"
identity = "~/.ssh/id_ed25519_lab"
root     = "/home/lab/project"                  # the same tree, over there

[artifacts.bitstream]
store       = "fpga/bitstreams/variants"
active      = "fpga/bitstreams"
files       = ["design_1_wrapper.bit"]
extra_files = ["design_1_wrapper.xsa"]
sources     = ["rtl/**/*.v"]                     # what the provenance hash covers
build       = "bitstream"

[flows.bitstream]
plugin  = "vivado"
project = "fpga/EBAZ4205/EBAZ4205.xpr"
tcl     = "fpga/scripts/build.tcl"
output  = "fpga/EBAZ4205/EBAZ4205.runs/impl_1/design_1_wrapper.bit"
tail    = ["fpga/EBAZ4205/EBAZ4205.runs/impl_1/runme.log"]

[vcs.rtl]
path = "rtl"
```

From that, the runner generates the steps a project would otherwise script by
hand: check whether the bitstream matches the sources, build a new one, switch
between variants, show the source revision, fetch and pull it, and verify that
the tools are installed where the configuration says.

Machine specific paths belong in `xtr.local.toml` next to it, which is not
committed. Every key is documented in
[docs/configuration.md](docs/configuration.md).

## The interfaces

All three drive the same run and share the same state.

| Command | Interface |
|---|---|
| `xtr run` | console; exits non zero on failure, for scripts and pipelines |
| `xtr --tui` | terminal interface; needs only a terminal, works over SSH |
| `xtr --gui` | window: tests, artifacts, run log, history |

The window has four tabs. **Tests** shows the tree, the live output and the
measurements next to the accepted values. **Artifacts** shows the stored
bitstreams, which one is active and whether it still matches the sources.
**Run log** collects everything every test printed, verbose lines included,
which is where a long synthesis run is watched. **History** lists previous runs
and how a measurement moved between them.

![The artifacts tab](docs/images/window-artifacts.png)

## Command line

```bash
xtr                                     # run everything, print a table
xtr run sim --jobs 4                    # only what matches "sim", four at a time
xtr run --matrix bitstream=fix,old hw   # the same tests on two builds, compared
xtr run --seed 1234                     # replay a random run exactly
xtr run --accept                        # record these results as expected
xtr artifacts list                      # what is stored, what is active
xtr artifacts activate old_build
xtr history trend etu                   # is that measurement drifting
xtr issue                               # a ticket draft from the last failure
xtr doctor                              # check tools, targets and paths
xtr selftest                            # the runner's own tests
```

## A complete example on real hardware

[`examples/pynq_z1`](examples/pynq_z1/) is a full project for a Digilent
PYNQ-Z1: a written specification with seven numbered requirements, the RTL, a
simulation testbench, and tests that run on the board. It is programmable logic
only, so no software has to be built for the processor and no SD card is
involved; the host talks to the design over the same JTAG cable that programs
it.

```
ok    program              31.3 s  device configured, VIO answering, counter running
ok    bitstream up to date  0.0 s  built from the current sources (9eda4dd157f06f4b)
ok    syntax                1.0 s  2 file(s) compile clean, 0 warning(s)
ok    step_counter_tb       5.1 s  16 checks, all as specified
ok    lfsr                 27.7 s  16 values match the model, the seed reload works
ok    steps                27.7 s  reset, edge stepping, up, down, hold and wrap behave as specified
```

## Documentation

| Document | Contents |
|---|---|
| [Getting started](docs/getting-started.md) | installation, first run, adopting scripts you already have |
| [Writing tests](docs/writing-tests.md) | exit codes, output protocol, header fields, environment, seeds |
| [Configuration](docs/configuration.md) | every key, and which of the four files it belongs in |
| [Artifacts](docs/artifacts.md) | bitstream variants, provenance, the configuration matrix |
| [Remote execution](docs/remote.md) | SSH targets, shared trees, stopping remote processes |
| [Baseline and history](docs/baseline.md) | how a regression is decided, when to accept, trends |
| [Architecture](docs/architecture.md) | the three layers and how to extend them |
| [Recipes](docs/recipes.md) | Vivado, XSim, Cadence HAL, a board over SSH, CI |
| [Firmware results](examples/firmware/) | self describing results from C running on the target |

Russian translations of the main documents are in [docs/ru/](docs/ru/), and the
overview is in [README.ru.md](README.ru.md).

## Requirements

| | |
|---|---|
| Python | 3.11 or newer, standard library only; on 3.10, `tomli` for TOML |
| Operating system | Linux (POSIX process groups and file locks) |
| Window | PyQt6 or PyQt5, optional |
| YAML configuration | PyYAML, optional; TOML needs nothing |
| Vendor tools | whatever your tests use; the runner only has to find them |

Nothing is downloaded at run time and no usage data is collected.

## Design decisions

These are deliberate, and they are the reason the tool stays small.

* **The core knows nothing about FPGAs.** Everything vendor specific is either
  configuration or a test. If that stops being true, the tool is no longer
  portable to the next project, and a shell script would do the job instead.
* **No external test manifest.** Test metadata lives in the header of the test
  file, so adding a test means adding a file, and nothing can drift out of sync
  with a file that does not exist.
* **Skipping is not failing.** Absent hardware, an unlicensed tool and a dead
  connection are three different situations, and none of them is a defect in
  the design.
* **A retry that hides instability is worse than no retry.** The report always
  states that a test needed a second attempt.
* **Accepting a baseline is a deliberate act.** A baseline that updates itself
  compares a run against itself.
* **No job queue, no agents, no scheduler.** Those products exist. This one
  runs on the machine in front of the hardware.

## Contributing

Bug reports, recipes for tools nobody has tried yet, and patches are all
welcome. The suite that guards the behaviour above runs with no dependencies:

```bash
xtr selftest        # 94 tests
```

[CONTRIBUTING.md](CONTRIBUTING.md) describes the layering rule and what a
change is expected to come with.

## License and trademarks

MIT. See [LICENSE](LICENSE).

This project is independent and is not affiliated with, endorsed by or
sponsored by AMD, Cadence, Siemens or Intel. Product names are used only to
describe which third party tools the runner can drive. No vendor artwork is
included: the icon and the logo are original work. See
[TRADEMARKS.md](TRADEMARKS.md).
