runcomposer.

Selection is the hard part. Execution is somebody else's.

Every test corpus of any size has the same problem: you can run everything, or you can run one thing, but running exactly the right subset — and being able to prove later which subset that was — is awkward. Tags help, until the tag query lives in a CI job parameter that nobody can reconstruct three weeks later.

runcomposer splits the problem in two. Composing the selection is a first-class, reviewable act that produces a document. Executing it is somebody else's job — and deliberately so. The document is the whole interface: an executor needs to understand three fields, not an API.

What it is not: a CI system, a scheduler, or a test framework. There is no cron, no cross-instance queue, no ambition to replace the thing that already runs your builds. Compose, dispatch, ingest, browse — that is the entire product.

Sixty seconds, if you'd rather see it

The same argument as the rest of this page, drawn out: the awkwardness of selecting precisely, what changes when the selection becomes a document, and where that document can travel afterwards.

69 seconds, narrated

Or watch someone use it

Four and a half minutes over the shoulder, in the actual interface: picking out tonight's tests area by area, leaving the known-broken ones behind, watching verdicts arrive one at a time until the run says complete — and the next morning, rerunning the single test that failed. One idea about how it works underneath, at the point where the story needs it.

4 minutes 36 seconds, narrated · captions

From a tag to a completed run

Six steps, and the run's lifecycle state is computed at every one of them — never guessed.

1 · Catalog

A test source enumerates your tests as items — an opaque stable id plus tags — and content-hashes the catalog. A Robot Framework source mints ids from longnames; a manifest source takes a plain JSON list, so a pytest corpus arrives as node ids.

2 · Compose

Navigate the taxonomy — a curated tree over tag patterns, data rather than code — build a filter, and watch the preview recompile as you type. The filter language is small and lossless: a bare word is a literal tag, prefix:Checkout- is sugar for an anchored regex, regex: is the escape hatch, and the three operators are AND, OR, NOT.

3 · Materialize

The filter is compiled against the catalog snapshot and the resulting item list is written into the document. This is the step that makes a run reproducible: the filter stays for provenance, but the embedded list is what actually runs.

4 · Dispatch

Hand the document to a runner, or export it and hand it to anything at all. Each hand-off mints a dispatch; re-running the same spec is a new dispatch under the same run.

5 · Deliver

Results come back as a bundle carrying a correlation marker. Push it over HTTP, drop it in a watched directory, or pass it on the command line. Every delivery is content-hashed.

6 · Complete

When every declared shard has delivered, the run completes and its verdicts become history — which is where the next selection starts.

The run spec

Versioned YAML, JSON-isomorphic and accepted either way, with a published JSON Schema. The core sections are generic and closed. Exactly one section is open — runner — and the core never looks inside it.

runspec: "1.0"

run:
  id: "01JZ9GQ2W8KJ3F6M4P5R7T9V"      # minted at compose time; the correlation key
  title: "Payments regression without quarantined tests"
  created_at: "2026-07-06T09:14:03Z"
  labels:                              # free-form provenance; stored, never interpreted
    requested_by: "alex"

selection:
  tag_filter:                          # kept for provenance — not re-compiled by executors
    op: AND
    items:
      - op: OR
        items: ["Payments", "prefix:Checkout-", "regex:^Cart(V2)?$"]
      - not: "prefix:Quarantine-"
  materialized:                        # THE authoritative executed set
    item_ids: ["Tests.Payments.Visa Payment Succeeds", "Tests.Payments.Declined Card Shows Error", …]
    count: 33

source:
  provider: "robotframework"            # this source mints ids from Robot longnames
  snapshot: "sha256:21ee74fed1fcf5e6…"   # integrity check — drift is detectable

results:
  expect: [{ format: "robot-output-xml" }]
  deliver: "api"
  token: "rct_…"                       # per-run ingest token

runner:                                # the ONE open section — plugin's own vocabulary
  robot-pool:
    suite_root: "tests/"
    partitions: ["env1", "env2"]
    variables: { STAGE: "test" }

Because the item list travels inside the document, a dispatched spec is self-sufficient. The machine that runs it needs no access to your catalog, your database, or your network. And because the catalog snapshot travels with it, an executor can tell whether the corpus moved underneath the plan.

A consumer must accept any spec of the same major version: unknown fields inside known sections are ignored, unknown top-level sections warn, a higher major is refused outright. runcomposer validate checks a document against the schema; adding --for-dispatch additionally demands everything execution requires.

Bring your own run agent

This is the part worth understanding, because it is what the whole design exists to make possible. runcomposer does not need to reach your test machines. It does not need credentials on them, a network route to them, or an agent it controls. It hands out a document; something on the other side runs it and sends a bundle back, in its own time.

The executor contract

Whatever runs the spec — our runner, your script, a CI job, a colleague on a laptop — owes exactly three things. Everything else in the document, including the entire runner section, may be ignored.

• Execute exactly the materialized item list. Not the filter. Executors do not re-compile selections — that is what makes the run reproducible.
• Check for drift before executing. If the live corpus hash differs from the one in the spec, refuse by default. With an explicit override, run the intersection and report the difference as skips.
• Reference the run id in the results. That, plus a shard label when the work was split, is all correlation needs.

What "your own agent" actually looks like

The consumer is deliberately tiny: a single, self-contained, standard-library-only Python file. There is nothing to install on the executing side — copy it in with your code, or fetch it from a release, where it ships as its own asset. It reads the spec as JSON, so a machine with nothing but python3 can run it.

# 1 — here: compose and freeze the plan
runcomposer spec 'Regression' --title "Nightly" \
    --format json -o spec.json --export

# 2 — there: one vendored file, no install, your own runner command
python3 runcomposer_exec.py spec.json --out results \
    --command "./run-tests.sh {ids_file} {out_dir}"

#      → results/output.xml            (whatever your tests produced)
#      → results/runcomposer_run.json  (run id + spec hash — the marker)

# 3 — here: the bundle comes back however you like, then
runcomposer ingest results
runcomposer runs
#      RUN ID                      STATE     RESULT
#      01JZ9GQ2W8KJ3F6M4P5R7T9V    COMPLETE  FAIL

Step 3's "however you like" is the point. The bundle is a directory. Commit it to a results branch, rsync it, drop it on a share, attach it to a build — the marker inside carries the correlation, so the transport is your business and none of ours.

The marker is what makes this safe. It carries the run id and a hash of the exact spec bytes that were handed out. A bundle whose hash does not match, or that claims a run nobody dispatched, does not quietly enter your history — it lands in a visible quarantine inbox for a human to attach, promote, or discard.

Results, from anywhere, more than once

Dispatch and result-return are fully decoupled. Three transports feed one pipeline: an HTTP push guarded by the run's own ingest token, a watched file-drop directory for git-transported or air-gapped bundles, and plain runcomposer ingest from the command line.

Redelivery is not an edge case — pollers re-poll, CI retries webhooks, git re-pulls. So the rules are explicit: a byte-identical bundle is a no-op. A different bundle for the same shard replaces that shard's verdicts, last writer wins. There is no monotonic merge, because a correction has to be able to turn a FAIL back into a PASS.

Per-item verdicts are PASS FAIL SKIP ERROR, each with duration, message, artifacts, and an attempt counter — so a retry inside a run is representable and flakiness is derivable rather than stored as a guess.

Rerun what failed

Once runs accrue, history becomes a selection source. Ask for the failures of the latest completed run and they resolve at compose time into a normal, static, reproducible spec — with a provenance record of how the list was derived.

$ runcomposer spec --from-history 'failed@latest' --title "Rerun"

selection:
  item_ids: ["Tests.Payments.Expired Card Is Rejected Loudly"]
  derived_from:
    - provider: "history"
      query: { run: LATEST, verdicts: [FAIL] }
      resolved_run_id: "01JZ8ZZ…"

"Latest" means latest completed — a run still awaiting results is not silently treated as an answer. And on a fresh install these features are simply dark, which the tool says out loud rather than returning a confidently empty list.

Everything specific is a plugin

The core knows about items, tags, selections, specs, runs, and verdicts. It knows nothing about Robot Framework, pytest, Jenkins, or XML. Those live in plugins, loaded either by entry point or by a plain module: path in your config — no environment variables, no magic discovery.

The boundary is enforced, not merely intended: a guard test fails the build if core code so much as mentions a framework's vocabulary. Native result names always travel through the source that owns the id space, so no normalisation quirk can leak inward.

Run it in a minute

Self-hosted, single config file, sqlite by default. The web UI ships pre-built inside the wheel in English and German, so evaluating it needs no Node toolchain.

git clone https://github.com/StochasticEntropy/runcomposer
cd runcomposer

# the guided demo — 60 tagged tests, a full compose → run → rerun loop,
# seeded into ./runcomposer-demo/ (rm -rf it to undo)
pipx run --spec . runcomposer demo

# the web UI and API on http://127.0.0.1:8100
pipx run --spec . runcomposer serve

# or in a container
docker build -t runcomposer . && docker run -p 8100:8100 runcomposer

# executing Robot Framework in-process needs one extra
pip install ".[robot]"

There is no published package yet. runcomposer 0.1.0 is installed from a clone — that is what every command on this page assumes. Nothing is on PyPI or npm, so pip install runcomposer will not find it; watch releases for when that changes.

The CLI is the whole product without the browser. All eleven commands: demo to boot the sample world, catalog to list the corpus and its snapshot, compile to preview a selection, spec to compose one, validate to check a document against the schema, dispatch to run it, ingest to take results back, runs to browse, export to hand results to another tool, gc to keep it all bounded, and serve for the web UI and API.

ADOPTING.md walks through pointing it at your own corpus and your own machines; DESIGN.md is the architecture and the reasoning behind every decision here.

Six ways out of this page

Everything described above lives in one public repository. It is a young project with a single maintainer, so the issue tracker is the front door for anything that does not fit.