Metadata-Version: 2.4
Name: flowharness-ci-runner
Version: 0.1.0
Summary: flowharness CI runner — detect changed AI surfaces, resolve the committed suite lockfile, and load the local gate policy (local, deterministic, zero network)
License-Expression: Apache-2.0
License-File: LICENSE
Classifier: Development Status :: 3 - Alpha
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Dist: flowharness-core==0.1.0
Requires-Dist: flowharness-evaluation==0.1.0
Requires-Dist: flowharness-inspection==0.1.0
Requires-Python: >=3.12
Project-URL: Homepage, https://github.com/flowharness-ai
Description-Content-Type: text/markdown

# flowharness-ci-runner

The CI runner distribution (`flowharness-ci`). CI-PR6a ships the **read-the-world**
slice: detect changed AI surfaces, resolve the committed suite lockfile, and load the
local gate policy. Execution + scoring (the `CaseExecutor` port, replay cassettes, and
metrics) land in CI-PR6b.

> **Note:** these packages currently install top-level modules named `core`, `inspection`,
> `evaluation`, and `portability`. Prefer `uvx` (isolated environments) over `pip install`
> into a shared venv until the 0.2 module-namespace rename lands.

## `vibe-check`

```
flowharness-ci vibe-check [--base REF] [--suite PATH] [--gate-policy PATH]
                          [--executor replay|live] [--format table|json|github]
                          [--upload] [--baseline REF]
                          [--external-findings LIST] [--external-tool file=tool]
```

- **No AI-surface change** -> prints `no AI-surface changes` and exits `0` (the fast path:
  most PRs never start the job).
- **A surface changed** -> resolves `.flowharness/suite.lock.json` + the gate policy, then —
  because execution is not yet wired (CI-PR6b) — emits `executor pending (CI-PR6b)` and exits
  `1` (`NEEDS_HUMAN`). It is never a silent PASS.
- `--executor live` is accepted but refused with a clear message (CI-PR9). `--upload` is now
  **wired** (CI-PR8) — see [Tier 1 ingestion](#tier-1-ingestion---upload-ci-pr8) below.
- `--external-findings a.sarif,b.json` merges third-party scanner findings into the verdict through
  the `[external]` gate (and `--external-tool a.sarif=snyk_agent_scan` pins a tool per file) — see
  [`ingest-findings`](#ingest-findings) below (PR-P1.4).

## Files it reads (fixtures-are-the-contract)

- `.flowharness/suite.lock.json` — schema `flowharness/ci-suite/v1`. Unknown/missing `schema`,
  oversize input, or malformed JSON **fail closed** with a typed error.
- `.flowharness/gate-policy.toml` — optional. **Absent -> the safe default -> `NEEDS_HUMAN`**
  (never auto-PASS an unconfigured repo). Commit one to opt into automation.

## Tier 1 ingestion (`--upload`, CI-PR8)

`--upload` POSTs the signed governance run to a FlowHarness instance. It is **telemetry, never
judgment**: the exit code is ALWAYS the local verdict (PASS 0 / NEEDS_HUMAN 1 / FAIL 2), and no
upload outcome — success, failure, or misconfiguration — ever changes it. Every outcome is one
`notice:` line on **stderr**; stdout stays the byte-frozen artifact (so `--format json` output is
**byte-identical with and without `--upload`**).

**Configuration (env only — nothing secret is ever a CLI flag or on argv):**

- `FLOWHARNESS_TOKEN` — the raw HMAC key. Read from the environment ONLY (there is no `--token`
  flag by construction); it is never logged, never in a repr, never in an error, on any path.
  **Absent or empty ⇒ Tier 0**: the run succeeds locally and the upload is skipped with one
  notice — never an error.
- `FLOWHARNESS_API_URL` — the instance base URL. An **allowlist**: only `https://`, plus
  `http://` when the host is exactly `localhost` / `127.0.0.1` / `::1`. Every other scheme
  (`ftp:`, `file:`, `data:`, `ws:`, schemeless) is refused before any request is built.
- `GITHUB_RUN_ID` / `GITHUB_RUN_ATTEMPT` — supply the idempotency nonce
  (`runner_nonce = {run_id}-{attempt}`). Absent (a local run) ⇒ skipped with a notice.

**Tier ladder (D-CI8-7).** No `--upload` flag → nothing uploads. `--upload` with no token →
**Tier 0** skip. `--upload` with a token → **Tier 1**: exactly one signed POST to
`{FLOWHARNESS_API_URL}/v1/evaluation/runs`, then one of:

| Outcome | Meaning | Retried? |
| --- | --- | --- |
| `verified` | the server accepted and verified the signature | — |
| `duplicate` | the server already ingested this run (suite × candidate × cell × nonce, HTTP 409) | no (idempotent success) |
| `rejected` | the server stored the run `rejected` (HTTP 200) | **no — see below** |
| `failed` | transport error / 5xx / 429 after bounded retries, or a 4xx / 3xx | up to 3 attempts (5xx/429/transport only) |
| `skipped` | Tier 0, missing URL/nonce, or a refused URL | — |

**A `rejected` landing is the CORRECT, tamper-evident outcome — not a client bug.** The server
verifies an HMAC over exactly the bytes it re-derives from the report; if the signature does not
match, it stores the run `rejected` (HTTP 200) as an audit trail rather than erroring. The
runner **reports this faithfully and does NOT retry-to-force it** — retrying a `rejected` run
would defeat the tamper-evidence. If you see `rejected`, the token or the report bytes are the
place to look, not the runner.

**Transport hardening (security).** The token is the signing key, so the transport dials exactly
one address: it **refuses all redirects** (a 3xx is terminal — the bearer is never re-sent to a
`Location` target, cross-origin included) and **never honors ambient proxies** (`http_proxy` /
`HTTPS_PROXY` / `ALL_PROXY` and macOS system proxy config are ignored — the key travels only
toward `FLOWHARNESS_API_URL`). Each attempt has a 10 s idle timeout; the CI job's own timeout is
the total-wall backstop.

**Supply-chain residual (stated honestly, D-CI8-9).** In the composite action, scope
`FLOWHARNESS_TOKEN` / `FLOWHARNESS_API_URL` to the **vibe-check step's `env:`** — never the job
`env` (which would hand the HMAC key to every step, including PR-controlled setup steps). A
composite action's own internal steps (`astral-sh/setup-uv`, the `gh` comment step) still
inherit the invoking step's process env; that exposure is not client-fixable. Mitigations: pin
the action by SHA, and note the key's blast radius is one ingest endpoint, not a repo credential.

## `ingest-findings`

```
flowharness-ci ingest-findings --input findings.sarif
                               [--tool snyk_agent_scan|aguara|medusa|inkog|sarif_generic]
                               [--format table|json|github]
```

Converts ONE third-party scanner artifact (SARIF or the tool's native JSON) into FlowHarness
findings and renders them — **standalone, no gate is applied**, so the exit code is `0` (converted)
or `2` (the artifact was malformed / oversize / unreadable — fail-closed). Use it to inspect what a
scanner would contribute before wiring it into `vibe-check --external-findings`.

- **Tool detection.** SARIF auto-detects the tool from every `tool.driver.name` (all `runs[]` are
  read); `--tool` overrides it. Native (non-SARIF) JSON has no driver name, so `--tool` is required.
- **`--format`** — `table` (the collapsed `▸ Third-party checks` section), `json` (the byte-exact
  `flowharness/ci-findings/v1` artifact), or `github` (the `::error file=…,line=…::` annotation stream).
- **Hardening.** A 256 KiB size cap before read, JSON-only parsing (no YAML), an iterative depth guard
  against nesting bombs, and `redact_path` over every path/message before it enters a finding.

Findings map to `core.Severity` through **`external_severity_map.v1`** — the single authority in
`core`, shared with product/06 evidence intake later: SARIF `level` none/note→INFO, warning→WARN,
error→ERROR, with a SARIF `security-severity` value **≥ 9.0 lifting any finding to CRITICAL**;
per-tool native codes/levels for Snyk (`E###`→ERROR, curated-critical→CRITICAL, `W###`→WARN),
Aguara/Medusa (low/medium/high/critical), and Inkog (structural classes→ERROR); an unknown or absent
severity → WARN + a `severity_unmapped` marker. Every finding carries `third_party` provenance and
the `risk.external` dimension, and the path **never** touches `safety_score_bp`.

### The `[external]` gate (`vibe-check --external-findings`)

`vibe-check --external-findings a.sarif,b.json [--external-tool a.sarif=snyk_agent_scan]` merges the
same artifacts into a run and gates them through an `[external]` table in
`.flowharness/gate-policy.toml`:

```toml
[external]
max_critical = 0              # any external CRITICAL beyond this FAILs a configured repo
max_error = 5                 # beyond this ⇒ NEEDS_HUMAN
unmapped_severities = "needs_human"   # or "ignore"
```

The external verdict combines with the eval verdict by precedence — `max(eval, external)` under
`FAIL > NEEDS_HUMAN > PASS`. **An absent `[external]` section is the safe default:** a CRITICAL
external finding on an unconfigured repo raises `NEEDS_HUMAN`, never `FAIL` (a FAIL requires an
explicitly committed policy). The external gate always runs, even on the fast path, so external
findings can never be silently bypassed; and a `vibe-check` run without `--external-findings`
produces a byte-identical `flowharness/ci-run/v1` artifact (additive wire).

## Consumer packaging — the `vibe-check` composite action (§2.7)

Drop this workflow into a consumer repo. The action renders the run as a **sticky PR comment**
(create-or-update, keyed by a hidden `<!-- flowharness-vibe-check -->` marker, so re-pushes edit
ONE comment) and propagates the verdict exit code.

```yaml
# .github/workflows/vibe-check.yml
name: FlowHarness Vibe Check
on:
  pull_request:                       # the pull_request trigger ONLY — never its _target twin (S4)
    paths: ["CLAUDE.md", ".claude/**", ".cursor/**", ".cursorrules", "AGENTS.md",
            ".continue/**", ".opencode/**", "playbook/**", ".flowharness/**",
            "**/crew*.py", "**/agents.y*ml", "**/tasks.y*ml"]
permissions:
  contents: read
  pull-requests: write                # sticky comment only
jobs:
  vibe-check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@<40-hex-sha>   # v6 — SHA-pin third-party actions (S7)
        with: { fetch-depth: 0 }              # diffscope needs the merge base
      - uses: flowharness-ai/vibe-check-action@<40-hex-sha>   # v1 — SHA-pin
        with:
          executor: replay                    # 'live' requires a provider key (CI-PR9)
          upload: ${{ secrets.FLOWHARNESS_TOKEN != '' }}   # Tier 1 iff a token is configured (CI-PR8)
        env:                                   # secret + URL on THIS step only — never job env, never with:
          FLOWHARNESS_TOKEN: ${{ secrets.FLOWHARNESS_TOKEN }}
          FLOWHARNESS_API_URL: ${{ vars.FLOWHARNESS_API_URL }}
```

**Tier 1 upload (CI-PR8).** `upload: ${{ secrets.FLOWHARNESS_TOKEN != '' }}` opts a repo into
signed ingestion only once a token secret is set; the action forwards `--upload` to the CLI. Scope
`FLOWHARNESS_TOKEN` / `FLOWHARNESS_API_URL` to the **`env:` of this `uses:` step** — never the job
`env` (which would hand the HMAC key to every step of the job, PR-controlled setup steps included),
and never `with:` (which would echo the secret into the action's declared input surface). See the
[Tier 1 ingestion](#tier-1-ingestion---upload-ci-pr8) section for the tier ladder, outcomes, and
the honestly-stated composite-action residual (the action's own internal steps inherit this step's
env).

The `paths:` list mirrors the AI-surface set `inspection.domain.surfaces` classifies (spec
§2.7's own example): the exact root files (`CLAUDE.md`, `.cursorrules`, `AGENTS.md`), the
surface trees (`.claude/`, `.cursor/`, `.continue/`, `.opencode/`, `playbook/`), the
governance dir, and the orchestration globs. If the classifier gains a surface, add it here
too — a path missing from this list means changes to that surface never trigger the workflow
(the runner itself would still classify them on any run that does start).

**S7 — pin every third-party action by commit SHA**, with the tag in a trailing comment. The
in-repo action already pins `astral-sh/setup-uv` this way; do the same for `actions/checkout`
and the published action above.

**How the action runs the CLI (D-CI7-30).** `astral-sh/setup-uv` installs only the `uv`/`uvx`
binaries — it does NOT install this package's console script — so every CLI step in the action
runs `uvx --from flowharness-ci-runner flowharness-ci …`, the §2.7 fetch-and-run form. The
`--from` clause is required because the distribution name (`flowharness-ci-runner`) differs
from the script name (`flowharness-ci`); a bare `uvx flowharness-ci` would resolve the wrong
package. **Publication dependency:** `flowharness-ci-runner` must be published to PyPI before
the action is runnable anywhere — until then the vibe-check step fails at resolution (a
non-zero capture, so never a silent PASS). At first publication, pin the package
(`--from flowharness-ci-runner==X.Y.Z`) so a SHA-pinned action cannot float to a newer release.

**gh version floor: `>= 2.47`.** The action lists comments with plain `gh api --paginate`, which
for a top-level array endpoint (`/issues/{n}/comments`) splices the pages into ONE flat array —
MEASURED on gh 2.89.0 against a local three-page server: `json.loads` yields 5 flat items across
3 pages, and `--paginate` auto-sets `per_page=100`, so the marker comment is found even off the
first page. **Do NOT "fix" this onto `--slurp`:** `gh help api` documents `--slurp` as
returning *an array of all pages*, i.e. it NESTS the wire shape into an array OF pages, so the
selector would find zero comments and every push would post a fresh spam comment.

### Sticky-selector residual (accepted, documented — D-CI7-22)

The selector requires `user.type == "Bot"`, `user.login == "github-actions[bot]"`, the marker on
the body's exact first line, and the vibe-check heading on the next non-empty line. All four are
**content**, so this is a **collision filter, not an authenticity check**: `github-actions[bot]`
is the SHARED identity of every workflow in the repo, and a sibling workflow that echoes
attacker-controlled text could reproduce these lines. The realistic (accidental-echo) class is
removed; a deliberate same-bot forgery is not, and no stateless rule can close it. Impact if it
ever happens is bounded to **defacement + suppression of the report — never a verdict change**
(`post-comment` always exits 0). **Hardening option:** run the action under a **dedicated GitHub
App or PAT-backed bot identity**, at which point `user.login` becomes a genuine authenticity
signal — trade-off: a secret to manage, and it is unavailable on fork PRs (which have no token).

### Governance-file symlink residual (S6 — keep them REAL files)

S6 warns and forces a full run when the diff touches `.flowharness/suite.lock.json` or
`.flowharness/gate-policy.toml`. If either governance file is committed as a **symlink**, `git
diff` reports the link *target* path — which the derived S6 matcher does not watch — so a PR
could weaken the suite or the gate threshold with **no warning line and no forced run**. Keep
**both** `.flowharness/suite.lock.json` **and** `.flowharness/gate-policy.toml` as real files
under `.flowharness/`; if a shared layout is required, point `--suite` / `--gate-policy` at the
real location so the matcher watches it directly. (Cassette evidence is unaffected — the case-ref
reader fails closed on any symlinked component.)

## Boundaries

Zero network on the default path (no `httpx`); `git` and `gh` are the only local processes
shelled, each behind an injected runner so the core is offline-testable — and Python never reads
a token (`gh` authenticates from its own env). The runner reuses `core`, `evaluation.domain`
(the `GatePolicy` VO), and `inspection` (surface classifier + exit-code table) only — statelessly,
with no UnitOfWork or web/ORM stack (enforced by the import-linter fences in the root pyproject).

## SDK boundary (ci-spec §5.1 / G10) — the CI runner is NOT the SDK

The CI runner (`flowharness-ci`, this package) and the FlowHarness **Python SDK**
(`flowharness`, under `sdks/python/`) are deliberately separate products, and **the CI runner is
not the SDK**:

- **Different wire.** The runner speaks a **JSON governance-run ingestion** wire — one
  HMAC-signed `POST /v1/evaluation/runs` (`--upload`, above), and otherwise runs fully offline.
  The SDK speaks the **ConnectRPC machine wire** to the management API (typed RPCs, binary
  framing) — the Phase-1 client under `sdks/python/`.
- **Different audience.** The runner is a **governance gate for CI pipelines**, invoked as a CLI
  (`uvx --from flowharness-ci-runner flowharness-ci vibe-check`). The SDK is for **application
  code** that embeds audit calls programmatically.

Programmatic embedding of the runner is out of scope: it exposes a CLI, not a stable client API.
If you want to call the platform from code, use the SDK (`sdks/python/README.md`); if you want to
gate a PR, use this runner.
