Playbook 03 · CI

CI-gated agent releases

Goal: no prompt, policy, tool, or model change reaches production without passing its worst eval cohort. Two workflows do it: a PR gate that blocks the merge, and a weekly cron that catches provider drift you didn't cause. Both are plain GitHub Actions running two Zolva commands that exit 1 on failure, which is the entire integration.

Prerequisites #

• Your agent repo: agents/, evals/, tools.py, and an importable AgentApp (Step 1).
• A model provider key stored as a GitHub Actions secret (OPENAI_API_KEY or ANTHROPIC_API_KEY), never in the repo: Zolva's config loader rejects inline credentials anyway.
• Judge-graded cohorts also need a judge: pass --judge-provider and --judge-model. Cohorts graded with exact, contains, or tool_called need no judge.

Step 1 · Repo layout and the --app entry point #

zolva eval imports your app as module:attr from the working directory, so expose one:

"""CI entry point: `zolva eval evals/ --app app:app` imports this."""

import tools  # noqa: F401  registers @zolva.tool functions

from zolva import AgentApp

app = AgentApp.from_config("agents/")

your-repo/
├── agents/
│   ├── collections.yaml
│   ├── collections.md
│   └── policies/collections.yaml
├── evals/
│   ├── core.yaml            # grader: judge, min_pass_rate per cohort
│   ├── refusals.yaml        # min_pass_rate: 1.0
│   └── regressions.yaml     # grows via triage, never shrinks
├── tools.py
├── app.py
└── .github/workflows/
    ├── agent-gate.yml
    └── agent-drift.yml

Mock your tools for CI in tools.py (a fixtures flag or a staging base URL): the gate should test agent behavior, not your core banking uptime.

Step 2 · The PR gate #

Config validation is free and instant; the eval gate exits 1 if any cohort misses its min_pass_rate, and a great average never rescues a failing cohort:

name: agent-gate
on:
  pull_request:
    paths: ["agents/**", "evals/**", "tools.py", "app.py"]

jobs:
  gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v6
      - uses: actions/setup-python@v6
        with: { python-version: "3.12" }
      - run: pip install zolva
      - name: Validate config
        run: zolva validate agents/
      - name: Eval gate (worst cohort blocks the merge)
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: >
          zolva eval evals/ --app app:app --gate
          --judge-provider openai --judge-model gpt-5
          --out eval-report.json
      - name: Keep the report on the PR
        if: always()
        uses: actions/upload-artifact@v7
        with: { name: eval-report, path: eval-report.json }

Make the gate job a required status check on your default branch and the loop is closed: a failing refusals cohort is now physically unable to merge.

Step 3 · Weekly drift run #

Providers update models under you. The same command on a schedule catches behavior drift you didn't cause, and the dated report gives you a git-diffable history:

name: agent-drift
on:
  schedule:
    - cron: "0 3 * * 1"      # every Monday 03:00 UTC
  workflow_dispatch: {}       # allow manual runs

jobs:
  drift:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v6
      - uses: actions/setup-python@v6
        with: { python-version: "3.12" }
      - run: pip install zolva
      - name: Full eval sweep
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: >
          zolva eval evals/ --app app:app --gate
          --judge-provider openai --judge-model gpt-5
          --out "drift-$(date +%F).json"
      - uses: actions/upload-artifact@v7
        if: always()
        with: { name: drift-report, path: "drift-*.json" }

A red Monday run with no code change is a provider drift signal: pin or adjust the model in the agent YAML, and let the same gate prove the fix.

Step 4 · Failure → permanent regression #

Production failures land in the feedback queue (escalations automatically, thumbs-downs via record()). Triage them into the eval set, and the bug can never silently return:

$ zolva triage failures.db
#12  [thumbs_down]  agent=collections-agent  note='wrong due date'  last_user='when is my emi due?'
1 pending failure(s)

$ zolva triage failures.db --accept 12 --cohort evals/regressions.yaml \
    --expect "states the correct due date from the ledger"
failure 12 promoted to evals/regressions.yaml

$ zolva export-dataset failures.db dataset.jsonl   # fine-tuning on-ramp, when you want it

The promoted case rides the same PR gate forever. Commit evals/regressions.yaml like the test file it is.

Exit codes & secrets #

• zolva validate: exit 1 on any config error, including inline credentials.
• zolva eval --gate: exit 1 if any cohort misses its floor. Without --gate it reports but never fails the build.
• zolva scorecard audit.db: exit 1 if the audit chain fails verification, which makes chain integrity itself CI-checkable on an operations schedule.
• Keys live in Actions secrets and reach the process as environment variables; agent YAML references them as ${ENV:VAR} if needed, and the loader rejects anything else that looks like a credential.

← All playbooks