Metadata-Version: 2.5
Name: certops-cli
Version: 0.1.1b1
Summary: CertOps CLI — The AI Quality Assurance Gatekeeper
Project-URL: Homepage, https://certops.ai
Project-URL: Documentation, https://docs.certops.ai
Project-URL: Repository, https://github.com/vignesh865/certops-cli
Author-email: Vignesh Baskaran <vignesh865@gmail.com>
License-Expression: MIT
License-File: LICENSE
Keywords: ai,certops,ci-cd,evaluation,llm,quality-assurance,testing
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.11
Requires-Dist: click>=8.3.1
Requires-Dist: httpx>=0.28.1
Requires-Dist: pydantic>=2.0
Requires-Dist: pyyaml>=6.0.3
Requires-Dist: rich>=14.3.3
Description-Content-Type: text/markdown

# CertOps CLI

The CI/CD gatekeeper for [CertOps](https://certops.ai) — an AI quality-assurance
platform that black-box tests any AI agent over HTTP and decides whether it ships.

`certops run` executes a certification suite against your deployed (or local)
agent, blocks until there's a verdict, and exits non-zero if the agent fails its
quality gates. Drop it in a pipeline and a regression can't reach production.

> **Trust, But Verify.**

## Install

```bash
pip install certops-cli
```

Requires Python 3.11+.

## Quickstart

```bash
# 1. Authenticate (username is tenant\email)
certops login --username 'acme\alice@acme.com'

# 2. Upload a golden dataset → prints a dataset ID for your manifest
certops upload data/golden_set.csv

# 3. Certify a deployed agent
certops run -f certops.yaml --host chatbot=https://staging.acme.com

# 4. Once it passes, make it the baseline future runs are compared against
certops tag <run_id> prod
```

## Exit codes

The whole point of the CLI. `certops run` and `certops status` exit:

| Code | Meaning | What CI should do |
|---|---|---|
| `0` | **Certified** — every blocking gate passed | Promote |
| `1` | **Rejected** — a blocking gate failed | Fail the build; this is a real quality regression |
| `2` | **System error** — no verdict was reached (timeout, API down, auth failure, Ctrl+C) | Fail the build, but don't blame the model |

The `1`/`2` split is deliberate: a network blip must never be reported as a
quality failure.

## Commands

| Command | Purpose |
|---|---|
| `certops login` / `logout` / `whoami` | Session management |
| `certops run -f certops.yaml` | Trigger a suite and block for the verdict |
| `certops status <run_id>` | Re-attach to a run already in flight |
| `certops runs` | List recent runs |
| `certops tag <run_id> <tag>` | Tag a run (e.g. as the `prod` baseline) |
| `certops certificate generate <run_number>` | Issue a Certificate of Conformity |
| `certops certificate show <run_id>` | Fetch an existing certificate |
| `certops certificate verify <cert_id>` | Publicly verify a certificate (no login) |
| `certops upload <file>` | Upload a CSV/JSON dataset |

Run `certops <command> --help` for full flags.

### `certops run`

```bash
certops run \
  -f ./certops.yaml \
  --host retriever=https://pr-45-retriever.acme.com \
  --host generator=https://pr-45-generator.acme.com \
  --tag staging
```

| Flag | Notes |
|---|---|
| `-f, --manifest` | Path to `certops.yaml` (or JSON). Required. |
| `--host` | `target_id=url`. Repeatable. A bare `url` applies to **every** target. |
| `--tag` | Tag the run at trigger time (avoids a second `certops tag` call). |
| `--notes` | Free-text note attached to the run. |
| `-d, --dataset` | Override the dataset ID for all targets. |
| `--timeout` | Max seconds to block. Default 1800. Exceeding it exits `2`. |
| `--no-wait` | Fire-and-forget; prints the run ID and exits. |

### Configuration

| Setting | Flag | Env var |
|---|---|---|
| API base URL | `--api-url` | `CERTOPS_API_URL` |

Precedence: flag → env → the URL stored at login → `https://api.certops.ai`.
Credentials and relay config are cached in `~/.certops/config.json`.

## Testing a local agent (Hybrid Bridge)

You don't have to deploy to certify. Point a target at `local:PORT` and the CLI
opens an ephemeral tunnel so the SaaS can reach your laptop:

```bash
certops run -f certops.yaml --host chatbot=local:8080

# with a path prefix
certops run -f certops.yaml --host chatbot=local:8080/agent2
```

The CLI runs one `frpc` process multiplexing every local target over a single TLS
connection to the relay, at `https://{prefix}-{target_id}.{relay_host}`. The
subdomain prefix and relay host both come from the server at login. Tunnels are
bound to the lifetime of the command and torn down on exit.

Because an evaluation can fire 1,000+ requests at your agent, the tunnel enables
`tcpMux`; without it the connection overhead would dominate.

> **Ctrl+C:** tunnels close and the CLI exits `2`, but the run continues
> server-side — the API has no cancellation endpoint yet. Use
> `certops status <run_id>` to re-attach.

## The manifest

`certops.yaml` defines *endpoints*, never *hosts* — that's what makes one manifest
certify dev, staging and prod. Hosts are injected at runtime via `--host`.

```yaml
version: "1.0"

suite:
  name: "RAG Pipeline Certification"
  owner: "search-team"

targets:
  - id: "generator"
    name: "Answer Generator"
    endpoint: "/v1/chat"          # relative path only
    method: "POST"
    headers:
      Authorization: "Bearer ${env.GENERATOR_KEY}"

    request:
      format: "json"              # json (default) | multipart | urlencoded | raw
      body: |
        { "messages": [{"role": "user", "content": "{{ user_query }}"}] }

    response_path: "choices[0].message.content"

    dataset:
      id: "ds_generator_golden_v5"

    # Optional resilience — retry a flaky endpoint before calling the sample failed
    retry_count: 2
    retry_delay: 1.0

    # Pointwise evaluation
    evaluation:
      metrics_mapping:
        input: "user_query"
        reference: "golden_answer"
      deterministic:
        - metric: "cosine-similarity"
          threshold: 0.85
          operator: "gte"          # gte | gt | lte | lt | eq
          blocking: true
      llm:
        - metric: "hallucination"
          threshold: 0.1
          operator: "lte"
          blocking: true

configuration:
  judge_model_config_id: "model-config-uuid"
  concurrency: 5
  stop_on_failure: true
```

A target is **Rejected** if any *blocking* gate fails. The suite is **Certified**
only if every target is.

### Comparison: two independent axes

A target may declare `regression`, `pairwise`, both, or neither. They are sibling
blocks — there is no `comparison` umbrella and no `pairing` selector.

**`regression`** — *directional*, against the latest run carrying a given tag,
matched by sample index. Answers "did we get worse?"

```yaml
    regression:
      baseline: "prod"
      deterministic:
        - metric: "cosine-similarity"
          max_drift: 0.05           # average can't drop more than 5%
      metrics_mapping:
        input: "user_query"
      llm:
        - metric: "general-quality"
          max_loss_rate: 0.3        # or: min_win_rate
```

If no run carries the baseline tag yet, comparison is skipped with a non-blocking
notice and the run still passes on evaluation alone.

**`pairwise`** — *symmetric*, within a single run, comparing counterfactual
variants against each other. Answers "are we consistent across groups?" (fairness).

```yaml
    pairwise:
      mode: "group"                 # group | contrastive
      group_by: "group_id"
      role: "role"
      llm:
        - metric: "bias"
          min_equivalence_rate: 0.9  # or: max_divergence_rate, max_mean_divergence
```

Gate keys are axis-specific — a `max_loss_rate` inside a `pairwise` block is a
manifest error and will be rejected, not ignored.

> **Migrating?** The old unified `comparison:` block with `pairing.mode` is
> retired. The API rejects it with a migration hint rather than silently dropping
> your gates. Split it into `regression` (takes `baseline`) and/or `pairwise`
> (takes `mode`).

### Chaining targets

`depends_on` maps a variable to an upstream target's response, so you can certify a
pipeline stage-by-stage. Upstream targets must be declared first.

```yaml
  - id: "generator"
    depends_on:
      context: "retriever.data.documents[0].content"
```

### Built-in metrics

`invocation-success` is injected into every run automatically — it's the fraction
of samples whose HTTP call succeeded. You don't declare it, but you can gate on it:

```yaml
      deterministic:
        - metric: "invocation-success"
          threshold: 1.0
          operator: "gte"
          blocking: true
```

## CI example

```yaml
# .github/workflows/certify.yml
- name: Certify agent
  env:
    CERTOPS_API_URL: https://api.certops.ai
  run: |
    pip install certops-cli
    certops login --username "${{ secrets.CERTOPS_USER }}" \
                  --password "${{ secrets.CERTOPS_PASSWORD }}"
    certops run -f certops.yaml \
      --host chatbot=${{ steps.deploy.outputs.url }} \
      --tag staging
```

The CLI detects GitHub Actions / generic CI from the environment and attributes the
run's trigger source accordingly.

## Development

```bash
uv sync
uv run pytest
uv run certops --help
```

Point at a local backend with `certops --api-url http://localhost:8000 ...` or
`CERTOPS_API_URL=http://localhost:8000`.

`tests/fixtures.py` holds the API contract — payload shapes transcribed from the
backend's source and tests. If the backend contract changes, update that file
first; the tests follow from it. `test_gate_types_are_exhaustive` fails loudly if
the backend grows a gate type the renderer doesn't handle.
