Metadata-Version: 2.4
Name: redcrown
Version: 0.1.23
Summary: Regression checks and release evidence for AI features: score supplied outputs, check a change against a reference run and a written policy offline, compare harness runs, and push a record with receipts
License-Expression: Apache-2.0 AND LicenseRef-Proprietary
Keywords: ai,evaluation,regression,llm,testing,release-evidence
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE.md
Requires-Dist: pydantic>=2.9
Requires-Dist: pydantic-settings>=2.5
Requires-Dist: httpx>=0.27
Requires-Dist: PyJWT>=2.9
Requires-Dist: cryptography>=43
Provides-Extra: transcribe
Requires-Dist: aioboto3>=15; extra == "transcribe"
Requires-Dist: amazon-transcribe>=0.6; extra == "transcribe"
Provides-Extra: server
Requires-Dist: aioboto3>=15; extra == "server"
Requires-Dist: amazon-transcribe>=0.6; extra == "server"
Requires-Dist: fastapi>=0.115; extra == "server"
Requires-Dist: uvicorn[standard]>=0.30; extra == "server"
Requires-Dist: python-multipart>=0.0.30; extra == "server"
Requires-Dist: sse-starlette>=2.1; extra == "server"
Requires-Dist: asyncpg>=0.30; extra == "server"
Requires-Dist: stripe>=10; extra == "server"
Dynamic: license-file

# RedCrown

**RedCrown helps teams check AI changes against defined requirements and
inspect the evidence behind the result.** Use your own cases and requirements
to compare a changed AI feature with a reference run or supported accepted
baseline. Inspect failures, understand missing evidence, and share the
available results with the people making the decision.

The check runs on your machine. `redcrown score` and `redcrown check` read
files, make no network call, and need no account and no provider key.

## Install

```bash
pip install redcrown
```

The CLI installs with a small dependency footprint. The AWS Transcribe and
Bedrock providers need `pip install "redcrown[transcribe]"`; the API needs
`pip install "redcrown[server]"`. Requires Python 3.11+.

## Quickstart: check a change against a reference run

The example is synthetic: 40 invented support tickets, one label each, and
the outputs of an imaginary ticket router at three revisions. No model ran.
Every number below comes from the released CLI.

```bash
base=https://redcrown.ai/docs/quickstart/regression-check
curl -sSO $base/score-request.json -O $base/other-feature.json -O $base/policy.json
```

Score every revision's outputs against the labels. The evidence file is
private: it holds the inputs, outputs and references.

```
$ redcrown score score-request.json --evidence-json evidence.json
Experiment ticket-router  objective=quality  bar=0.95  baseline=labels

Step outputs:
  candidate                 quality     $/min         $/mo       cost   latency  flags
  reference-v1                1.000       n/a          n/a    unknown   unknown  baseline, WINNER
  candidate-v3                0.975       n/a          n/a    unknown   unknown
  candidate-v2                0.900       n/a          n/a    unknown   unknown
  candidate-v2-partial        0.875       n/a          n/a    unknown   unknown  1 errors
```

Check the reference run against itself. This is the control: the policy in
`policy.json` asks for 40 cases, no errors, no ungraded cases, a declared
configuration, exact match at least 0.95, and a regression of at most 0.05.

```
$ redcrown check evidence.json --candidate-id reference-v1 --baseline evidence.json --baseline-candidate-id reference-v1 --policy policy.json
redcrown check: pass (exit 0)
- minimum_items: pass [requirement_met]; observed=40, threshold=40, count=40, denominator=40
- maximum_error_rate: pass [requirement_met]; observed=0.0, threshold=0.0, count=0, denominator=40
- maximum_ungraded_rate: pass [requirement_met]; observed=0.0, threshold=0.0, count=0, denominator=40
- require_known_config: pass [requirement_met]; observed=True, threshold=True, count=1, denominator=1
- quality.minimum: pass [requirement_met]; observed=1.0, threshold=0.95, count=40, denominator=40
- quality.maximum_regression: pass [requirement_met]; observed=-0.0, threshold=0.05, count=40, denominator=40
```

Check the candidate. It did not meet two of the six requirements, and the
exit code says so.

```
$ redcrown check evidence.json --candidate-id candidate-v2 --baseline evidence.json --baseline-candidate-id reference-v1 --policy policy.json
redcrown check: fail (exit 2)
- minimum_items: pass [requirement_met]; observed=40, threshold=40, count=40, denominator=40
- maximum_error_rate: pass [requirement_met]; observed=0.0, threshold=0.0, count=0, denominator=40
- maximum_ungraded_rate: pass [requirement_met]; observed=0.0, threshold=0.0, count=0, denominator=40
- require_known_config: pass [requirement_met]; observed=True, threshold=True, count=1, denominator=1
- quality.minimum: fail [threshold_breached]; observed=0.9, threshold=0.95, count=40, denominator=40
- quality.maximum_regression: fail [threshold_breached]; observed=0.1, threshold=0.05, count=40, denominator=40
```

List the cases that changed. Add `--changed` to the same command. The
per-case scores are in the evidence file; `--changed` reads them and computes
nothing new. It does not change the status or the exit code. `--changed`
prints outputs. Do not use it in a CI job whose logs other people can read.

```
$ redcrown check evidence.json --candidate-id candidate-v2 --baseline evidence.json --baseline-candidate-id reference-v1 --policy policy.json --changed
redcrown check: fail (exit 2)
- minimum_items: pass [requirement_met]; observed=40, threshold=40, count=40, denominator=40
- maximum_error_rate: pass [requirement_met]; observed=0.0, threshold=0.0, count=0, denominator=40
- maximum_ungraded_rate: pass [requirement_met]; observed=0.0, threshold=0.0, count=0, denominator=40
- require_known_config: pass [requirement_met]; observed=True, threshold=True, count=1, denominator=1
- quality.minimum: fail [threshold_breached]; observed=0.9, threshold=0.95, count=40, denominator=40
- quality.maximum_regression: fail [threshold_breached]; observed=0.1, threshold=0.05, count=40, denominator=40
changed cases: 4 regressed, 0 fixed, 0 excluded, 40 in the set
- case-07: regressed; baseline 'account', candidate 'billing'
- case-18: regressed; baseline 'technical', candidate 'account'
- case-25: regressed; baseline 'billing', candidate 'technical'
- case-36: regressed; baseline 'shipping', candidate 'account'
```

Three more shapes the same files show:

- `--candidate-id candidate-v3` passes (0.975, regression 0.025, exit 0).
- `--candidate-id candidate-v2-partial` fails on `maximum_error_rate` (one
  case timed out). The two quality rules are `indeterminate` with the reason
  `incomplete_quality_coverage`, count 39 of 40. A missing score is never a
  pass and never a zero.
- Score `other-feature.json` to `other-evidence.json` and check it against
  `evidence.json`: exit 2 with the evidence issues `workload_mismatch` and
  `content_hash_mismatch`. A different feature cannot be compared with this
  reference run.

`check` exits 0 on pass, 2 on fail or indeterminate, and 1 on a command, file
or policy error (`redcrown check policy.json --policy policy.json` is the
error case: `candidate_raw_report_required`). `--format json` prints the same
decision with every rule's `id`, `status`, `reason`, `observed`, `threshold`,
`count` and `denominator`.

## Score outputs you already have (no model calls)

If another harness produced the outputs, score them locally with a
deterministic metric (`exact_match`, `similarity`, `json_field_accuracy`,
`f1`). This command never connects to a provider or needs an API key:

```bash
redcrown score score-request.json --report-json scored.json
```

`score-request.json` uses `items` (`item_id`, `input`, `reference`) and
`candidates` (`id`, `label`, `configuration`, `outputs`) from the
ScoreOutputsRequest schema. For tabular workflows, keep the items in CSV and
put one candidate output per JSONL line:

```bash
redcrown score --items-csv items.csv --outputs-jsonl outputs.jsonl \
  --metric exact_match --quality-bar 0.8 --report-json scored.json
```

```json
{"candidate_id":"model-a","item_id":"row1","output":"billing","latency_ms":42}
```

CSV accepts `item_id` (or `id`), `input`, and `reference`. JSONL also accepts
`candidate_label`, `error`, `cost_usd`, and `latency_ms`. Omitted cost or
latency stays unknown rather than being treated as zero. The resulting file
can be sent to RedCrown with `redcrown push scored.json`.

The CSV+JSONL form is scorable but not checkable against a reference run: it
carries no declared configuration, so `redcrown check` returns
`missing_or_invalid_config` and every rule is indeterminate. The request JSON
form with a `configuration` per candidate is the one `check` accepts.

## Check prompt and model changes in CI

Export private evidence from each score, then check the candidate against the
reference run and a policy:

```bash
redcrown score baseline-request.json --evidence-json baseline-evidence.json
redcrown score candidate-request.json --evidence-json candidate-evidence.json
redcrown check candidate-evidence.json --baseline baseline-evidence.json \
  --policy policy.json --format json
```

Use the same nonblank `experiment_id` and the same evaluation items and
references in both score requests. Each selected candidate needs a complete
configuration declaration, for example:

```json
{
  "experiment_id": "support-routing",
  "quality_metric": "exact_match",
  "items": [{"item_id": "one", "input": "Refund request", "reference": "billing"}],
  "candidates": [{
    "id": "external-a",
    "configuration": {
      "provider": "external-harness",
      "model": "routing-model-v1",
      "prompt": "Return the routing category.",
      "system_prompt": null,
      "params": {},
      "source_revision": null,
      "endpoint_revision": null
    },
    "outputs": [{"item_id": "one", "output": "billing"}]
  }]
}
```

All seven configuration fields are required; null declares an absent setting,
not an unknown one. RedCrown scores these supplied outputs but does not
witness their generation. Configuration assurance remains **declared**. Keep
secrets out of declarations and evidence files.

Example `policy.json` (choose thresholds for your own requirements):

```json
{
  "schema_version": 1,
  "requirements": {"quality": {"minimum": 0.85, "maximum_regression": 0.01}}
}
```

When a report has multiple candidates, specify `--candidate-id` and
`--baseline-candidate-id`; checks never select a winner automatically.
Quality rules require complete scored coverage. Missing or incompatible
evidence cannot silently pass. JSON is the supported policy format; YAML is
deferred. Cost and latency rules parse but stay indeterminate until
measurement provenance is supported.

`check` makes no network, provider, or authentication calls. JSON, Markdown,
and text results describe the same decision. `eval --evidence-json` also
exports private raw evidence; real evaluations may call providers. Keep a
stable `id` in the experiment across those evaluations. `--report-json`
remains the upload format for `push`; it is different from the private raw
format consumed by `check`. Keep private evidence out of public CI artifacts.
Select baseline and policy files from a trusted source.

## Compare harness runs

You already keep a journal of what your AI feature did on a fixed set of cases
(one JSONL row per case with a `status` and named `checks`, each `ok`, `fail`
or `unmeasured`). RedCrown reads it, compares two runs check by check, and
exports a payload for the record.

```bash
redcrown harness report <run> --runs runs/
redcrown harness compare <base-run> <head-run> --runs runs/     # exit 1 on any regression, 2 on corpus drift
redcrown harness export <run> --runs runs/ --name "one-button" --label "one-button@<commit>" --out payload.json
redcrown push payload.json
```

A status the harness wrote is authoritative. An unmeasured check is never a
pass or a fail; a case with nothing measured is BLOCKED and is disclosed on the
record, not counted as a failure. The two denominators are printed side by side.
Coding agents: `redcrown/harness/AGENTS.md` ships in this package with the exact
commands, exit codes and what never to claim.

`redcrown matrix-plan` previews explicit model and prompt axes offline and can
export a new experiment file with `--output`. It never runs the experiment.
The hosted matrix workspace, regression diagnosis and continuous evaluation
are server features behind their own flags and are not part of this package.

## Bundled sample (no keys needed)

Try the bundled offline demo, no provider keys required:

```bash
redcrown eval --sample extract
```

This runs a small bundled extraction sample using stub outputs. The output says
**demo mode** because no real models are called. Connect your provider keys in
`~/.redcrown/credentials.json` (or via `redcrown login`) for real quality scores.

## Bring your own data (text tasks)

If you have a CSV or JSONL file of inputs, import it directly:

```bash
redcrown build-dataset --from-csv mydata.csv --task classify --out exp.json
redcrown eval exp.json --report-json out.json
```

**CSV/JSONL columns:**

| column | required | notes |
|--------|----------|-------|
| `input` | yes | the text sent to the model |
| `reference` | no | expected output (ground truth); omit to use the no-labels path below |
| `id` | no | stable row ID; auto-generated if absent |

**`--task` values:** `extract`, `classify`, `summarize`, `qa`

The importer fills in the provider scaffold for you. Edit `exp.json` to swap in
the candidates you want to compare before running.

## Running a full evaluation

Model comparison is the supporting capability: the same cases across several
models or configurations, ranked on the quality bar you set.

```bash
# run the fan-out locally, on your machine, with your keys
redcrown eval exp.json --report-json out.json

# (optional) sign in once per machine via device-code OAuth
redcrown login

# (optional) push the results to a shareable, no-login report page
redcrown push out.json --proof-link
```

`redcrown eval` ranks every config on cost, quality, and latency against your own
ground truth and names the cheapest one that clears your quality bar. Example:

```
RANKED  transcription · cheapest config at or above your 0.85 quality bar
  deepgram · nova-3-medical    quality 0.883    $294/mo   winner, 40% cheaper
  aws · transcribe-standard    quality 0.879    $487/mo   incumbent
  openai · whisper-1           quality 0.820    $122/mo   below your bar
```

That run is published as a live, no-login report page:
<https://app.redcrown.ai/proof/O9iYVdWuaYjaL6mnImeIsD6TB1W_S6h4Frbx04YqAYQ>

> **Note on `build-dataset primock57`:** this command downloads the PriMock57
> clinical-transcription corpus, which requires `git-lfs`. It is an audio eval
> designed for transcription benchmarking, not a general first step. Start with
> the regression quickstart above, `redcrown eval --sample extract`, or
> `build-dataset --from-csv` instead.

## The exp.json schema

Every eval is a plain JSON file. Here is a minimal 2-candidate text example you
can copy, edit, and run:

```json
{
  "name": "classify-support-tickets",
  "quality_metric": "similarity",
  "quality_bar": 0.75,
  "reference_source": "labels",
  "pipeline": [
    {
      "id": "step1",
      "order": 1,
      "name": "classify",
      "input_type": "text",
      "output_type": "text",
      "input_binding": "dataset",
      "incumbent_candidate_id": "gpt4o-mini",
      "candidates": [
        {
          "id": "gpt4o-mini",
          "provider": "openai",
          "model": "gpt-4o-mini",
          "label": "GPT-4o Mini (incumbent)"
        },
        {
          "id": "gpt-oss-20b",
          "provider": "groq",
          "model": "openai/gpt-oss-20b",
          "label": "GPT-OSS 20B via Groq"
        }
      ]
    }
  ],
  "dataset": [
    {
      "id": "row1",
      "payload": { "kind": "text", "text": "My invoice is wrong." },
      "reference": "billing"
    },
    {
      "id": "row2",
      "payload": { "kind": "text", "text": "App crashes on login." },
      "reference": "bug"
    }
  ]
}
```

**Required fields:** `name`, `quality_metric`, `quality_bar`, `reference_source`,
`pipeline[].incumbent_candidate_id`, `pipeline[].candidates[].id`,
`pipeline[].candidates[].provider`, `pipeline[].candidates[].model`,
`dataset[].id`, `dataset[].payload.text`.

## No labels? Use the incumbent as reference

If you do not have labeled ground truth, leave `reference` off your dataset rows
and set `reference_source: "incumbent"`:

```json
{
  "reference_source": "incumbent",
  ...
  "dataset": [
    { "id": "row1", "payload": { "kind": "text", "text": "My invoice is wrong." } }
  ]
}
```

RedCrown scores every candidate against your current model's own output, so you
only need to beat or match what you run today. This is the fastest path to a cost
comparison when you have real traffic but no labeled examples.

## Free by construction

Evaluations run on your machine with your own provider keys, so RedCrown never
sees your raw data and the run costs you nothing beyond your own inference.
Only the results you choose to `push` become a hosted report. `--no-receipts`
keeps raw outputs local and uploads aggregates only.

## Already ran an evaluation elsewhere?

You do not have to run anything through RedCrown to get a shared report. Take
the results from an evaluation you already ran, as a JSON in the RedCrown
results format, and push them:

```bash
redcrown push results.json --proof-link
```

You get the same ranked, receipted, shareable report. The fastest path, with no
install, is the web app at <https://app.redcrown.ai/upload>. An aggregate import
carries no case-level evidence unless the payload carried receipts.

## Evaluate your own OpenAI-compatible endpoint locally

If you have a private or self-hosted API that speaks the OpenAI chat-completion format,
point `redcrown eval` at it without ever uploading your endpoint secret:

```bash
export REDCROWN_CUSTOM_HTTP='{"base_url":"https://api.example.com/v1","auth_header_name":"Authorization","auth_value":"Bearer sk-...","body_style":"openai"}'
redcrown eval exp.json
```

Set a candidate in your `exp.json` with `"provider": "custom_http"` and any `"model"` string
your endpoint recognizes. The JSON connection stays local; it is read directly from the env var
at eval time and is never persisted or uploaded.

## For coding agents (MCP)

Coding agents (Claude, Cursor, Codex) drive the model evaluation loop over the
hosted MCP server at `mcp.redcrown.ai`: scaffold an experiment, run it, import
results, review outputs, and create a share link. `score`, `check`, harness
compare and baseline acceptance are CLI or app only; there is no MCP tool for
them. The server is open source: <https://github.com/RedCrown-ai/redcrown-mcp>

## Links

- Site: <https://redcrown.ai>
- App: <https://app.redcrown.ai>
- Quickstart: <https://redcrown.ai/docs/quickstart/>
- MCP server (open source): <https://github.com/RedCrown-ai/redcrown-mcp>

## License

`redcrown.harness`, the `harness`, `score`, `check`, `push` and `verify`
commands, and the pure scoring helpers they import: Apache-2.0. The hosted
record (the `server` extra and the API modules): Proprietary, (c) RedCrown.ai.
One wheel carries both until the open modules move to a public repository.
