Metadata-Version: 2.4
Name: redcrown
Version: 0.1.21
Summary: Make the AI in your product behave like software: a regression harness (report, compare, export), local scoring and checks, and a push to a hosted record with receipts
License: Apache-2.0 (redcrown.harness, the harness, score, check, push and verify commands, and the scoring helpers); Proprietary (the hosted record: the server extra and the API)
Requires-Python: >=3.11
Description-Content-Type: text/markdown
Requires-Dist: pydantic>=2.9
Requires-Dist: pydantic-settings>=2.5
Requires-Dist: httpx>=0.27
Requires-Dist: PyJWT>=2.9
Requires-Dist: cryptography>=43
Provides-Extra: transcribe
Requires-Dist: aioboto3>=15; extra == "transcribe"
Requires-Dist: amazon-transcribe>=0.6; extra == "transcribe"
Provides-Extra: server
Requires-Dist: aioboto3>=15; extra == "server"
Requires-Dist: amazon-transcribe>=0.6; extra == "server"
Requires-Dist: fastapi>=0.115; extra == "server"
Requires-Dist: uvicorn[standard]>=0.30; extra == "server"
Requires-Dist: python-multipart>=0.0.30; extra == "server"
Requires-Dist: sse-starlette>=2.1; extra == "server"
Requires-Dist: asyncpg>=0.30; extra == "server"
Requires-Dist: stripe>=10; extra == "server"

# RedCrown

**Make the AI in your product behave like software.** Grade what the product did
against rules you state, compare every run with the last one that passed, check
prompt and model changes with an exit code your CI can act on, and push the
result to a record with a receipt for every case.

## Install

```bash
pip install redcrown
```

The harness and the CLI install with a small dependency footprint. The AWS
Transcribe and Bedrock providers need `pip install "redcrown[transcribe]"`; the
API needs `pip install "redcrown[server]"`.

## The harness loop

You already keep a journal of what your AI feature did on a fixed set of cases
(one JSONL row per case with a `status` and named `checks`, each `ok`, `fail`
or `unmeasured`). RedCrown reads it, compares two runs check by check, and
exports a payload for the record.

```bash
redcrown harness report <run> --runs runs/
redcrown harness compare <base-run> <head-run> --runs runs/     # exit 1 on any regression
redcrown harness export <run> --runs runs/ --name "one-button" --label "one-button@<commit>" --out payload.json
redcrown push payload.json
```

A status the harness wrote is authoritative. An unmeasured check is never a
pass or a fail; a case with nothing measured is BLOCKED and is disclosed on the
record, not counted as a failure. The two denominators are printed side by side.
Coding agents: `redcrown/harness/AGENTS.md` ships in this package with the exact
commands, exit codes and what never to claim.

`redcrown matrix-plan` previews explicit model and prompt axes offline and can
export a new experiment file with `--output`. It never runs the experiment.
The hosted matrix workspace, regression diagnosis and continuous evaluation
are server features behind their own flags and are not part of this package.

Requires Python 3.11+.

## Quickstart (no keys needed)

Try the bundled offline demo in about five seconds, no provider keys required:

```bash
redcrown eval --sample extract
```

This runs a small bundled extraction sample using stub outputs. The output says
**demo mode** because no real models are called. Connect your provider keys in
`~/.redcrown/credentials.json` (or via `redcrown login`) for real quality scores.

## Bring your own data (text tasks)

If you have a CSV or JSONL file of inputs, import it directly:

```bash
redcrown build-dataset --from-csv mydata.csv --task classify --out exp.json
redcrown eval exp.json --report-json out.json
```

**CSV/JSONL columns:**

| column | required | notes |
|--------|----------|-------|
| `input` | yes | the text sent to the model |
| `reference` | no | expected output (ground truth); omit to use the no-labels path below |
| `id` | no | stable row ID; auto-generated if absent |

**`--task` values:** `extract`, `classify`, `summarize`, `qa`

The importer fills in the provider scaffold for you. Edit `exp.json` to swap in
the candidates you want to compare before running.

## Score outputs you already have (no model calls)

If another harness produced the outputs, score them locally with a deterministic
metric. This command never connects to a provider or needs an API key:

```bash
redcrown score score-request.json --report-json scored.json
```

`score-request.json` uses `items` (`item_id`, `input`, `reference`) and
`candidates` (`id`, `label`, `outputs`) from the ScoreOutputsRequest schema. For
tabular workflows, keep the items in CSV and put one candidate output per JSONL
line:

```bash
redcrown score --items-csv items.csv --outputs-jsonl outputs.jsonl \
  --metric exact_match --quality-bar 0.8 --report-json scored.json
```

```json
{"candidate_id":"model-a","item_id":"row1","output":"billing","latency_ms":42}
```

CSV accepts `item_id` (or `id`), `input`, and `reference`. JSONL also accepts
`candidate_label`, `error`, `cost_usd`, and `latency_ms`. Omitted cost or latency
stays unknown rather than being treated as zero. The resulting file can be sent
to RedCrown with `redcrown push scored.json`.

## Check prompt/model changes locally

Starting with 0.1.19, export private evidence for offline regression checks:

```bash
redcrown score baseline-request.json --evidence-json baseline-evidence.json
redcrown score candidate-request.json --evidence-json candidate-evidence.json
redcrown check candidate-evidence.json --baseline baseline-evidence.json \
  --policy policy.json --format json
```

Use the same nonblank `experiment_id` and the same evaluation items/references
in both score requests. Each selected candidate needs a complete configuration
declaration, for example:

```json
{
  "experiment_id": "support-routing",
  "quality_metric": "exact_match",
  "items": [{"item_id": "one", "input": "Refund request", "reference": "billing"}],
  "candidates": [{
    "id": "external-a",
    "configuration": {
      "provider": "external-harness",
      "model": "routing-model-v1",
      "prompt": "Return the routing category.",
      "system_prompt": null,
      "params": {},
      "source_revision": null,
      "endpoint_revision": null
    },
    "outputs": [{"item_id": "one", "output": "billing"}]
  }]
}
```

All seven configuration fields are required; null declares an absent setting,
not an unknown one. RedCrown scores these supplied outputs but does not witness
their generation. Configuration assurance remains **declared**. Keep secrets
out of declarations and evidence files.

Example `policy.json` (choose thresholds for your own requirements):

```json
{
  "schema_version": 1,
  "requirements": {"quality": {"minimum": 0.85, "maximum_regression": 0.01}}
}
```

When a report has multiple candidates, specify `--candidate-id` and
`--baseline-candidate-id`; checks never select a winner automatically. Quality
gates require complete scored coverage. Missing or incompatible evidence cannot
silently pass. JSON is the supported policy format; YAML is deferred. Cost and
latency rules remain indeterminate until measurement provenance is supported.

`check` makes no network, provider, or authentication calls. Exit 0 means pass;
exit 2 means fail or indeterminate; exit 1 means a command, file, or policy error.
JSON, Markdown, and text results describe the same decision.

`eval --evidence-json` also exports private raw evidence; real evaluations may
call providers. Keep a stable `id` in the experiment across those evaluations.
`--report-json` remains the upload format for `push`; it is different from the
private raw format consumed by `check`. Keep private evidence out of public CI
artifacts. Select baseline and policy files from a trusted source.

## The exp.json schema

Every eval is a plain JSON file. Here is a minimal 2-candidate text example you
can copy, edit, and run:

```json
{
  "name": "classify-support-tickets",
  "quality_metric": "similarity",
  "quality_bar": 0.75,
  "reference_source": "labels",
  "pipeline": [
    {
      "id": "step1",
      "order": 1,
      "name": "classify",
      "input_type": "text",
      "output_type": "text",
      "input_binding": "dataset",
      "incumbent_candidate_id": "gpt4o-mini",
      "candidates": [
        {
          "id": "gpt4o-mini",
          "provider": "openai",
          "model": "gpt-4o-mini",
          "label": "GPT-4o Mini (incumbent)"
        },
        {
          "id": "gpt-oss-20b",
          "provider": "groq",
          "model": "openai/gpt-oss-20b",
          "label": "GPT-OSS 20B via Groq"
        }
      ]
    }
  ],
  "dataset": [
    {
      "id": "row1",
      "payload": { "kind": "text", "text": "My invoice is wrong." },
      "reference": "billing"
    },
    {
      "id": "row2",
      "payload": { "kind": "text", "text": "App crashes on login." },
      "reference": "bug"
    }
  ]
}
```

**Required fields:** `name`, `quality_metric`, `quality_bar`, `reference_source`,
`pipeline[].incumbent_candidate_id`, `pipeline[].candidates[].id`,
`pipeline[].candidates[].provider`, `pipeline[].candidates[].model`,
`dataset[].id`, `dataset[].payload.text`.

## No labels? Use the incumbent as reference

If you do not have labeled ground truth, leave `reference` off your dataset rows
and set `reference_source: "incumbent"`:

```json
{
  "reference_source": "incumbent",
  ...
  "dataset": [
    { "id": "row1", "payload": { "kind": "text", "text": "My invoice is wrong." } }
  ]
}
```

RedCrown scores every candidate against your current model's own output, so you
only need to beat or match what you run today. This is the fastest path to a cost
proof when you have real traffic but no labeled examples.

## Running a full eval

```bash
# run the fan-out locally, on your machine, with your keys
redcrown eval exp.json --report-json out.json

# (optional) sign in once per machine via device-code OAuth
redcrown login

# (optional) push the results to a shareable, no-login proof page
redcrown push out.json --proof-link
```

`redcrown eval` ranks every config on cost, quality, and latency against your own
ground truth and names the cheapest one that clears your quality bar. Example:

```
RANKED  transcription · cheapest config at or above your 0.85 quality bar
  deepgram · nova-3-medical    quality 0.883    $294/mo   winner, 40% cheaper
  aws · transcribe-standard    quality 0.879    $487/mo   incumbent
  openai · whisper-1           quality 0.820    $122/mo   below your bar
```

That run is published as a live, no-login proof page:
<https://app.redcrown.ai/proof/O9iYVdWuaYjaL6mnImeIsD6TB1W_S6h4Frbx04YqAYQ>

> **Note on `build-dataset primock57`:** this command downloads the PriMock57
> clinical-transcription corpus, which requires `git-lfs`. It is an audio eval
> designed for transcription benchmarking, not a general first step. Start with
> `redcrown eval --sample extract` or `build-dataset --from-csv` instead.

## Free by construction

Evals run on your machine with your own provider keys, so RedCrown never sees your
raw data and the run costs you nothing beyond your own inference. Only the results
you choose to `push` become a cloud proof. `--no-receipts` keeps raw outputs local
and uploads aggregates only.

## Already ran an eval elsewhere?

You do not have to run anything through RedCrown to get a proof. Take the results from
an eval you already ran, as a JSON in the RedCrown results format, and push them:

```bash
redcrown push results.json --proof-link
```

You get the same ranked, receipted, shareable report. The fastest path, with no install,
is the web app at <https://app.redcrown.ai/upload>.

## Eval your own OpenAI-compatible endpoint locally

If you have a private or self-hosted API that speaks the OpenAI chat-completion format,
point `redcrown eval` at it without ever uploading your endpoint secret:

```bash
export REDCROWN_CUSTOM_HTTP='{"base_url":"https://api.example.com/v1","auth_header_name":"Authorization","auth_value":"Bearer sk-...","body_style":"openai"}'
redcrown eval exp.json
```

Set a candidate in your `exp.json` with `"provider": "custom_http"` and any `"model"` string
your endpoint recognises. The JSON connection stays local; it is read directly from the env var
at eval time and is never persisted or uploaded.

## For coding agents (MCP)

Coding agents (Claude, Cursor, Codex) drive the whole loop over the hosted MCP server
at `mcp.redcrown.ai`: scaffold an experiment, run it, review outputs, and mint a proof.
The server is open source: <https://github.com/RedCrown-ai/redcrown-mcp>

## Links

- Site: <https://redcrown.ai>
- App: <https://app.redcrown.ai>
- MCP server (open source): <https://github.com/RedCrown-ai/redcrown-mcp>

## License

`redcrown.harness`, the `harness`, `score`, `check`, `push` and `verify`
commands, and the pure scoring helpers they import: Apache-2.0. The hosted
record (the `server` extra and the API modules): Proprietary, (c) RedCrown.ai.
One wheel carries both until the open modules move to a public repository.
