Metadata-Version: 2.4
Name: margin-cost
Version: 0.2.0
Summary: Cost per outcome across your AI-agent vendors, from a local corpus. No account, no backend.
License: MIT
Keywords: llm,cost,finops,agents,cost-per-outcome,observability
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: dev
Requires-Dist: pytest>=7; extra == "dev"
Dynamic: license-file

# margin-cost

[![tests](https://github.com/trymargin/margin-cost/actions/workflows/test.yml/badge.svg)](https://github.com/trymargin/margin-cost/actions/workflows/test.yml)
[![PyPI](https://img.shields.io/pypi/v/margin-cost.svg)](https://pypi.org/project/margin-cost/)

Your invoice shows the bill. It rarely shows which vendor earned it.

```bash
pip install margin-cost
margin-cost --example | margin-cost -
```

That is the whole thing. No file to author, no config, no account, no backend —
`--example` emits a sample estate and the second call reports on it, so you get real
output on the first command you run. Point it at your own file when you have one.

Then make it a **gate**, which is the part nothing else in this category does:

```bash
margin-cost estate.json --fail-over 0.14        # exit 1 when cost per outcome is over budget
margin-cost estate.json --fail-if-unmeasured    # exit 1 when an outcome was guessed, not measured
```

Exit codes are the contract: **0** pass, **1** the gate says no, **2** could not run — so
a CI author can tell a refusal from a failure. Drop it in a workflow and a cost regression
fails the build instead of showing up on next month's invoice.


Your invoice shows the bill. It rarely shows which vendor earned it.

`margin-cost` divides spend by outcomes, per vendor, from a file on your disk.
No account, no signup, no backend. Nothing leaves the machine.

## Why cost per outcome

Cost per token tells you what you spent. It says nothing about what the spend
bought. Two agents can burn the same tokens and resolve very different numbers
of tickets. The question a budget owner actually has is "which of my vendors is
worth its invoice?", and the honest denominator for that is the outcome, not the
token.

That number works even on vendors that expose nothing. Glean bills per seat and
you will never see its tokens, but you know what it produced, so you can still
divide its invoice by the outcomes it delivered.


## What this tool does not do

`margin-cost` measures. It does not act.

It will tell you that one vendor costs $4.03 per outcome and another costs $0.018. It will not
switch anything, and it has no opinion about whether the cheaper one would have held up.

That second question is the hard one, and it is deliberately not in here:

| in this tool, free | not in this tool |
|---|---|
| cost per outcome, per vendor | routing a call to a cheaper model |
| retry tax, cache efficiency, token yield | proving the cheaper model held quality first |
| the three coverage tiers | promoting a route, and reverting it when parity slips |
| runs on your machine, no account | any of it running continuously |

**Why the line sits there.** Measuring your spend is becoming table stakes — several vendors give
it away now, and gatekeeping it would be both futile and annoying. Deciding *what to change* and
proving the change did not cost you quality is the part that is actually hard, and that is the
commercial product.

If you run this and dislike the number, that is the tool working. What to do about the number is
at **[trymargin.io](https://trymargin.io)**.

⚠️ No account, no signup, and no telemetry — this tool never phones home, including to us. We
have no idea you ran it. (`margin-meter`, published separately, is the opposite by design: it is
the client that *does* send call economics to a Margin ingest API. Different package, different
job, on purpose.)

## Install

```
pip install margin-cost
```

Or run it straight from a clone, since it has no dependencies:

```
python -m margin_cost estate.json
```

## Use

Write a corpus file describing your vendors, then point the tool at it:

```
margin-cost estate.json
margin-cost estate.json --json     # machine-readable
margin-cost --example              # print a sample corpus to edit

# gate your build on it — exit codes, ready for CI
margin-cost estate.json --fail-over 0.14       # exit 1 when cost/outcome is over budget
margin-cost estate.json --fail-if-unmeasured   # exit 1 on an outcome that was guessed
```

Output:

```
COST PER OUTCOME  ·  measured locally, nothing sent anywhere

  vendor                       tier        cost/outcome         spend   passed   retry   cache
  --------------------------------------------------------------------------------------------
  support-agent (self-hosted)  measured         $0.0182         $0.04        2   32.7%   14.0%
  Glean                        allocated        $4.0323     $5,000.00     1240       —       —
  Zapier                       uncovered   NOT MEASURED       $300.00        0       —       —

COVERAGE  ·  a vendor is only 'measured' if you gave it the calls
  measured         $0.04   real tokens, real per-call cost
  allocated    $5,000.00   invoice ÷ outcomes
  uncovered      $300.00   spend seen, no outcomes — NOT priced
  ------------------------
  total        $5,300.04   94.3% of spend has a cost per outcome
```

## The three tiers, and why the tool will not overstate them

Spend cannot always be measured to the same depth. The tool names the depth it
had rather than pretending to more:

- **measured** — you gave it the per-call rows, so it sees real tokens and real
  per-call cost. Full metrics: cost per outcome, retry tax, cache efficiency,
  token yield.
- **allocated** — you gave it an invoice total and an outcome count, so it
  divides. A real cost per outcome, coarser than metered, and the only per-token
  rates it can honestly show are none.
- **uncovered** — you gave it spend and no outcomes. It reports the money and
  says the per-outcome cost is not measured.

The tier is derived from the evidence in your file. You cannot mark a vendor
`measured` by writing it in the JSON; only per-call rows earn that. A cost per
outcome with no data behind it prints `NOT MEASURED`, never a flattering zero.

## Gate your builds on it

A report is a number you read. A gate is a decision your pipeline can act on,
and the exit code is the whole contract:

| exit | meaning |
|---|---|
| `0` | pass — the gate ran and nothing failed it |
| `1` | the gate says no — a measured number is over budget, or an outcome was guessed |
| `2` | could not run — unreadable input, or no outcome in the estate to price |

`--fail-over 0.14` compares the estate's blended cost per outcome (the same
measured-plus-allocated number the report prices) against your budget and exits
`1` when it strictly exceeds it. `--fail-if-unmeasured` exits `1` the moment any
vendor's outcome is not positively determined — spend with no outcome count, or
a metered vendor whose calls all failed. That second flag is the honesty rule
made executable: a cost-per-outcome whose denominator was assumed is a
fabricated number, so the gate refuses rather than passing a guess.

Wire it up like any other CI check:

```yaml
# .github/workflows/ai-spend.yml
name: ai-spend gate
on:
  schedule:
    - cron: "17 9 * * 1"   # every Monday; run it whenever your corpus updates
jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install margin-cost
      - run: margin-cost estate.json --fail-over 0.14 --fail-if-unmeasured
```

`exit 2` exists so a CI author can tell a refusal from a failure — a corpus that
stops producing outcomes stops the pipeline with "could not run", not with a
pass.

## What it does not do

It measures. It does not act. There is no routing, no model swapping, no "we
will cut this for you." That line is deliberate: measuring your spend is
something you should be able to do yourself, for free, without trusting anyone.

## Corpus format

A JSON object with a `vendors` list. Each vendor is one of three shapes:

```json
{
  "vendors": [
    {
      "name": "support-agent (self-hosted)",
      "calls": [
        {"cost_usd": 0.0121, "input_tokens": 820, "output_tokens": 190,
         "cache_read_tokens": 400, "is_retry": false, "outcome": "pass"}
      ]
    },
    {
      "name": "Glean",
      "invoice_usd": 5000.0,
      "outcomes": {"passed": 1240, "total": 1500}
    },
    {
      "name": "Zapier",
      "invoice_usd": 300.0
    }
  ]
}
```

- A vendor with `calls` is measured. Each call needs `cost_usd`; tokens, the
  retry flag, and the `pass`/`fail` outcome are optional. An outcome that is
  neither `pass` nor `fail` is refused, not guessed.
- A vendor with `invoice_usd` and a positive `outcomes.passed` is allocated.
- A vendor with `invoice_usd` and no outcome count is uncovered.

### Invoices as CSV

Invoices usually arrive as a spreadsheet export, so a CSV works directly:

```
margin-cost invoices.csv
```

```
vendor,invoice_usd,passed,total
Glean,5000.00,1240,1500
Siena,2200.00,860,900
Zapier,300.00,,
```

Headers are matched loosely (`vendor`/`name`/`service`, `invoice_usd`/`spend`/`cost`,
`passed`/`resolved`, `total`). A `$` and thousands commas in the spend column are
fine. A CSV is the invoice path, so every row is `allocated` or `uncovered`; the
`measured` tier needs per-call rows, which are JSON.

## License

MIT.
