Metadata-Version: 2.4
Name: llm-market
Version: 0.1.1
Summary: Open clearinghouse for LLM bids, councils, and evidence-backed routing.
Author: LLM Market contributors
License-Expression: MIT
Project-URL: Repository, https://github.com/strangeloopcanon/llm-market
Project-URL: Issues, https://github.com/strangeloopcanon/llm-market/issues
Project-URL: Security, https://github.com/strangeloopcanon/llm-market/security/advisories/new
Keywords: llm,routing,llm-market,openrouter,agents
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: dev
Requires-Dist: build; extra == "dev"
Requires-Dist: mypy; extra == "dev"
Requires-Dist: pytest; extra == "dev"
Requires-Dist: ruff; extra == "dev"
Dynamic: license-file

# LLM Market

[![CI](https://github.com/strangeloopcanon/llm-market/actions/workflows/ci.yml/badge.svg)](https://github.com/strangeloopcanon/llm-market/actions/workflows/ci.yml)

LLM Market is an open router for LLM work. Give it a prompt, model cards, prices,
and optional prior evidence; it chooses the process with the best expected value:
one model, a bid-informed single model, or a small council judged into one answer.

The point is simple: stop sending every prompt to the most expensive model when a
cheaper route can match the answer.

## Current Result

The latest local Codex eval in this repo uses six complex multi-step tasks:
code review, source-grounded research, documentation synthesis, patch-policy
review, eval diagnosis, and a server-gateway decision memo.

The first hard run was useful because it failed the product bar: LLM Market
matched quality, but over-spent versus the best cheap single. After ingesting
that run as evidence, the router learned to send the same task family directly
to `codex:gpt-5.4-mini:low`.

Post-ingest result:

| Run | Correct | Mean rubric score | Estimated cost |
| --- | ---: | ---: | ---: |
| `gpt-5.5:xhigh` for every task | 6 / 6 | 1.000 | 0.17546 |
| `gpt-5.4-mini:low` for every task | 6 / 6 | 0.976 | 0.0043865 |
| LLM Market routed setup | 6 / 6 | 0.979 | 0.00438725 |

That is equal quality at about 2.5% of the estimated xhigh cost on this
complex-task slice. Versus the best cheap single, LLM Market is effectively the
same cost after routing to that model directly.

Use this as a local evidence-loop result, not a broad benchmark claim. The
public summary is tracked at `outputs/current/codex_cost_smoke_summary.json`;
it records the aggregate result, limitations, and tracked model-card config, but
not raw provider JSON, stderr, or full answer dumps. Those are left out because
they can contain machine-specific details. Codex CLI does not return billing
data, so production users should replace these estimates with their own provider
prices.

## Quickstart

```bash
python -m pip install llm-market
llm-market plan "Should this prompt use one model, bids, or a council?"
```

For local development:

```bash
python -m pip install -e ".[dev]"
python -m pytest
```

Inspect a route without calling a model:

```bash
llm-market plan "Should this prompt use one model, bids, or a council?"
```

Run locally with deterministic mock models:

```bash
llm-market run --mock --trace "Compare carbon taxes and cap-and-trade."
```

Mock answers are synthetic. Use them to inspect route mechanics, traces, and
cost accounting before spending live provider calls.

The public package and CLI are `llm-market`. The Python import/module is
`llm_market` because Python module names cannot contain hyphens, so
`python -m llm_market` works too.

## Run With Real Models

### Codex CLI

Codex model ids use `codex:<model-slug>:<reasoning-effort>`, for example
`codex:gpt-5.5:low` or `codex:gpt-5.5:xhigh`. The adapter shells out to an
authenticated local `codex exec`.

Plan with the estimated-cost Codex cards:

```bash
llm-market plan \
  --config examples/codex_estimated_cost_models.json \
  "Compute 19 * 23 and give only the answer."
```

Run a live Codex call:

```bash
llm-market run \
  --provider codex \
  --config examples/codex_estimated_cost_models.json \
  --max-tokens 140 \
  "Compute 19 * 23 and give only the answer."
```

By default, Codex provider runs ignore user-level Codex config so evals stay
reproducible. Pass `--codex-use-user-config` when you want the user's local
Codex settings involved. Pass `--codex-oss` when you intentionally want Codex
OSS or local-provider mode, such as Ollama or LM Studio.

### OpenRouter

Set `OPENROUTER_API_KEY`, then use an OpenRouter model-card config:

```bash
export OPENROUTER_API_KEY=...

llm-market run \
  --provider openrouter \
  --config examples/openrouter_models.json \
  --trace \
  "Find the best route for this code-review prompt."
```

## Codex Skill

This repo includes a Codex skill in `skills/llm-market`. Install that folder
as `~/.codex/skills/llm-market` to call LLM Market from any Codex thread:

```bash
mkdir -p ~/.codex/skills/llm-market
cp -R skills/llm-market/. ~/.codex/skills/llm-market/
```

The skill wrapper resolves LLM Market in this order:

1. an explicit `--repo` path
2. `LLM_MARKET_REPO`
3. the current repo or a parent directory
4. the skill directory or a parent directory
5. an installed Python package for plain CLI calls
6. a cached GitHub checkout when repo assets are requested or `--update-github`
   is used

If no checkout is found, the wrapper can clone the GitHub repo into
`$CODEX_HOME/llm-market/repo` and run from that cached copy:

```bash
python ~/.codex/skills/llm-market/scripts/llm_market_cli.py \
  --update-github \
  plan \
  --config examples/codex_estimated_cost_models.json \
  "Should this prompt use one model, bids, or a council?"
```

Run the read-only provider doctor before Codex local-provider work:

```bash
python ~/.codex/skills/llm-market/scripts/codex_provider_doctor.py
```

The skill can distribute the package, examples, eval packs, and wrapper from
GitHub. Live calls still need local provider state: Codex installed and
authenticated, or OpenRouter/OpenAI-compatible API keys. The skill reports local
provider readiness; it does not edit `~/.codex/config.toml`.

PyPI is not required for the Codex skill. The wrapper can clone the GitHub repo
and run LLM Market from source. The PyPI package is useful for plain CLI calls
and ordinary Python installs with `pip install llm-market`.

## How Routing Works

LLM Market makes process selection explicit:

1. `QueryAnalyzer` profiles the prompt for task dimensions, complexity, risk,
   recency, context size, and council tendency.
2. `MarketRouter` scores model cards by capability fit, prior evidence,
   reliability, cost, latency, context window, and diversity.
3. With `--market auto`, non-routine prompts collect compact bids before the
   final route is locked.
4. The router chooses a single model when one route has earned it by utility,
   evidence, or bid dominance.
5. The router chooses a council when the evidence is mixed, bids are close, or
   the prompt needs synthesis, critique, or coverage.
6. `MarketEngine` executes the route, applies fallbacks, and records a trace.
7. `MarketJudge` merges council answers, checks accepted bid promises, preserves
   useful unique findings, and returns the final answer.

Prompt profiling is a weak prior. Bids, evidence, budgets, and observed
outcomes are the load-bearing signals.

## Configuration

Model cards are editable beliefs about cost, latency, reliability, context, and
capability:

```json
{
  "models": [
    {
      "id": "openai/gpt-latest",
      "provider": "openai",
      "capabilities": {
        "coding": 0.94,
        "reasoning": 0.92,
        "writing": 0.82
      },
      "input_cost_per_mtok": 5.0,
      "output_cost_per_mtok": 15.0,
      "context_tokens": 256000,
      "latency_ms": 2500,
      "reliability": 0.94,
      "tags": ["structured", "tool-use", "generalist"]
    }
  ]
}
```

Evidence rows record what worked on prior prompts:

```json
{"model":"openai/gpt-4.1-mini","dimensions":{"coding":1.0},"score":1.0,"correct":true,"complexity":0.32,"risk":0.12}
```

Exact or strongly similar prompt history counts more than broad task-family
averages. A single model needs enough margin before it can suppress council
routing.

## Python API

```python
import asyncio

from llm_market import HeuristicMarketJudge, MarketEngine, MockModelClient
from llm_market.presets import default_model_cards

client = MockModelClient.from_static(
    {
        "openai/gpt-latest": "Strong coding and structured reasoning answer.",
        "anthropic/claude-opus-latest": "Careful critique and risk analysis.",
        "google/gemini-pro-latest": "Broad research and long-context synthesis.",
    }
)

engine = MarketEngine(
    model_cards=default_model_cards(),
    client=client,
    judge=HeuristicMarketJudge(),
)

result = asyncio.run(engine.run("Compare two approaches to a risky migration."))
print(result.answer)
print(result.trace.route.to_dict())
```

## CLI Reference

`plan` explains the route without model calls:

```bash
llm-market plan --format json \
  --expected-output-tokens 800 \
  --max-expected-cost-usd 0.02 \
  "Write a SQL query for weekly retention."
```

Installed-package fallback is intentionally narrower: it can run the CLI and
default in-package model cards, but repo assets such as `examples/`, `outputs/`,
and `skills/` require a source checkout or the cached GitHub checkout. The
real-model commands below assume you are in a source checkout or the skill has
resolved a cached GitHub checkout.

`run` executes a route:

```bash
llm-market run --mock --trace "Should this question use a council?"
llm-market run --mock --market auto --trace "Compare these migration options."
llm-market run --mock --council off --trace "Force single-model routing."
```

Presets set the operating posture without changing model cards:

- `--preset fast`: single-model, no bids, no council; uses evidence and cost
  pressure to prefer cheap earned routes.
- `--preset balanced`: default evidence/cost-aware routing.
- `--preset max`: bid-first and more quality-seeking; allows larger councils,
  one fallback, and lower cost/latency penalties without blindly forcing a
  council.

```bash
llm-market run \
  --provider codex \
  --config examples/codex_estimated_cost_models.json \
  --evidence outputs/current/evidence_codex_current.jsonl \
  --preset fast \
  "Summarize this diff and flag likely bugs."
```

`live-eval` calls real providers and compares the routed answer with baselines:

```bash
llm-market live-eval \
  --provider codex \
  --config examples/codex_estimated_cost_models.json \
  --evidence outputs/current/evidence_codex_current.jsonl \
  --tasks examples/codex_complex_multistep_tasks.jsonl \
  --baseline-mode external \
  --baseline-config examples/codex_complex_baselines.json \
  --provider-timeout-seconds 240 \
  --max-tokens 700 \
  --progress \
  --output outputs/archive/new_codex_run.json
```

Use `--baseline-mode external --baseline-config ...` when you want a clean
comparison against direct baselines such as xhigh-for-everything and cheap
fixed-single arms without direct-running every selectable candidate. Omit
`--baseline-mode` to keep the default full-baseline behavior.

Use `--best-single auto` for product routing. When similar prior outcomes show
that a selectable model repeatedly dominates, the router can use that model
directly.

`evidence-ingest` turns live-eval artifacts into future routing evidence:

```bash
llm-market evidence-ingest \
  --results outputs/archive/<new-run>.json \
  --output outputs/current/evidence_codex_current.jsonl
```

Evidence can include prompt text, run identity, latency, route metadata, and
model outcomes. Redact or keep private any eval artifact that contains customer,
repo, or personal task text before publishing an evidence store.

## Repo Map

- `src/llm_market/analysis.py`: prompt profiling.
- `src/llm_market/routing.py`: utility, cost, latency, diversity, and judge
  selection.
- `src/llm_market/bidding.py`: model self-bids for market routing.
- `src/llm_market/evidence.py`: prior-result observations and evidence deltas.
- `src/llm_market/engine.py`: route execution and fallback behavior.
- `src/llm_market/judge.py`: heuristic judge and LLM judge.
- `src/llm_market/clients.py`: OpenRouter/OpenAI-compatible client and Codex
  CLI client.
- `examples/codex_estimated_cost_models.json`: cost-aware Codex model cards.
- `examples/codex_complex_multistep_tasks.jsonl`: harder rubric-graded Codex
  eval tasks.
- `examples/codex_complex_baselines.json`: direct xhigh and cheap-single
  baselines for complex-task evals.
- `outputs/current/codex_cost_smoke_summary.json`: tracked summary for the
  README result.
- `outputs/current`: current evidence packet and active evidence JSONL.
- `outputs/archive`: archived run-specific reports behind the current packet.
- `skills/llm-market`: Codex skill wrapper and integration notes.

## OpenRouter Fusion Relationship

OpenRouter Fusion made the council primitive legible: ask multiple models, have
a judge compare answers, then produce one response. LLM Market keeps that
primitive open and adds the route policy around it: when to pay for a council,
when to ask for bids, when a cheap model is enough, when prior evidence should
promote one model, and what trace data should be saved for the next prompt.

## Scope

- Codex cost comparisons use estimated relative costs because Codex CLI does not
  expose per-call billing.
- Provider-specific cache pricing, KV-cache accounting, and cache-hit discounts
  are future cost-model work.
- The included live result is a small complex-task eval. Run your own task
  families before trusting any routing policy in production.
- Model cards need current prices, reliability estimates, and observed evidence
  to stay useful.

## Third-Party Notices

See `THIRD_PARTY_NOTICES.md`.
