Metadata-Version: 2.5
Name: artemis-ai
Version: 0.7.0
Summary: Run a small model on-device, escalate only the hard tool calls, and let every escalation train your small model to need the big one less.
Project-URL: Homepage, https://lunarlabs.ai
Project-URL: Repository, https://github.com/LunarLabs-AI/artemis
Author: Lunar Labs
License-Expression: MIT
License-File: LICENSE
Keywords: agents,cost,distillation,llm,on-device,tool-calling
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Libraries
Classifier: Topic :: Software Development :: Quality Assurance
Requires-Python: >=3.10
Provides-Extra: dev
Requires-Dist: pytest>=7; extra == 'dev'
Description-Content-Type: text/markdown

# Artemis

**Run a small model on-device. Escalate only the hard tool calls to a frontier model. Let every escalation train your small model to need the big one less.**

Apps that use AI pay the big, expensive model (Claude, GPT) for *every* tool call, even the trivial ones a tiny local model could handle. Artemis puts a small model on the device, checks its tool calls with a **deterministic, schema-grounded verifier** (not another LLM), and only escalates to the frontier model when the small one is actually wrong. Every escalation is auto-labeled correct/incorrect and saved as training data — so the small model gets better every night, deflects more, and the bill drops while you sleep.

```
small model → verify (deterministic) → pass?  ✓ done: on-device, private, free
                                     → fail?  ↑ frontier → verify → log the fix as training data
                                                                    ↓
                              nightly retrain → FITNESS GATE (promote only if it deflects
                                                more, without regressing) → ship
```

## Why it's not clone-bait

- **The verifier is deterministic.** Parsed? Valid tool? Args match the JSON schema? Optional downstream oracle? Every call is labeled *for free* — no human, no LLM grader. That's what makes the data flywheel real.
- **It compounds and it's gated.** Escalations become a private training set; the fitness gate ships a retrain only when it measurably improves. Your traces and your fine-tuned model are yours.
- **It works with real small models.** Native OpenAI tool-calling *and* a prompted-JSON mode + streaming, so models that don't do native tools (most local ones, including Apple's on-device FM) still work.

## Start with your own bill

Before changing any code, point Artemis at an inference log you already have. It
tells you what you spent and how much of that spend ran through calls a
deterministic verifier could have checked — the ceiling on what Artemis can deflect.

```bash
artemis audit mylog.jsonl                                    # ceiling, rate UNMEASURED
artemis audit mylog.jsonl --html report.html                 # shareable, self-contained
artemis audit mylog.jsonl --model http://localhost:11434/v1  # replace the ceiling with a MEASUREMENT
```

Runs entirely on the machine holding the log. No API key is needed for the
analysis pass and nothing is uploaded — which answers "prove it on my traffic"
and "I'm not sending you my prompts" with the same move. `--html` writes a
report you can forward to whoever owns the budget, with the confidence
statement and every assumption on its face.

It reads OpenAI, Anthropic, generic request/response, and Artemis's own trace
format. What it will not do is flatter you: unpriceable calls are excluded
rather than estimated, free-form prose is reported as not deflection-eligible,
and a log spanning under a day gets **no** monthly projection at all.

## Adopt it with one env var

```bash
artemis serve --upstream https://api.openai.com/v1 --small-model system
export OPENAI_BASE_URL=http://localhost:8787/v1     # that's the whole integration
```

`--mode shadow` forwards every call upstream unchanged and measures what the
small model *would* have deflected — zero behavior change while you decide.
Any Artemis-internal error falls through to your provider untouched.

> **Single tenant only.** Artemis has no concept of a tenant. Every caller of one
> proxy writes full prompt text into one shared trace file, which is the corpus
> the nightly retrain consumes. Do not put one Artemis proxy in front of two
> parties who should not read each other's prompts. `/artemis/stats` and
> `/artemis/report` are unauthenticated; keep the proxy on loopback.

## See it in 30 seconds

```bash
python examples/flywheel.py     # deflection 17% -> 100% after one overnight retrain
python -m artemis dashboard        # writes artemis_report.html — open it
python examples/smoke_live.py   # runs against a REAL on-device model if one's up
```

`examples/flywheel.py` output:

```
DAY 1 (fresh small model)   1/6 on-device (17%) | captured 5 training pairs from the misses
  ...overnight retrain, then the fitness gate...
  [PROMOTED] deflection 17% -> 100% (more deflection, no regression)
DAY 2 (retrained model)     6/6 on-device (100%)
```

Verified live against Apple's on-device Foundation Model: 3/3 tool calls correct, 100% on-device.

## Use it with real models

```python
from artemis import Request, TraceStore, SavingsMeter, route, ollama, local_fm, OpenAICompatibleProvider

small    = ollama("qwen3:4b")                 # local, native tool-calling
# or:    = local_fm("system")                 # Apple on-device FM (prompted mode)
frontier = OpenAICompatibleProvider("frontier", "https://api.openai.com/v1", "gpt-4o", api_key="...")

r = route(Request(system="...", messages=[{"role":"user","content":"..."}]),
          tools, small, frontier, trace=TraceStore(), savings=SavingsMeter())
print(r.handled_by, r.call)   # "small" most of the time, once it's trained
```

Nightly loop:

```python
from artemis import nightly, MockTrainer  # swap MockTrainer for MLXTrainer to fine-tune for real
res = nightly(TraceStore("artemis_traces.jsonl"), small, frontier, eval_cases, MockTrainer(), keep_cases=eval_cases)
print(res.gate.summary())     # only promotes a retrain that deflects more without regressing
```

## Layout

| module | what it is |
|---|---|
| `verifier.py` | deterministic, schema-grounded tool-call verification (the moat seed) |
| `router.py` | the small→verify→escalate cascade |
| `providers.py` | Mock + OpenAI-compatible (native/prompted, streaming); `ollama`, `local_fm` |
| `trace.py` | captures escalations as SFT training pairs |
| `eval.py` | accuracy + deflection measurement |
| `foundry.py` | retrain + **fitness gate** (promote only if better) |
| `dashboard.py` | self-contained HTML savings report |
| `agreement.py` | replay oracle — AGREE / DISAGREE / **ABSTAIN** against your recorded answer |
| `provenance.py` | did the model *read* this value or invent it — works live, zero config |
| `structural.py` | deep schema (nested, `$ref`, `format`, ranges) + the checkability census |
| `execution.py` | did the *world* accept the call — read off the tool result already on the wire |

## The honest part: schema != correctness

The deterministic schema check catches *malformed* calls. It does **not** catch calls that are well-formed but wrong — verified live: a 0.5B model scored 33% and the schema check waved every wrong-but-well-formed call through (0 captured). Reproduced on this repo's own labeled fixture corpus: **the schema check accepts 12 of the 25 tool-call candidates outright, 12 of which are wrong.** That is the failure the oracle families below exist to route around.

```python
route(request, tools, small, frontier, downstream=my_oracle)   # oracle = your correctness signal
```

## Four oracle families, and what each one refuses to claim

Every one is deterministic plain code. No LLM-as-judge anywhere in the labeling path — an LLM judge costs money per call, which destroys the free-label property that is the entire point.

| family | needs | works | catches | cannot see |
|---|---|---|---|---|
| **agreement** | your recorded answer (already in the log) | replay / audit | wrong-but-valid enum labels, wrong values | anything your schema does not define — it **abstains** |
| **provenance** | nothing | **live** | invented ids, placeholders, wrong recipients | computed numbers; legitimately generative text |
| **structural** | your schema | live | nested typos, bad formats, out-of-range, `readOnly` | semantically wrong but well-formed |
| **execution** | the tool result the app already sends back | retrospective | calls the world rejected (4xx) | valid-but-wrong calls that return 200 |

**The distinguishing move is the third verdict.** `compare_calls` returns AGREE, DISAGREE, or **ABSTAIN**, and abstention is load-bearing rather than a hedge. "Legitimately different but equally correct" is the obvious objection to scoring against a recorded answer, and the answer is not to adjudicate it more cleverly — it is to detect *mechanically* that your schema cannot decide, and decline. An abstention escalates (costs money, never ships a wrong answer) and writes **no** training label (poisons nothing). Both failure directions are routed into the safe one.

Equivalence is derived per argument from **your own JSON Schema**, never from meaning: `enum` members and numbers are adjudicable; an open `string` that differs, a datetime resolved from a relative expression, and an optional argument present on one side only are not.

```python
from artemis import verified_agreement_oracle, provenance_oracle, structural_oracle

verify = verified_agreement_oracle(recorded_answer, tools)   # replay: schema AND agreement
verify = provenance_oracle(request_text, tools)              # live: did it invent that literal?
verify = structural_oracle(tools)                            # live: deep schema, zero config
```

### Measured on the labeled fixture corpus (`tests/fixtures_traffic.py`, 11 records / 36 candidates)

Two error directions, named apart because they cost different things. A **false flag** (oracle says wrong, answer was right) escalates needlessly *and* writes a bad training pair — that is the one that poisons. A **false accept** (oracle says fine, answer was wrong) inflates the rate and ships a wrong answer.

```
                                        false flag   false accept   abstained   scored
structural + agreement  (the default)        0             0             6         19
shipped exact-arg match (before this)        3             0             0         25
schema check alone      (ships today)        0            12             0         25
deep structural alone                        0             7             0         25
provenance alone (live, no reference)        0             6             0         22
structured-output oracle (extraction)        0             0             0          6
numeric provenance (grounded prose)          0             0             0          5
```

The corpus is authored and hand-labeled. These are real counts over a stated, inspectable corpus — **not** an estimate of what your traffic would do. That number is what `artemis audit --model` exists to measure on *your* log.

## `artemis audit --model` now measures agreement, not eligibility

This is the change that matters most. The headline used to be *"spend that ran through calls a verifier could check"* — an eligibility ceiling. It is now *"spend on calls your small model answered exactly as your frontier model did"* — a measurement, because **in replay the reference is free**: the frontier's answer is already sitting in the log you handed over.

```
  MEASURED AGREEMENT WITH YOUR RECORDED ANSWERS
  replayed                  : 9 calls against qwen2.5:0.5b
  agreed  (would deflect)   : 1
  disagreed (would escalate): 3
  ADJUDICATED               : 4   -> rate 25%
  ABSTAINED (undecidable)   : 5   -> coverage 44% of replayed calls
         5x  optional present on candidate only
         4x  open string, schema declares no equivalence
```

Real run, real model. Note what it refuses to do: abstentions are in **neither** the numerator nor the denominator, and the report says so on its face. Counting them as hits fabricates, counting them as misses understates, and burying them hides the coverage question.

**It is AGREEMENT, not correctness, and the report says that too.** Its ceiling is your frontier model's own accuracy — where that model was wrong, a small model agreeing with it is scored here as a success. A recorded answer that is empty or fails its own schema is refused the role of ground truth outright.

## Is your catalog even checkable?

`constraint_census` answers, before anyone is promised a deflection rate, whether schema checking can work on your tools at all — a number no observability tool prints.

```python
from artemis import constraint_census
print(constraint_census(my_tools).summary())
# 2 tools, 19 arg slots, 1.05 constraints/arg, 32% nested -> DENSE
```

A **BARE** catalog means schema verification will capture close to zero training pairs, and the zero-capture result above will reproduce exactly. Density is a lever you can pull: adding only the constraints your tool descriptions *already state in prose* (`format: date-time` on a field documented as ISO 8601, an enum copied onto a deprecated alias) is the cheapest capture improvement available.

## The flywheel is real, not a mock

`python examples/train_mlx.py` ran a real LoRA fine-tune of a 0.5B model on captured tool-call data (16GB M1 Pro, 30 iters): **val loss 6.03 → 0.29**. On a held-out prompt:

```
base model  : As an AI language model, I don't have real-time weather data...
fine-tuned  : {"name": "weather", "arguments": {"city": "Phoenix"}}
```

The base model can't tool-call; after training on captured data, the same 0.5B model emits a structured call for a city it never saw. That's the whole promise, running.

## Beyond tool calls: one machine, any verifier

`verified_cascade` is the tool-call cascade with the verifier made pluggable, so the same small→verify→escalate→capture loop works anywhere a cheap objective check exists. Proven in a second domain in `examples/code_cascade.py`: the small model drafts code, **the tests are the oracle**, failures escalate and capture `(task → working code)` pairs into the *same* trace store and retrain. That is the thesis — *spend big compute only where a cheap check can't confirm the cheap answer* — generalizing.

(There's also `artemis/speculative.py`, real token-level speculative decoding via MLX. Honest benchmark on a 16GB M1 Pro with a 0.5B→1.5B gap: 0.48x — *slower*. Token speculation needs a frontier-scale size gap; the action-level cascade is the better lever at this scale.)

## Status

v0.7.0 — **four deterministic oracle families** (agreement / provenance / structural / execution) wired into the existing cascade, `artemis audit --model` reporting a MEASURED agreement rate with explicit abstentions, real token pricing, shareable HTML audit, OpenAI-compatible proxy, fail-open cascade + kill switch, fitness-gated retrain, **real MLX (`mlx_lm lora`) fine-tuning run end-to-end**. Real-model verified: Apple on-device FM, Ollama qwen2.5:0.5b (live replay through the full audit path), a real LoRA fine-tune. **230 tests**, stdlib-only core.

**Next:** the honest gap is that abstention rate and capture rate are measured on an authored corpus, not on customer traffic — point `artemis audit --model` at a real log and count what actually abstains. Then: the retrospective consequence-label channel (a human correcting the model's output hours later is a free, perfect training label, and it is the cheapest unbuilt thing here).

MIT © Lunar Labs
