Metadata-Version: 2.4
Name: lunarlabs-core
Version: 0.1.0
Summary: The product-neutral Lunar Labs spine: capture, deterministic verification, fail-open escalation, training-pair extraction, and the fitness gate.
Author: Lunar Labs
License: MIT
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Dynamic: license-file

# lunarlabs-core

**The product-neutral spine shared by Lunar Labs products.**

```
candidate (cheap/local) → verify (deterministic oracle) → pass?  ✓ done
                                                        → fail/abstain?  ↑ teacher (frontier) → verify
                                                                              ↓
                                        append-only trace capture (both verdicts label-worthy)
                                                                              ↓
                    training-pair extraction (retrieval-task Q→A, NEVER raw transcripts)
                                                                              ↓
                          nightly consolidation → FITNESS GATE (promote only if deflection
                                                  improves without regression) → ship or roll back
```

Two products are built on this spine:

- **Artemis** — a small-model-first cascade for tool calls. It uses the spine to
  deflect spend from frontier models to on-device models; its four oracle
  families (agreement, provenance, structural, execution) are product-specific
  and stay in Artemis.
- **Avalon** — an adaptive AI runtime. It uses the same spine for its own
  capture → verify → consolidate → gate loop over runtime behavior. Avalon's
  vision is much larger than this library; `lunarlabs-core` is only the shared
  contracts both products import so the safety semantics (abstention, fail-open,
  gating) are defined — and tested — exactly once.

This library is stdlib-only and deliberately boring. What is shared is the
*contract*; what is measured, drafted, and trained stays in each product.

## What the spine defines

| module | contract | who consumes it |
|---|---|---|
| `verdict.py` | The three-valued verdict: PASS / FAIL / **ABSTAIN**, expressed as `ok` × `label`. | Both products' oracles. |
| `verifier.py` | The `Oracle` protocol (`output → Verdict`), `all_of` composition, one trivial example oracle. | Both products plug in their own domain oracles. |
| `escalation.py` | `verified_cascade`: candidate-first routing, **fail-open** on any internal error, `LUNARLABS_*` kill switch. | Artemis's router; Avalon's runtime decision path. |
| `trace.py` | `TraceRecord` + append-only JSONL `TraceWriter`, with the dual-label capture gate as the only write path. | Artemis's escalation capture; Avalon's observed-event capture. |
| `pairs.py` | `TrainingPair`: retrieval-task Q→A with provenance. **Never raw transcripts.** | Both products' nightly consolidation / LoRA pipelines. |
| `fitness.py` | `fitness_gate`: pure decision over measured metrics — promote only on meaningful deflection gain with no keep-set regression. | Both products' nightly retrain jobs. |

## The ABSTAIN third verdict is load-bearing

Every oracle answers three things, not two. When a check cannot *mechanically*
decide, it declines:

- **PASS + label** — route on it, and it is strong enough to write training data.
- **FAIL + label** — real evidence of a wrong answer; the pair may be captured.
- **ABSTAIN** (`ok=False, label=False`) — escalate (costs money, never ships a
  wrong answer) but the trace writer **refuses the pair**, so it poisons
  nothing. Training on abstentions would teach the model the reference's
  arbitrary surface conventions, which raises future agreement, which inflates
  measured deflection: a self-fulfilling metric.

`TraceWriter.record_escalation` enforces both conjuncts — the teacher's pass
must be label-worthy AND the candidate's failure must be label-worthy — so a
weak oracle cannot poison the retrain it feeds.

## Fail-open, and the kill switch

Every spine-internal failure — candidate down, timeout, malformed reply, a bug
in the verifier itself — routes to the teacher instead of raising. Breaking
the product loses the user permanently; a missed deflection costs a fraction
of a cent. Nothing in `verified_cascade` raises past its caller.

```bash
export LUNARLABS_DISABLED=1     # every call goes straight to the teacher
export LUNARLABS_DEFLECT=off    # same effect, second switch
```

Both are read fresh on every call — flipping the env var takes effect without
a restart or redeploy.

## Training pairs are retrieval-task Q→A — never raw transcripts

From the JARVIS research: a raw transcript trains surface imitation (phrasing,
length, the teacher's tics) and drowns the signal you paid an escalation for.
A Q→A pair isolates exactly the thing the candidate got wrong: *given THIS
observed context, the correct answer is THAT.* Every `TrainingPair` therefore
has `task_type="retrieval"`, a question projected from the trace's context,
the teacher's verified output as the answer, and full provenance (source trace
id, candidate failure stage/reasons). A pair without provenance is
unauditable and should be refused downstream.

## Usage

```python
from lunarlabs_core import (
    TraceWriter, verified_cascade, non_empty_oracle,
    extract_pairs, write_pairs_jsonl, FitnessMetrics, fitness_gate,
)

trace = TraceWriter("traces.jsonl")

result = verified_cascade(
    candidate=lambda: small_model.answer(request),   # cheap path first
    teacher=lambda: frontier_model.answer(request),  # only on verified failure
    verify=my_oracle,                                # your deterministic oracle
    context={"request_id": "..."},                   # opaque to the spine
    trace=trace,
)
result.handled_by    # "candidate" | "teacher"
result.failed_open   # True if the spine broke and got out of the way

# Nightly consolidation:
pairs = extract_pairs(trace.records())
write_pairs_jsonl(pairs, "train.jsonl")

decision = fitness_gate(
    FitnessMetrics(deflection_rate=0.61, keep_set_score=0.92),  # base, measured by you
    FitnessMetrics(deflection_rate=0.68, keep_set_score=0.92),  # candidate retrain
)
print(decision.summary())   # [PROMOTED] deflection 61% -> 68% (more deflection, no regression)
```

## What is deliberately NOT here

- Artemis's four oracle families, its providers/proxy, pricing, audit and
  dashboard — cost/tool-call-specific, they stay in Artemis.
- Avalon's runtime, planners, and consolidation wiring.
- Any trainer. `fitness.py` decides; it does not train. Each product owns its
  LoRA/fine-tune stack and its metric measurement.

## Docs

- `docs/data-volume-falsifier.sql` — the JARVIS data-volume falsifier: SELECT-only
  per-day counts of Carter-signal (user messages, observed-provenance commitments,
  episodic memories, user facts, eval scorecards) over the last 30 days, to test
  whether the real system produces enough usable training pairs/day to feed the
  nightly trainer (the "~50/day starves it" question). Run with
  `psql "$DATABASE_URL" -f docs/data-volume-falsifier.sql`.

## Development

```bash
python -m pytest        # stdlib-only package; pytest is the only dev dependency
```

MIT © Lunar Labs
