# Product Requirements: LearnedSwitch

**Product:** LearnedSwitch — Graduated-Autonomy Classification Primitive
**Owner:** Benjamin Booth
**Status:** Draft
**Last Updated:** 2026-04-19
**Repository:** Standalone (language-agnostic, minimal dependencies)

---

## 1) Elevator Pitch

A drop-in replacement for every `if/else`, `case/switch`, and rule-based
classifier in your system. Starts as a deterministic rule on day one.
Accumulates labeled outcome data. Graduates to an ML classifier when the
evidence justifies it. Falls back to the rule when confidence is low.
Retrains continuously. One primitive, one lifecycle, deployed everywhere
classification decisions are made.

Ship it as a library in Python, Rust, TypeScript, Go, and Java. ONNX for
model portability. Zero required external dependencies beyond the standard
library and an ONNX runtime.

---

## 2) Problem / Opportunity

### The universal pattern nobody has formalized

Every production system has classification decisions: triage a support
ticket, route a query, classify an interaction, assess content quality,
select a retrieval strategy. These decisions start as handcrafted rules
because there's no training data on day one. Over time, outcome data
accumulates — but the rules stay frozen because replacing them with ML
requires a custom migration for each decision point.

**Current approaches and their failures:**

| Approach | Problem |
|----------|---------|
| Hardcoded rules forever | Can't improve; doesn't adapt to distribution shift |
| ML from day one | No training data; cold-start failure |
| Ad-hoc rule→ML migration | Custom per decision point; no reusable pattern; no safety guarantees |
| Shadow mode deployment | Industry practice but not formalized; no confidence-gated fallback; no lifecycle management |
| Online learning systems | Continuous adaptation but no deterministic fallback; can't guarantee safety on edge cases |

**What doesn't exist:** A reusable primitive that:
1. Starts as a rule (zero training data required)
2. Logs every `(input, output, outcome)` triple
3. Trains an ML classifier when sufficient data exists
4. Activates ML only when statistically validated
5. Falls back to the rule on low confidence
6. Retrains continuously from new outcomes
7. Provides formal guarantees about transition safety
8. Works in any language, any system, any domain

### The opportunity

If this primitive exists as a library, every classification decision in
every system gets a free upgrade path from rules to ML — without custom
engineering per decision point. The same library handles signal triage,
intent classification, content moderation, retrieval routing, quality
assessment, and any other classification task.

For Axiom specifically: 6+ classification decisions will use this
primitive, producing a research paper on transition curves and scale
thresholds across a real federated system.

---

## 3) Goals & Success Metrics

- **Primary goal:** A standalone, language-agnostic library that any
  project can adopt to graduate classification decisions from rules to
  ML with formal safety guarantees.

- **Success metrics:**
  - Library published in 5 languages (Python, Rust, TypeScript, Go, Java)
    with package manager distribution (PyPI, crates.io, npm, Go modules,
    Maven Central)
  - Zero required external dependencies beyond ONNX runtime
  - Deployed on 6+ classification decisions in Axiom with measured
    transition curves
  - Research paper submitted documenting scale thresholds and category
    taxonomy
  - At least one external adoption (non-Axiom project using the library)

---

## 4) Key Users / Personas

- **Application developer:** Wants to replace a hardcoded classifier
  with something that improves over time, without building ML
  infrastructure
- **ML engineer:** Wants a standardized primitive for the rule→ML
  transition pattern they implement ad hoc on every project
- **Platform architect:** Wants a uniform classification primitive
  deployed across their system with consistent lifecycle management
- **Researcher:** Wants to study transition curves, scale thresholds,
  and category-specific switching criteria

---

## 5) Scope — Key Capabilities

### 5.1 Core Primitive: LearnedSwitch

A generic classifier with four lifecycle phases:

**Phase 0 — Rule**
- User provides a deterministic classification function
  `rule(input) → output`
- Every invocation logged: `(input_features, output, timestamp)`
- Outcome callback: `record_outcome(input, output, outcome)` — called
  when the system later learns whether the classification was correct
- No ML. No training data required. Ships on day one.

**Phase 1 — Shadow**
- ML classifier trains on accumulated outcome data
- On each invocation: rule executes AND classifier predicts
- Classifier output is NOT used — only logged for validation
- Disagreement log: `(input, rule_output, ml_output, eventual_outcome)`
- Transition criteria: configurable minimum data volume, minimum
  shadow duration, statistical validation

**Phase 2 — ML with Fallback**
- ML classifier is primary when confidence ≥ threshold
- Below threshold: fall back to rule, log the fallback event
- Outcome feedback continues driving periodic retraining
- Confidence threshold is tunable (conservative = high threshold =
  more fallbacks; aggressive = low threshold = more ML)
- Automatic regression detection: if ML error rate exceeds rule
  error rate over a window, auto-revert to Phase 1

**Phase 3 — ML Primary**
- ML classifier handles all decisions
- Rule retained as circuit breaker: triggered on anomaly detection
  (input distribution shift, confidence collapse)
- Continuous retraining on schedule or data-volume trigger
- Full lifecycle telemetry: phase transitions, fallback rates,
  confidence distributions, retraining events

### 5.2 Outcome Logging

The feedback loop that makes LearnedSwitch learn:

```
Invocation:
  switch.classify(input) → output
  
Later (seconds, minutes, days — depends on domain):
  switch.record_outcome(input, output, outcome)
  
Outcome types:
  - CORRECT: classification was right (positive reinforcement)
  - INCORRECT: classification was wrong (negative + correction label)
  - OVERRIDE: human overrode the classification (implicit INCORRECT)
  - ESCALATED: low confidence triggered fallback (neither right nor wrong)
  - UNKNOWN: outcome never observed (excluded from training)
```

The outcome log is append-only, durable, and exportable. It IS the
training data. No separate data pipeline required.

### 5.3 Retraining Policy

Configurable triggers for when to retrain the ML classifier:

| Trigger | Description | Default |
|---------|-------------|---------|
| `data_volume` | Retrain when N new outcomes accumulated | 100 |
| `schedule` | Retrain on time interval | 24 hours |
| `drift_detected` | Retrain when input distribution shifts | KS-test p < 0.01 |
| `error_spike` | Retrain when error rate exceeds threshold | 2x baseline |
| `manual` | Retrain on explicit API call | — |

Retraining produces a new model version. Old model retained for
rollback. Model lineage tracked (version, training data hash,
performance metrics).

### 5.4 Transition Safety

Formal guarantees for phase transitions:

**Rule → Shadow:**
- Minimum outcome volume: configurable (default: 50)
- Automatic when threshold met (no human approval required)

**Shadow → ML-with-Fallback:**
- Statistical test: ML error rate < rule error rate with significance
  p < 0.05 (configurable)
- Minimum shadow duration: configurable (default: 7 days)
- No regression on labeled edge cases (if edge-case test set provided)
- Requires explicit approval if `safety_critical = true`

**ML-with-Fallback → ML-Primary:**
- Sustained ML superiority over configurable window (default: 30 days)
- Fallback rate < threshold (default: 5%)
- No safety-critical edge case failures
- Requires explicit approval if `safety_critical = true`

**Auto-revert (any phase → previous phase):**
- Error rate exceeds 2x baseline for configurable window
- Confidence distribution collapses (mean confidence drops below threshold)
- Input distribution shift detected (KS-test on feature distributions)

### 5.5 Model Format and Portability

- **Training:** scikit-learn (Python) produces the model
- **Export:** ONNX format for cross-language inference
- **Runtime:** ONNX Runtime (C++/Rust/Java/.NET/JS bindings available)
- **Fallback:** For environments where ONNX Runtime is unavailable,
  support decision tree export as native code (generated if/else
  statements in the target language — zero dependency inference)

Model types supported:
- Gradient-boosted trees (XGBoost/LightGBM → ONNX) — default
- Logistic regression (sklearn → ONNX) — for simple decisions
- Random forest (sklearn → ONNX) — for interpretability
- NOT deep learning — LearnedSwitch targets tabular/structured
  classification, not vision/NLP

### 5.6 Feature Extraction

The primitive doesn't prescribe how features are extracted from inputs.
The user provides a feature extractor function:

```
feature_extractor(input: Input) → FeatureVector (numeric array)
```

The library handles:
- Feature normalization (optional, configurable)
- Feature drift detection (distribution shift on individual features)
- Feature importance tracking (which features drive decisions)
- Missing feature handling (configurable: impute, error, or fallback)

### 5.7 Telemetry and Observability

Every LearnedSwitch instance exposes metrics:

| Metric | Type | Description |
|--------|------|-------------|
| `phase` | Gauge | Current lifecycle phase (0-3) |
| `invocations_total` | Counter | Total classifications |
| `ml_decisions` | Counter | Decisions made by ML classifier |
| `rule_fallbacks` | Counter | Decisions falling back to rule |
| `outcomes_total` | Counter | Outcomes recorded (by type) |
| `confidence_histogram` | Histogram | ML confidence distribution |
| `error_rate` | Gauge | Rolling error rate |
| `retrain_events` | Counter | Retraining events |
| `phase_transitions` | Counter | Phase transitions (with direction) |
| `model_version` | Info | Current model version + training hash |

Exposed via callback interface (user provides sink: OpenTelemetry,
Prometheus, Langfuse, file, or custom).

### 5.8 Persistence

LearnedSwitch state must survive process restarts:

- **Outcome log:** Append-only file or database table
- **Current model:** ONNX file on disk
- **Phase state:** JSON file with phase, transition timestamps,
  model version, performance baselines
- **Feature statistics:** Running mean/variance for normalization
  and drift detection

Storage backend is pluggable: filesystem (default), SQLite, PostgreSQL.
The library provides a `Storage` interface; users implement for their
backend.

### 5.9 Multi-Language API Surface

The API should be identical across languages (same method names, same
semantics, same configuration schema):

```python
# Python
switch = LearnedSwitch(
    name="signal_triage",
    rule=my_triage_rule,
    feature_extractor=extract_signal_features,
    labels=["fast_path", "deferred", "urgent"],
    config=SwitchConfig(
        confidence_threshold=0.85,
        retrain_policy=RetrainPolicy(data_volume=100, schedule="24h"),
        safety_critical=False,
    ),
    storage=FileStorage("/var/lib/learned-switch/signal_triage/"),
)

# Classify
result = switch.classify(signal)
# result.output = "fast_path"
# result.source = "ml" | "rule" | "rule_fallback"
# result.confidence = 0.92
# result.phase = Phase.ML_WITH_FALLBACK

# Record outcome (later, when we know if triage was correct)
switch.record_outcome(signal, result.output, Outcome.CORRECT)

# Inspect state
status = switch.status()
# status.phase = Phase.ML_WITH_FALLBACK
# status.model_version = "v3"
# status.outcomes_total = 4280
# status.ml_error_rate = 0.03
# status.rule_error_rate = 0.11
# status.fallback_rate = 0.04
```

### 5.10 Governance & Approval Modes

LearnedSwitch's P1→P2 and P2→P3 transitions update the *active
classifier* without requiring a code redeploy — the ML head is a
versioned data artifact (ONNX file) hot-swapped via pointer update.
That makes these transitions cheap *and* potentially risky, so the
library ships a first-class governance model with a conservative
default.

**Principle: rule stays code, ML head is data.** The user-authored
rule function (Python/Rust/etc.) remains as the circuit-breaker
floor and is never modified by the library. Proposals carry a new
ONNX head + evidence; adoption updates a pointer in
`active.json`. Rollback is also a pointer update. No code redeploy
at any stage.

**Three modes, progressively disclosed:**

| Mode | When to use | What happens on a transition-ready event |
|---|---|---|
| `manual` (default) | Any switch where the author wants to eyeball proposals | Library writes a `RuleUpdateProposal` to `./learnedswitch/proposals/`, invokes `on_proposal` callback if set, and stays on the current active head. Adoption requires explicit approval. |
| `conservative` (opt-in) | Non-safety-critical switches with stable input distributions | Library auto-approves proposals that pass *all* numeric preconditions (see 5.10.3). Adoption is delayed by `adoption_delay_hours` (default 1h) with a notification written synchronously; humans can revert during the window. |
| `strict` (opt-in) | Safety-critical switches; defers to a caller-supplied backend | Library never auto-approves. Every proposal routes to a consumer-supplied `ApprovalBackend` (identity verification, quorum, RACI, whatever the caller needs). |

#### 5.10.1 Zero-config defaults

```python
# Level 0 — works immediately, manual governance
switch = LearnedSwitch(name="signal_triage", rule=my_rule, author="ben")

# Level 1 — opt into conservative auto-approval with defaults
switch = LearnedSwitch(name="signal_triage", rule=my_rule, author="ben",
                       automation="conservative")

# Level 2 — tune a precondition
switch = LearnedSwitch(name="signal_triage", rule=my_rule, author="ben",
                       automation={"mode": "conservative", "min_samples": 5000})

# Level 3 — plug a callback or a custom approval backend
switch = LearnedSwitch(name="signal_triage", rule=my_rule, author="ben",
                       on_proposal=my_notifier,
                       approval_backend=AxiomApprovalBackend(...))
```

Level 0 is the target experience for 80% of users. Each subsequent
level surfaces more knobs without breaking the default path.

#### 5.10.2 Proposal contents

Every proposal includes (see spec §14 for the full schema):

- `old_head_hash`, `proposed_head_hash`, `proposed_at`, `author`
- **Evidence:** old-rule performance on new data, proposed-head
  performance on held-out, N samples, p-value, shadow-period
  divergence, confidence-collapse metrics, `delta_flip_rate`
  (fraction of decisions that change)
- **Context:** detected drift type (input distribution shift,
  confidence collapse, error spike), window, affected feature(s)

Proposals are content-addressed (hashable JSON) so consumers who
mirror them into external stores (Axiom's CompositionService,
etc.) can verify authenticity without custom signing.

#### 5.10.3 Conservative auto-approval — default preconditions

Shipped constants (all must hold); callers override via config dict:

| Precondition | Default | Rationale |
|---|---|---|
| `min_samples` | 1000 | Statistical floor |
| `min_p_value` | 0.01 | Reject weak-effect proposals |
| `min_beat_pct` | 2.0 | Must beat current head by ≥2 absolute % |
| `max_flip_rate` | 0.05 | Proposal may flip ≤5% of decisions vs current |
| `shadow_period_hours` | 24 | Catches "looks good on held-out, breaks in prod" |
| `adoption_delay_hours` | 1 | Human intervention window post-approval |
| `safety_critical` must be false | — | Safety-critical switches always require `strict` |
| `author_unreachable_fallback` | `"manual"` | Library never auto-adopts on timeout — consumer's `ApprovalBackend` can opt into reviewer fallback via RACI if they want that |

Any single precondition violation aborts auto-approval and the
proposal falls through to the manual path (audit log records which
precondition failed, for debugging).

#### 5.10.4 Library vs consumer layer

Everything above ships in the standalone library with zero external
runtime dependencies beyond ONNX Runtime. The library provides:

- `ApprovalBackend` protocol (opaque identity string, pluggable)
- Built-in backends: `ManualBackend`, `ConservativeBackend`
- `on_proposal` callback hook
- CLI: `learnedswitch list | approve <id> | reject <id> | revert`
- File-based proposal store (`./learnedswitch/proposals/*.json`)

Consumer-specific integrations (Axiom, enterprise orgs, SaaS users)
plug in via the `ApprovalBackend` protocol and `on_proposal`
callback. Axiom's consumer layer adds: Axiom-principal signature
verification, `MemoryFragment(episodic)` mirror of proposals, SCAN
daily-brief surfacing, MCP tool for Claude-Code-based approval,
RACI reviewer fallback on author unreachability, webhook fanout.

The library never knows Axiom exists.

---

## 6) Non-Functional / Constraints

### 6.1 Performance

| Metric | Target |
|--------|--------|
| Classification latency (Phase 0, rule) | < 1ms |
| Classification latency (Phase 2+, ONNX) | < 5ms |
| Outcome recording | < 1ms (async append) |
| Retraining (100 outcomes) | < 1s |
| Retraining (10,000 outcomes) | < 30s |
| Memory footprint (per switch instance) | < 10MB |

### 6.2 Dependencies

**Required:** ONNX Runtime (for ML inference in Phase 1+)
**Optional:** scikit-learn (for training — only needed in the process that retrains, not in the inference process)

**Zero-dependency mode:** If ONNX Runtime is unavailable, the library
can export trained decision trees as native code (generated if/else
in the target language). Inference is then zero-dependency. Training
still requires Python + scikit-learn.

### 6.3 Thread Safety

All public methods are thread-safe. Outcome log uses append-only
writes with platform-appropriate locking. Model swap is atomic
(pointer swap, not file replacement during inference).

### 6.4 Backward Compatibility

Outcome log format is versioned and append-only. Old logs readable
by new library versions. Model format is ONNX (industry standard,
backward compatible). Phase state is JSON with schema version.

---

## 7) Repository Structure

```
learned-switch/
  README.md
  LICENSE                          # Apache 2.0
  docs/
    prd.md                         # This document
    spec.md                        # Technical specification
    paper/                         # Research paper drafts
  core/                            # Reference implementation (Python)
    learned_switch/
      switch.py                    # Core LearnedSwitch class
      config.py                    # SwitchConfig, RetrainPolicy
      outcome.py                   # Outcome log, outcome types
      training.py                  # Model training (sklearn → ONNX)
      inference.py                 # ONNX inference wrapper
      drift.py                     # Feature/concept drift detection
      telemetry.py                 # Metrics callback interface
      storage.py                   # Storage interface + FileStorage
      codegen.py                   # Decision tree → native code export
    tests/
    pyproject.toml
  rust/                            # Rust implementation
    src/
    Cargo.toml
  typescript/                      # TypeScript implementation
    src/
    package.json
  go/                              # Go implementation
    learned_switch/
    go.mod
  java/                            # Java implementation
    src/
    pom.xml
  bench/                           # Cross-language benchmarks
  examples/                        # Usage examples per language
```

### Language Implementation Strategy

| Language | Inference | Training | Priority |
|----------|-----------|----------|----------|
| **Python** | ONNX Runtime | scikit-learn (native) | P0 — reference implementation |
| **Rust** | ort (ONNX Runtime bindings) | Calls Python trainer via FFI or subprocess | P1 — performance-critical deployments |
| **TypeScript** | onnxruntime-node / onnxruntime-web | Calls Python trainer via subprocess | P1 — web/Node.js deployments |
| **Go** | onnxruntime-go | Calls Python trainer via subprocess | P2 |
| **Java** | ONNX Runtime Java | Calls Python trainer via subprocess | P2 |

Training always happens in Python (scikit-learn is the standard).
The ONNX model file is the portable artifact. Non-Python
implementations do inference only; training is a separate process
that produces an ONNX file.

---

## 8) Research Paper Deliverables

### 8.1 Scale Thresholds

For each classification decision deployed in Axiom, measure:
- At what outcome volume does ML first outperform the rule?
- What's the transition curve shape (sharp crossover or gradual)?
- How does the crossover point vary by:
  - Number of output classes
  - Feature dimensionality
  - Input distribution stability
  - Outcome latency (immediate vs. delayed feedback)

### 8.2 Category Taxonomy

Classification decisions fall into categories with different switching
characteristics:

| Category | Example | Outcome Latency | Distribution Stability | Expected Crossover |
|----------|---------|-----------------|----------------------|-------------------|
| **Operational routing** | Signal triage, query routing | Seconds-minutes | Moderate (new signal types appear) | ~200-500 outcomes |
| **Quality assessment** | RAG grounding score, content maturity | Minutes-hours | Stable (quality criteria don't change fast) | ~100-300 outcomes |
| **User modeling** | Interaction classification, intent detection | Seconds (follow-up behavior) | Unstable (user behavior shifts) | ~500-1000 outcomes |
| **Policy compliance** | Trust profile suggestion, access classification | Days (audit review) | Very stable (policy changes rarely) | ~50-100 outcomes |
| **Domain-specific** | Nuclear content classification, physics code detection | Variable | Domain-dependent | Research question |

### 8.3 Federation Dimension

When outcome data is aggregated across institutions (federated learning):
- How much faster does the crossover arrive with N institutions
  contributing vs. 1?
- What privacy-preserving aggregation methods work (federated
  averaging, differential privacy)?
- Does cross-institutional data help or hurt when institutions have
  different distributions?

---

## 9) Axiom Deployment Plan

### Deployment Sites (6+ classification decisions)

| # | Decision Point | Location in Axiom | Rule (Phase 0) | Outcome Signal | Priority |
|---|---|---|---|---|---|
| 1 | **Signal triage** | `signals/triage.py` | `(source, type, confidence) → (tier, priority)` | Human escalation overrides | P0 |
| 2 | **Intent classification** | `core/retrieval_policy.py` | Keyword matching → intent category | RAG relevance scores | P0 |
| 3 | **Interaction classification** | `classroom/classifier.py` | Per-turn label assignment | Instructor annotation corrections | P1 |
| 4 | **Knowledge maturity** | `rag/maturity.py` | Threshold rules (citations, age, usage) | CURIO quality gate pass/fail | P1 |
| 5 | **Trust profile suggestion** | `federation/trust.py` | Environmental mapping | Operator override frequency | P2 |
| 6 | **Retrieval source selection** | `core/retrieval_policy.py` | Access-tier filtering | RAG grounding scores | P2 |
| 7 | **Course objective mapping** | `classroom/objectives.py` | Embedding similarity | Instructor corrections | P2 |

### Integration Pattern

Each deployment site follows the same pattern:

1. Define `rule(input) → output` (the current deterministic logic)
2. Define `feature_extractor(input) → numeric_array`
3. Define `record_outcome()` call site (where the system learns
   whether the classification was correct)
4. Instantiate `LearnedSwitch` with configuration appropriate
   to the decision's safety criticality and outcome latency
5. Replace the existing classification call with `switch.classify()`
6. Add telemetry sink (Langfuse or OpenTelemetry)

---

## 10) Risks & Open Questions

### 10.1 Safety-Critical Decisions

Some classification decisions in Axiom are safety-critical
(export-control classification, authorization). These MUST NOT
graduate to ML without explicit human validation of every edge case.
The `safety_critical = true` flag requires human approval for phase
transitions and provides a labeled edge-case test set that must pass
with zero failures before any transition.

**Open question:** Should safety-critical decisions ever reach Phase 3
(ML primary)? Or should they permanently remain at Phase 2 (ML with
fallback) with the rule as an always-active safety net?

**Recommendation:** Safety-critical decisions stay at Phase 2
permanently. The rule is the safety net. ML improves throughput and
accuracy for the common case; the rule catches edge cases. This is
consistent with `spec-security.md §2` — deterministic code authorizes.

### 10.2 Outcome Attribution

For some decisions, the outcome is delayed (trust profile suggestion:
outcome is whether the operator overrides, which may happen days
later) or ambiguous (knowledge maturity: did promotion succeed because
the maturity assessment was right, or because the quality gate is too
lenient?). The research paper should address outcome attribution
methods and their effect on transition curves.

### 10.3 Cold Start in New Deployments

When Axiom is deployed at a new institution, all LearnedSwitch
instances start at Phase 0 (rules). If the federation has mature
switches at other institutions, should the new node bootstrap from
federated model weights? This accelerates the crossover but
introduces distribution mismatch risk.

**Recommendation:** Offer federated bootstrap as opt-in. New node
starts at Phase 1 (shadow) with the federated model, validates
locally before activating. If local validation fails, stays at
Phase 0 with local rules.

### 10.4 ONNX Runtime Size

ONNX Runtime is ~50MB. For minimal deployments (edge nodes, embedded),
this may be too heavy. The zero-dependency mode (decision tree →
native code export) addresses this but limits model complexity to
tree-based classifiers.

---

## 11) Contacts & Links

- **Owner:** Benjamin Booth (bbooth@utexas.edu)
- **Repository:** TBD (standalone, not inside axiom/ or Neutron_OS/)
- **Related Axiom docs:**
  - `spec-security.md §2.6` — Validated classification pattern
  - `design-resolution-protocol.md §4.5` — Intent classification
  - SCAN signal pipeline — `signals/` extractors + synthesizer
  - Trust profile system — ADR-020, trust policy profiles
_Copyright (c) 2026 The University of Texas at Austin and B-Tree Labs. Apache-2.0 licensed._
