Metadata-Version: 2.5
Name: meme-core
Version: 0.2.0
Summary: Memory-Compute Pareto analysis core: memory strategies (guarded + wording-only 2x2) and solver-based Best-of-N
Project-URL: Homepage, https://github.com/abell12134/meme-llm
Project-URL: Documentation, https://github.com/abell12134/meme-llm#readme
Project-URL: Repository, https://github.com/abell12134/meme-llm
Project-URL: Issues, https://github.com/abell12134/meme-llm/issues
Project-URL: Changelog, https://pypi.org/project/meme-core/#history
Author-email: Abel <abel@example.org>
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: best-of-n,compute,constraint-solver,inference,llm,memory,pareto,small-language-models,structured-memory,test-time
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Scientific/Engineering :: Mathematics
Requires-Python: >=3.10
Provides-Extra: dev
Requires-Dist: black>=23.0; extra == 'dev'
Requires-Dist: mypy>=1.0; extra == 'dev'
Requires-Dist: pytest-cov>=4.0; extra == 'dev'
Requires-Dist: pytest>=7.0; extra == 'dev'
Requires-Dist: ruff>=0.1; extra == 'dev'
Description-Content-Type: text/markdown

# meme-core

> **Memory-Compute Pareto analysis core** — memory strategies (with the guard's 2×2 decomposition) and solver-based Best-of-N for small language models

[![PyPI version](https://img.shields.io/pypi/v/meme-core.svg)](https://pypi.org/project/meme-core/)
[![Python](https://img.shields.io/pypi/pyversions/meme-core.svg)](https://pypi.org/project/meme-core/)
[![License](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](LICENSE)

## What is this?

**meme-core** extracts the reusable parts of the [MEME project](https://github.com/abell12134/meme-llm) as a standalone Python library:

### 1. Memory strategies, including the guard's 2×2 (`guard`)

Structured memory **hurts** pure-retrieval tasks: entity-grouped memory (M3) costs
10–50 percentage points in every model we tested (median −30pp on TriviaQA-lite,
5/5 models negative, sign test *p* = 0.0625).

The fix is to stop telling the model to trust a block that was derived from the
question. But "the guard" as usually implemented changes **two** things at once,
so it cannot say *which* change does the work. This package therefore ships the
full 2×2:

|              | block wording (`Using ONLY the facts above`) | question wording (`the QUESTION is authoritative`) |
|--------------|---------------------------------------------|----------------------------------------------------|
| **no fallback**  | `M2` / `M3` / `M4`                         | `M2W` / `M3W` / `M4W` (wording-only)               |
| **with fallback** | `M2G` / `M3G` / `M4G` (prose fallback)     | `M2Q` / `M3Q` / `M4Q`                             |

What the experiments behind this package found (full numbers in the paper):

- On **retrieval** tasks the two factors are *substitutes*: the prose fallback
  alone restores the whole loss in 14 of 14 cells that have one, the
  question-authoritative wording alone in 8 of the 14.
- On **constraint** tasks the fallback is inert (it never fires when there *is*
  material to structure) and the question-authoritative wording is **harmful**
  (−10 to −40pp) — there the block *is* the material.

So the rule is task-dependent, not universal: if the task has nothing to
structure, use prose; if it has everything to structure, keep the block
authoritative.

### 2. Solver-based Best-of-N (`solver_bon`)

Best-of-N without a verifier has no selection criterion and degenerates to a
random sample. Using a constraint solver as the reward model makes BoN
deterministic and free of extra LLM calls on constraint-satisfaction tasks.

**Key insight (MEME v3.9+):**
- For tasks with verified unique solutions (ZebraLogic), the constraint solver **is** the ground truth
- No LLM call needed for verification — deterministic, reproducible
- On our leak-free hard set (20 problems), solver-selected Best-of-N lifts the
  3B model from 40% to 85% (+45pp, McNemar *p* = 0.022) — the only significant
  *positive* cell in the whole study, and the only intervention whose sign we
  can predict before running it

`meme_core.answer.extract_answer` is the answer-extraction rule used for every
number we report, so re-scoring stored samples with this package reproduces the
paper's accuracies.

## Installation

```bash
pip install meme-core
```

Or from source:

```bash
pip install -e .
```

## Quick Start

### The 2×2: fallback × wording

```python
from meme_core.guard import GuardedMemory, Problem

problem = Problem(
    id=1,
    question="Who is the engineer?",
    expected="Alice",
    constraints=[
        "Alice is the engineer",
        "Bob is the doctor",
    ],
    structure_kind="constraints",
)

# block wording, no fallback
print(GuardedMemory(strategy="M3").render(problem))

# question wording, NO fallback  (the "wording-only" corner of the 2x2)
print(GuardedMemory(strategy="M3W").render(problem))

# question wording + prose fallback
print(GuardedMemory(strategy="M3Q").render(problem))
```

**When `structure_kind="none"`, the guarded strategies fall back to prose:**

```python
problem = Problem(
    id=2,
    question="What is the capital of France?",
    expected="Paris",
    constraints=[],
    structure_kind="none",
)

# M3G -> plain prose (no scaffold, no authority claim)
print(GuardedMemory(strategy="M3G").render(problem))

# M3W -> still renders the (empty) block, because the wording-only corner
#       must NOT get the fallback
print(GuardedMemory(strategy="M3W").render(problem))
```

### Best-of-N with the constraint solver

```python
from meme_core.solver_bon import best_of_n, best_of_n_from_texts
from meme_core.guard import Problem

problem = Problem(
    id=1,
    question="Who is the engineer?",
    expected="lawyer",
    aliases=["advocate", "attorney"],
)

# From extracted candidate answers ...
winner, scores = best_of_n(["doctor", "lawyer", "engineer", "lawyer"], problem, use_solver=True)
print(winner)                      # "lawyer"
print([s.confidence for s in scores])  # [0, 100, 0, 100]

# ... or straight from the raw model responses
texts = [
    "Let me think. The engineer is Bob.\nAnswer: Bob",
    "Alice holds that post.\nAnswer: lawyer",
]
winner, scores = best_of_n_from_texts(texts, problem, use_solver=True)
print(winner)                      # "lawyer"
```

## Memory Strategies

| Strategy | Fallback | Authority wording | Description | Use case |
|----------|----------|-------------------|-------------|----------|
| `M0` | — | — | Raw prose question (baseline) | Control / pure retrieval |
| `M1` | — | — | Full context + enumerate constraints | General purpose |
| `M2` | — | block | Structured bullet constraints | Compression |
| `M3` | — | block | Entity-grouped facts (auto-extracted) | Entity-centric reasoning |
| `M4` | — | block | Knowledge-graph triples | Relationship reasoning |
| `M2W/M3W/M4W` | no | question | Wording-only: the block is always rendered | Isolating the wording factor |
| `M2G/M3G/M4G` | yes | block | Guarded, block-authoritative | Constraint tasks (recommended) |
| `M2Q/M3Q/M4Q` | yes | question | Guarded, question-authoritative | Retrieval tasks (recommended) |
| `B0` | — | — | Long-context baseline (repeat the context) | Context-length baseline |
| `B1` | — | block | RAG baseline (top-k retrieved items) | Retrieval baseline |

`B1` needs the retrieved items, which is where your own retrieval layer comes in:

```python
prompt = GuardedMemory(strategy="B1").render(
    problem, retrieved=["...item 1...", "...item 2..."], k=2, n=10
)
```

## The `structure_kind` Field

Problems declare what auxiliary material they provide, which determines guard behavior:

| structure_kind | Example Task | Guard Behavior |
|----------------|--------------|----------------|
| `constraints` | ZebraLogic | Organize constraints into structured view |
| `quantities` | GSM8K-Hard | Organize numeric givens |
| `options` | ARC | Organize answer choices |
| `facts` | HotpotQA | Organize facts for chaining |
| `none` | TriviaQA, HumanEval | **Fall back to prose** (no scaffold) |

## Architecture

```
meme_core/
├── __init__.py           # Package entry point
├── answer.py             # extract_answer — the answer-extraction rule we score with
├── guard/
│   ├── __init__.py       # Guard module exports
│   ├── core.py           # GuardedMemory class, render_prompt, Problem
│   └── templates.py      # Prompt templates
└── solver_bon/
    ├── __init__.py       # Solver BoN exports
    └── core.py           # ConstraintSolver, best_of_n, normalize_for_comparison
```

## Examples

```bash
python3 examples/guard_factorial.py          # render the 2x2 for one problem
python3 examples/gemma4_26b_improvement.py   # stored gemma4:26b Best-of-N numbers
```

`examples/gemma4_26b_improvement.py` prints **stored** results from the MEME
project (no model calls). Note the task label: the constraint numbers for
gemma4:26b are on **ZebraLogic-easy** (10 problems, `data_sha 252183ea9177`),
not on the 20-problem hard set — the hard set was re-generated after the
leakage audit and gemma4:26b has not been re-run on it.

## Key Results (from the MEME project)

All numbers below are single cells or sign-replication counts, quoted with their
uncertainty. A 10-problem cell cannot resolve a 10pp difference (that is one
problem), so read these as effect sizes, not as verdicts.

### Retrieval: structured memory hurts, in every model

| Task | M0 | M3 (unguarded) | M3G | M3Q |
|------|-----|-----------------|-----|-----|
| TriviaQA (qwen3:1.7b) | 100% | 60% | **100%** | **100%** |
| HotpotQA (qwen3:1.7b) | 90% | 60% | **90%** | **90%** |
| GSM8K-Hard (qwen3:1.7b) | 70% | 70% | **80%** | 70% |
| ARC (qwen3:1.7b) | 90% | 100% | 90% | **100%** |

Cross-family, TriviaQA M3 → M3G: qwen3:1.7b +40pp, qwen3:8b +30pp,
llama3.2:3b +50pp, phi3.5 +30pp — the penalty is present in 5/5 models
(median −30pp, sign test *p* = 0.0625).

### Constraint tasks: the wording must not be relaxed

On the leak-free hard set (H1–H20, 20 problems) the question-authoritative
variants are *worse* than the block-authoritative ones in six of the seven
directly comparable pairs (median −12.5pp: −40, −30, −15, −10, −10, −10pp).
The four significant cells in the whole study are all of this kind
(−35 to −40pp, *p* < 0.05).

### Compute: verifier-based Best-of-N replicates, memory does not

Across 27 model×family Best-of-N cells: 16 positive, 11 ties, **0 negative**.
On the hard set: Qwen3-1.7B 75%→95%, Qwen3-8B 85%→100%, Llama-3.2-3B
40%→85%, Phi-3.5 45%→70% (median +22.5pp). The same four samples spent on a
majority vote instead are worth a median of +7.5pp and are *negative* in one
model. The difference is the selection rule, not the sampling.

## Citation

If you use meme-core in your research, please cite:

```bibtex
@misc{meme2026,
  title  = {Memory is the Cheap Axis: A Paired, Audited Re-analysis of
            Test-Time Memory and Compute for Small Language Models},
  author = {Abel},
  year   = {2026},
  note   = {Preprint, https://github.com/abell12134/meme-llm}
}
```

## References

- [MEME Project (meme-llm)](https://github.com/abell12134/meme-llm) — Full research project with data, code, and results
- [Paper (arXiv)](https://arxiv.org/abs/XXXX.XXXXX) — Full methodology and results (link added on upload)

## License

Apache 2.0 — see [LICENSE](LICENSE) for details.
