Metadata-Version: 2.5
Name: memworthy
Version: 0.2.0
Summary: Decide what an AI system should remember: policy-driven memory gating with typed judgments.
Project-URL: Homepage, https://github.com/AfnanHussain10/Memworthy
Project-URL: Repository, https://github.com/AfnanHussain10/Memworthy
Project-URL: Issues, https://github.com/AfnanHussain10/Memworthy/issues
Author: Afnan Hussain
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: agents,jev,llm,memory,policy
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Typing :: Typed
Requires-Python: >=3.10
Requires-Dist: httpx>=0.27
Requires-Dist: pydantic>=2.6
Requires-Dist: pyyaml>=6
Requires-Dist: typer>=0.12
Provides-Extra: dev
Requires-Dist: fastapi>=0.110; extra == 'dev'
Requires-Dist: matplotlib>=3.8; extra == 'dev'
Requires-Dist: mypy>=1.10; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.23; extra == 'dev'
Requires-Dist: pytest-cov>=5; extra == 'dev'
Requires-Dist: pytest>=8; extra == 'dev'
Requires-Dist: python-dotenv>=1; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Requires-Dist: types-pyyaml; extra == 'dev'
Requires-Dist: uvicorn>=0.29; extra == 'dev'
Provides-Extra: jev
Provides-Extra: mem0
Requires-Dist: mem0ai<2.3,>=2.2; extra == 'mem0'
Description-Content-Type: text/markdown

<h1 align="center">
  <picture>
    <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/AfnanHussain10/Memworthy/main/assets/logo-dark.png">
    <img alt="Memworthy" src="https://raw.githubusercontent.com/AfnanHussain10/Memworthy/main/assets/logo-light.png" width="380">
  </picture>
</h1>

<p align="center"><strong>Memory should be decided, not just extracted.</strong></p>

<p align="center">
  <img alt="Memworthy playground: a trip is ignored, a move updates memory, and a correction rolls it back" src="https://raw.githubusercontent.com/AfnanHussain10/Memworthy/main/assets/demo.gif" width="800">
</p>

Agent memory usually fails at the decision step, not at retrieval. Extraction pipelines are generous, so memory fills up with errors:
- temporary states stored as permanent facts ("in Dubai this week" becomes "lives in Dubai")
- plans stored as facts
- other people's facts stored as the user's
- stale values and ignored corrections
- the assistant's guesses saved as things the user said
- secrets
- no record of why anything was kept

Memworthy is a small Python library that sits between whatever proposes memories and whatever stores them. For each candidate it:
1. runs deterministic code checks
2. asks a decision model one batch of typed questions (TypeSafe's Jev by default; OpenAI's Decisions API, Microsoft-Decision-1 on Foundry or your own provider also work)
3. applies an ordered, user-editable policy to choose an action: `store`, `update`, `merge`, `supersede`, `reject`, `review`, `redact` or `store_labeled`

Every decision is written to a ledger you can audit, replay under a changed policy, and review.

## Quickstart

```bash
pip install "memworthy[jev]"              # Python 3.10+
export TYPESAFE_API_KEY=...             # https://typesafe.ai
```

```python
from memworthy import Candidate, DictStore, Gate

gate = Gate(policy="personal-memory", store=DictStore())
decisions = gate.ingest_messages([
    {"role": "user", "content": "I live in Dubai"},
    {"role": "user", "content": "Flying to Riyadh tomorrow"},
    {"role": "user", "content": "I moved to Riyadh last week"},
    {"role": "user", "content": "Actually I'm only considering it"},
])
for d in decisions:
    print(f"{d.action:10} {d.rule:16} {d.candidate.text}")
# store      rules[9]         I live in Dubai
# reject     not_memory       Flying to Riyadh tomorrow
# update     update_existing  I moved to Riyadh last week
# supersede  retracted        Actually I'm only considering it   (rolls back to Dubai)
```

The output above is what the bundled recorded Jev answers give; a live run may differ slightly. You can reproduce it with no API key:

```python
from memworthy import RecordedJudge
from memworthy.judges.recorded import bundled_fixture_path

gate = Gate("personal-memory", DictStore(),
            judge=RecordedJudge(bundled_fixture_path("personal-memory")))
```

Each decision also lands in `memworthy.ledger.jsonl` with every signal, its probability, the rule that fired, and the policy and model version.

### Other integrations

```python
# Wrap a Mem0 client: only approved content reaches Mem0 (pip install "memworthy[jev,mem0]")
from mem0 import Memory
from memworthy import GatedMemory
memory = GatedMemory(Memory(), policy="personal-memory")          # mode="input" or "candidate"
memory.add([{"role": "user", "content": "I'm in Lisbon this week"}], user_id="alice")  # rejected
```

```bash
# Turn Claude Code / Codex sessions into an Obsidian-compatible Markdown knowledge base
memworthy run dev-sessions ~/.claude/projects --store markdown:~/kb
memworthy run dev-sessions ~/.claude/projects --store markdown:~/kb --extractor llm   # OpenRouter via EXTRACTOR_MODEL
```

## CLI

| Command | What it does |
| --- | --- |
| `memworthy test POLICY [--judge recorded\|mock\|jev\|openai\|microsoft\|PLUGIN] [--pairs FILE]` | Runs the policy's tests; exits 1 on failure (CI-friendly with recorded answers) |
| `memworthy run POLICY SOURCE [--store dict\|markdown:PATH\|mem0] [--dry-run]` | Gates a candidate file, chat JSON or session folder |
| `memworthy replay LEDGER POLICY [--only-changed]` | Shows which past decisions a policy change would flip (no API calls when prompts are unchanged) |
| `memworthy lint POLICY [--strict]` | Overlapping types, unused signals, unreachable rules, vague prompts |
| `memworthy review LEDGER [--export-tests FILE]` | Accept, override or skip queued decisions; overrides become tests |
| `memworthy record POLICY INPUT [--judge jev\|openai\|microsoft\|PLUGIN]` | Records live judge answers as fixtures |
| `memworthy eval POLICY` | Contrast-pair metrics, calibration and plots into `eval/results/` |

## Judge providers

The judge answers the typed questions. Pick one with `--judge` or `MEMWORTHY_JUDGE`, or pass any object with a `name` and an `async judge(state, questions)` method to `Gate(judge=...)`.

| Provider | Configuration |
| --- | --- |
| `jev` (default) | `TYPESAFE_API_KEY`, optional `TYPESAFE_BASE_URL`, `MEMWORTHY_MODEL` (default `jev-1.13.0`) |
| `openai` | `OPENAI_API_KEY`, optional `OPENAI_BASE_URL`, `OPENAI_DECISIONS_MODEL` (default `gpt-6-luna`); uses the [Decisions API](https://developers.openai.com/api/docs/guides/decisions), in public beta |
| `microsoft` | `AZURE_FOUNDRY_ENDPOINT`, `AZURE_FOUNDRY_API_KEY`, optional `AZURE_FOUNDRY_DEPLOYMENT` (found automatically when unset); [Microsoft-Decision-1](https://learn.microsoft.com/en-us/azure/foundry/foundry-models/how-to/use-foundry-models-microsoft-decision) deployed in your Foundry resource (API-key auth) |

The bundled templates' thresholds and evaluation results come from Jev. Other providers' probabilities are on a different scale, so run `memworthy eval` with your judge before you trust a template's cut-offs.

To ship your own provider as a package, implement the `memworthy.Judge` protocol (see `memworthy/judges/openai_decisions.py` for a complete HTTP example) and register a zero-argument factory that reads its settings from the environment:

```toml
[project.entry-points."memworthy.judges"]
myprovider = "memworthy_myprovider:MyJudge"
```

If your provider speaks the System One API (the format Jev and Microsoft-Decision-1 use), you don't need to write the request and response mapping: return a configured `memworthy.SystemOneJudge(api_key=..., base_url=..., model=..., auth="bearer" or "api-key")` from your factory.

After `pip install`, `--judge myprovider` and `MEMWORTHY_JUDGE=myprovider` work. `memworthy.available_judges()` lists what is installed.

## Templates

Both are ordinary YAML files in `src/memworthy/templates/`; copy one and edit it.

**`personal-memory`** (v1.1) is for consumer assistants.
- **Types:** fact, preference, relationship, temporary, intent, hypothetical, noise.
- **Rules, in order:**
  1. Redact secrets.
  2. Reject anything the user didn't say.
  3. Roll back retractions.
  4. Reject temporary states, plans, hypotheticals and noise.
  5. Reject future or time-bounded statements. Dates are parsed by code, not the judge.
  6. Review sensitive data (health, finances, religion, precise location).
  7. Reject low-durability facts.
  8. Review facts about someone else.
  9. Update memories the new fact replaces.
  10. Store the rest.

```yaml
rules:
  - {name: secrets, when: "secret", then: redact}
  - {name: not_user, when: "not from_user", then: reject}
  - {name: retracted, when: "retraction > 0.7 and conflict", then: supersede, restore: true}
  - {name: not_memory, when: "type in ['temporary', 'intent', 'hypothetical', 'noise']", then: reject}
  - {name: bounded_time, when: "timing == 'bounded' or timing == 'future'", then: reject}
  - {name: sensitive, when: "sensitivity >= 1.5", then: review}
  - {name: low_durability, when: "durable < 0.6 or explicit < 0.5", then: reject}
  - {name: wrong_subject, when: "about_subject < 0.5", then: review}
  - {name: update_existing, when: "conflict and conflict.p > 0.6", then: update}
  - {else: store}
```

**`dev-sessions`** (v1.1) turns coding-agent sessions into a knowledge base.
- **Pipeline:**
  1. Sessions are split into episodes.
  2. Each episode is classified with one Jev call, and noise is dropped.
  3. Candidates are extracted.
  4. Each candidate is gated.
- **Types:** task, fix, decision, discussion, explanation, concept, convention, noise.
- **What happens to each kind:**
  - Unverified fixes and tasks are labeled `unverified`, meaning no passing tests, build, commit or user sign-off.
  - Explanations not grounded in files read are labeled.
  - Unfinished discussions become open questions.
  - Proposals the user did not endorse go to review.
  - Abandoned approaches and generic textbook content are rejected.
  - Later decisions supersede earlier ones, and the old entry moves to `_archive/`.
  - Secrets are redacted before the judge sees them.

Policies use a small, safe expression language (a hand-written parser, never `eval`). For anything it can't express, register Python checks and rules with `@memworthy.check` and `@memworthy.rule`.

## Evaluation results

All numbers come from real `jev-1.13.0` answers, regenerated with `memworthy eval <policy>`. Full reports, including every failure, are in [`eval/results/`](https://github.com/AfnanHussain10/Memworthy/blob/main/eval/results/README.md).

Contrast pairs are held out: one changed word flips the correct action, labels come from the template, and the policies were frozen before the pairs were recorded.

| Template | Cases | Action accuracy | Pair consistency | Target |
| --- | --- | --- | --- | --- |
| `personal-memory` v1.1 | 272 | 97.4% | 94.4% | ≥90% / ≥85%: met |
| `dev-sessions` v1.1 | 236 | 91.5% | 83.9% | pair consistency misses 85% |

- **First run.** Three pair templates had labeling or wording errors and were corrected after the first run. That run scored 94.5% / 88.1% and 87.3% / 75.4%; it is kept in the repo, and the corrections are listed in `eval/results/README.md`.
- **Remaining failures (genuine):**
  - A question that embeds a fact ("Since I live in Vienna, what…") is read as an intent.
  - Failed fixes are rejected as noise instead of being labeled unverified.
  - Some project-specific facts are typed as explanations and labeled ungrounded.
- **Latency and cost.** The median Jev round trip was about 1 s from the author's machine, which misses the 500 ms target. The median cost is about $0.00003 per candidate.
- **Calibration.** Raw ECE is 0.05–0.29 on small samples. See the reports for what the temperature-scaled numbers do and do not mean.
- **Built-in tests are not evidence.** The template tests (124 and 105) pass 100%, but they were used to tune the prompts.
- **LongMemEval knowledge-update, with vs without Memworthy: pending real run.** Run it with `python eval/longmemeval.py longmemeval_s_cleaned.json --limit 78`; details are in `eval/results/longmemeval/README.md`.

## Playground

`playground/` has a FastAPI backend that imports this library and a React front end:
- a guided Dubai → Riyadh demo, one timeline row per message: message → verdict → effect on memory, with "Why?" showing the rule, judge answers and thresholds
- free play with example messages
- a policy panel with one-click experiments, rule switches and a YAML editor; edits replay the conversation and mark changed decisions
- the recorded coding-session demo
- the evaluation page

```bash
pip install -e ".[dev,jev]"
uvicorn playground.backend.app:app --port 8765     # open http://localhost:8765
```

The built front end is committed in `playground/web/`. To change it, edit `playground/ui/` and rebuild:

```bash
cd playground/ui && npm ci
npm run dev        # http://localhost:5173, proxies /api to the backend on :8765
npm run build      # typechecks, then writes ../web
```

Replay mode uses the bundled recorded answers and needs no key. Live mode needs `TYPESAFE_API_KEY`. It is rate-limited per IP (40 calls per hour) and globally (600 per hour), and has a daily spend cap (`PLAYGROUND_DAILY_CAP_USD`, default $0.50). Every decision is labeled recorded, live or needs-live.

## Development

```bash
pyenv install -s 3.12 && python -m venv .venv
.venv/bin/pip install -e ".[dev,jev,mem0]"
.venv/bin/pytest -q && .venv/bin/mypy src && .venv/bin/ruff check .
```

Tests never call live APIs; `pytest -m live` runs a few real Jev calls when `TYPESAFE_API_KEY` is set, and OpenAI Decisions calls when `OPENAI_API_KEY` is set. Design decisions are in `docs/decisions.md` and build status is in `PROGRESS.md`.

## License

Apache 2.0.
