Metadata-Version: 2.4
Name: examen
Version: 0.3.0
Summary: Self-administered comprehension quiz for AI-assisted code
Project-URL: Homepage, https://github.com/abhiksark/examen
Project-URL: Repository, https://github.com/abhiksark/examen
Project-URL: Issues, https://github.com/abhiksark/examen/issues
Project-URL: Changelog, https://github.com/abhiksark/examen/blob/main/CHANGELOG.md
Author-email: Abhik Sarkar <abhiksark@gmail.com>
License: MIT License
        
        Copyright (c) 2026 Abhik Sarkar
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
License-File: LICENSE
Keywords: code-review,codebase,comprehension,developer-tools,fastapi,local-first,quiz
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Web Environment
Classifier: Framework :: FastAPI
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Education :: Testing
Classifier: Topic :: Software Development :: Quality Assurance
Requires-Python: >=3.11
Requires-Dist: click>=8.1
Requires-Dist: fastapi>=0.115
Requires-Dist: jinja2>=3.1
Requires-Dist: jsonschema>=4.23
Requires-Dist: pydantic>=2.9
Requires-Dist: python-multipart>=0.0.9
Requires-Dist: uvicorn[standard]>=0.32
Provides-Extra: dev
Requires-Dist: httpx>=0.27; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.24; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff>=0.7; extra == 'dev'
Requires-Dist: ty>=0.0.42; extra == 'dev'
Description-Content-Type: text/markdown

# Examen

> It's like an exam. For your own code. You will get a D.

I built this because I kept failing my own code reviews. I'd merge something
Claude wrote, walk into standup the next day, get asked "why did you handle
the edge case that way?" and freeze. Examen runs a 5-question quiz on your
own repo. Each question shows you the code, the answer options, and a
1–5 confidence chip on one screen. You submit both together, so your gut
goes on the record at the same time as your answer.

The results screen plots a heatmap: knowledge × calibration. Top-right is
good. Top-left is dangerous — confident and wrong.

```bash
pip install examen
examen serve quiz.json
```

Local-only. No telemetry. No SaaS. Quiz artifacts generated by the
[`examen` Claude skill](skill/examen/SKILL.md), or hand-rolled (see the
included fixture).

[Try the hosted demo →](https://examen-demo.fly.dev)

---

## Table of contents

- [Try it in 60 seconds (no Claude Code needed)](#try-it-in-60-seconds-no-claude-code-needed)
- [Full setup with Claude Code](#full-setup-with-claude-code)
- [How it works](#how-it-works)
- [CLI reference](#cli-reference)
- [File locations](#file-locations)
- [Known limitations (v0.2.0)](#known-limitations-v020)
- [Development](#development)
- [Troubleshooting](#troubleshooting)
- [Design docs](#design-docs)
- [License](#license)

---

## Try it in 60 seconds (no Claude Code needed)

The repo ships a hand-crafted test fixture. You can take a real 5-question
quiz against it without involving Claude or `code-review-graph` at all — good
for seeing the UX before committing to the full setup.

```bash
git clone <this-repo> examen && cd examen
python3.11 -m venv .venv
source .venv/bin/activate
pip install -e .

examen serve tests/fixtures/example_quiz_small.json
```

Your browser opens at `http://127.0.0.1:8780/start`. Click *Start quiz*, work
through the 5 questions (each one takes your answer and a 1–5 confidence
rating together), and end at a results screen with a topic heatmap and a
calibration score.

The fixture covers fictional auth/billing/api modules — the questions are
toys, but the flow is exactly what real generated quizzes look like.

---

## Full setup with Claude Code

The end-to-end use case is: *generate a quiz from the diff I just merged,
take it, see what I don't actually understand.*

### Prerequisites

- Python **3.11 or newer**
- [Claude Code](https://claude.com/claude-code) installed and working
- [`code-review-graph`](https://github.com/tirth8205/code-review-graph) for
  indexing the target repo's call graph

### One-time install

```bash
# 1. Install Examen from source (not on PyPI yet).
git clone <this-repo> examen && cd examen
python3.11 -m venv .venv && source .venv/bin/activate
pip install -e .

# 2. Drop the Claude Code skill into ~/.claude/skills/examen.
examen skill install

# 3. Install code-review-graph (see its repo for current install command).
pip install code-review-graph
```

### Per-repo setup

In the repo you want to be quizzed on:

```bash
cd /path/to/your-project
code-review-graph build         # indexes the repo; one-time per repo
```

This creates `.code-review-graph/` (gitignored by Examen's gitignore template,
but if you're adding to an existing project add it yourself).

### Generate and take a quiz

Inside that project, open Claude Code and type one of:

```
/examen:generate-diff                            # quiz on git diff vs main
/examen:generate-path src/auth                   # quiz on a directory
/examen:generate-repo                            # quiz on the whole repo
```

The skill writes a quiz artifact to `.examen/quizzes/quiz-<uuid>.json` in
the repo root and prints:

```
Ready. Run: examen serve .examen/quizzes/quiz-<uuid>.json
```

Run that command (from any shell, doesn't need to be Claude Code), and the
browser opens at `http://127.0.0.1:8780/start`.

---

## How it works

Two halves, one JSON contract between them.

```
┌─────────────────────────────────────────────────────────────────┐
│  Generator  (Claude Code skill: examen)                         │
│  - Reads code through code-review-graph MCP tools                │
│  - Claude authors questions grounded in graph facts              │
│  - Emits quiz-<uuid>.json (the artifact)                         │
└──────────────────────────┬──────────────────────────────────────┘
                           │ JSON file on disk
                           ▼
┌─────────────────────────────────────────────────────────────────┐
│  Runner     (Python, FastAPI + htmx + SQLite)                   │
│  - `examen serve <quiz.json>` opens browser at localhost         │
│  - Confidence committed alongside the answer (server-locked)     │
│  - Brier-style scoring: confident-and-wrong → 0.0, uncertain-    │
│    and-wrong → 1.0 (perfectly calibrated honesty)                │
│  - Two-channel heatmap: correctness × miscalibration            │
│  - Longitudinal history in ~/.examen/history.db                  │
│                                                                  │
│  HARD RULE: runner never calls an LLM, never makes network       │
│  requests beyond serving on 127.0.0.1.                           │
└─────────────────────────────────────────────────────────────────┘
```

The **artifact format is the contract.** Anyone could write a different
generator (different model, manual authoring, custom static analysis) and the
runner doesn't change.

### Why confidence-alongside-answer matters

A multiple-choice question with 4 options gives you a 25% chance of guessing
right. Without measuring confidence, "I got it right" doesn't distinguish
between *knowing* and *guessing*. With a confidence chip submitted alongside
each answer:

| Confidence | Correctness | Brier score | Interpretation |
|---|---|---|---|
| 5 (sure) | 1.0 | **1.00** | You knew it. |
| 5 (sure) | 0.0 | **0.00** | You were confidently wrong — the dangerous case. |
| 1 (no idea) | 0.0 | **1.00** | You said you didn't know, and you didn't. Calibrated. |
| 1 (no idea) | 1.0 | 0.44 | Got lucky — partial credit only. |
| 3 (medium) | 1.0 | 0.75 | Right with reasonable uncertainty. |

The "confidently wrong" cell is what the tool is hunting. The heatmap surfaces
**miscalibration** separately from **score** so a topic where you scored 0.6
but were wildly overconfident gets flagged with a ⚠.

---

## CLI reference

```
examen serve <artifact> [--port N] [--host ADDR] [--history-db PATH] [--no-browser]
```
Open the quiz UI in a browser for a given artifact. Default port `8780`;
picks the next free port if taken. `--no-browser` is useful for SSH/headless.
`--host` defaults to `127.0.0.1` (local-only); pass `0.0.0.0` to expose on all
interfaces, e.g. inside a container.

```
examen validate <artifact>
```
Validate an artifact against the schema without serving it — schema
violations, unsupported `schema_version`, and topic cross-reference errors.
Exits non-zero on any problem. Useful for catching bad generator output early.

```
examen history [--repo PATH]
```
List past sessions across all repos (5 most recent per repo), or filtered to
one repo's full history with `--repo /path/to/repo`.

```
examen trend --repo PATH [--topic LABEL]
```
ASCII bar chart of overall score over time for one repo; pass `--topic` to
plot just that topic's trend ("am I getting better at auth?").

```
examen revise --repo PATH [--limit N] [--questions N]
```
Surface a repo's weakest topics (worst score first, with calibration flags)
and print a ready-to-paste Claude Code command that regenerates a quiz focused
on those topics' files. Closes the loop: quiz → see weak spots → re-quiz them.
The runner never calls an LLM; generation stays in the skill.

```
examen prune --older-than 30d|6w|6mo|1y [--yes]
```
Delete completed sessions older than the given duration. Cascades through
answers and topic-snapshots. Confirms unless `--yes`.

```
examen skill install [--dest PATH]
```
Copy the Claude Code skill bundle to `~/.claude/skills/examen` (or `--dest`).
Asks before overwriting an existing install.

```
examen generate
```
Stub. Quiz generation lives in the Claude Code skill, not the CLI. Prints a
pointer and exits with code 2.

```
examen --version
examen --help
```

---

## File locations

| Path | What |
|---|---|
| `~/.examen/history.db` | Longitudinal SQLite store: every session, answer, topic snapshot. ~5 KB per session. |
| `<repo>/.examen/quizzes/quiz-<uuid>.json` | Quiz artifacts the skill writes. Gitignored. |
| `<repo>/.code-review-graph/` | `code-review-graph` index. Gitignored (it's regeneratable). |
| `~/.claude/skills/examen/` | Where `examen skill install` drops the skill bundle. |

Nothing leaves your machine. The runner only binds `127.0.0.1`.

---

## Known limitations (v0.2.0)

Be honest with yourself before relying on this:

- **No staleness detection.** The runner serves the artifact as-is and never
  checks whether your code has changed since the quiz was generated. (The
  v0.1.0 codegraph-backed drift check was removed in v0.2.0 — it flagged
  every blast-radius question as drifted; see
  `docs/codegraph-schema-notes.md`.) Regenerate the quiz after meaningful
  changes.
- **`predict_output` question type is not implemented.** The schema rejects it
  until a safe execution sandbox exists.
- **Single-user, single-machine.** No multi-user, no team mode, no
  shared-history. The history db is local.
- **No git rename detection.** If you rename `auth/` to `authentication/`,
  every topic in that area gets a new ID and trend continuity breaks.
- **Cheat-resistant only by social contract.** Copying a question into another
  LLM defeats the tool. It's a self-honesty instrument, not an adversarial
  assessment.
- **Session state is in-memory.** A runner restart mid-session loses the
  in-flight question (DB has the answers committed so far, but the adaptive
  state machine resets). Resume support is partial.

---

## Development

```bash
git clone <this-repo> examen && cd examen
python3.11 -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"

# Run the test suite (146 tests)
pytest

# Type checking (ty — checks src + tests)
ty check

# Lint
ruff check .

# Sync skill schema copy + example fixture (after editing source schema)
make sync-schema
make sync-example
```

Project layout:

```
src/examen/
├── adaptive/      # scoring + state machine
├── artifact/      # JSON schema, loader, typed artifact structures
├── store/         # SQLite store (migrations, repos, sessions, topics)
├── static/        # vendored htmx + CSS
├── templates/     # Jinja2 templates for the 4 screens
├── cli.py         # Click CLI
├── codeview.py    # code-context loading + syntax highlighting
├── sessions.py    # in-memory live-session registry
├── viewmodel.py   # results/heatmap view models
└── serve.py       # FastAPI app

skill/examen/      # Claude Code skill bundle (markdown prompts)
tests/             # unit, integration, smoke tests
docs/              # design spec, implementation plan, codegraph notes
```

Conventions: every Python file starts with a `# path/to/file.py` comment;
ty type checking; Google Python Style; no Claude attribution in commits.

---

## Troubleshooting

**"Module 'examen.cli' has no attribute 'main'" after `pip install -e .`**
Re-run `pip install -e ".[dev]"` after switching branches — entry points are
re-registered at install time.

**`examen --version` says command not found**
The console script is registered via `pyproject.toml`'s `[project.scripts]`.
Make sure the venv is activated, or call directly: `.venv/bin/examen --version`.

**`/examen:generate-*` doesn't appear in Claude Code**
Verify the skill is installed: `ls ~/.claude/skills/examen/SKILL.md`. If
missing, run `examen skill install`. Restart Claude Code to pick up new
skills.

**The skill complains that `code-review-graph` isn't built**
Run `code-review-graph build` from your repo root. Indexing 1000+ files
takes a couple of minutes the first time.

**Quiz artifact fails to load: "Artifact fails schema validation"**
The skill version that wrote the artifact may be out of sync with the runner
version. Regenerate the quiz with the current skill, or downgrade the runner
to match.

**"Port 8780 taken"**
Examen picks the next free port and prints the new URL on stderr. Or pass
`--port N` explicitly.

**The browser doesn't open automatically**
Use `--no-browser` and copy the URL yourself. Examen prints it before
launching uvicorn so you can see it regardless.

---

## Design docs

The full design is in `docs/superpowers/specs/2026-05-17-examen-design.md`
(~800 lines). The phased implementation plan is in
`docs/superpowers/plans/2026-05-17-examen-implementation.md` (~5000 lines).
Historical codegraph integration notes (integration removed in v0.2.0):
`docs/codegraph-schema-notes.md`.

---

## License

MIT. See `LICENSE`.
