Metadata-Version: 2.4
Name: examen
Version: 0.3.1
Summary: Self-administered comprehension quiz for AI-assisted code
Project-URL: Homepage, https://github.com/abhiksark/examen
Project-URL: Repository, https://github.com/abhiksark/examen
Project-URL: Issues, https://github.com/abhiksark/examen/issues
Project-URL: Changelog, https://github.com/abhiksark/examen/blob/main/CHANGELOG.md
Author-email: Abhik Sarkar <abhiksark@gmail.com>
License: MIT License
        
        Copyright (c) 2026 Abhik Sarkar
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
License-File: LICENSE
Keywords: code-review,codebase,comprehension,developer-tools,fastapi,local-first,quiz
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Web Environment
Classifier: Framework :: FastAPI
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Education :: Testing
Classifier: Topic :: Software Development :: Quality Assurance
Requires-Python: >=3.11
Requires-Dist: click>=8.1
Requires-Dist: fastapi>=0.115
Requires-Dist: jinja2>=3.1
Requires-Dist: jsonschema>=4.23
Requires-Dist: pydantic>=2.9
Requires-Dist: python-multipart>=0.0.9
Requires-Dist: rfc3339-validator>=0.1.4
Requires-Dist: uvicorn[standard]>=0.32
Provides-Extra: dev
Requires-Dist: httpx>=0.27; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.24; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff>=0.7; extra == 'dev'
Requires-Dist: ty>=0.0.42; extra == 'dev'
Description-Content-Type: text/markdown

# Examen

> It's like an exam. For your own code. You will get a D.

I built this because I kept failing my own code reviews. I'd merge something
Claude wrote, walk into standup the next day, get asked "why did you handle
the edge case that way?" and freeze. Examen runs a focused quiz on your own
repo. Each question shows you the code, the answer options, and a
1–5 confidence chip on one screen. You submit both together, so your gut
goes on the record at the same time as your answer.

The results screen plots a heatmap: confidence × correctness. Top-right is
good. Top-left is dangerous — confident and wrong.

```bash
pip install examen
examen serve quiz.json
```

Local-only. No telemetry. No SaaS. Quiz artifacts generated by the
[`examen` Claude skill](https://github.com/abhiksark/examen/blob/main/skill/examen/SKILL.md),
or hand-rolled (see the
included fixture).

[Try the hosted demo →](https://examen-demo.fly.dev)

---

## Table of contents

- [Try it in 60 seconds (no Claude Code needed)](#try-it-in-60-seconds-no-claude-code-needed)
- [Full setup with Claude Code](#full-setup-with-claude-code)
- [How it works](#how-it-works)
- [CLI reference](#cli-reference)
- [File locations](#file-locations)
- [Known limitations](#known-limitations)
- [Development](#development)
- [Troubleshooting](#troubleshooting)
- [License](#license)

---

## Try it in 60 seconds (no Claude Code needed)

The repo ships a hand-crafted test fixture. You can take a real quiz with a
five-question budget without involving Claude or `code-review-graph` at all —
good for seeing the UX before committing to the full setup.

```bash
git clone https://github.com/abhiksark/examen.git examen && cd examen
python3.11 -m venv .venv
source .venv/bin/activate
pip install -e .

examen serve tests/fixtures/example_quiz_small.json
```

Your browser opens at `http://127.0.0.1:8780/start`. Click *Start quiz* and work
through up to 5 questions (each one takes your answer and a 1–5 confidence
rating together). The adaptive loop can finish early once every topic has a
floor; a miss can replace a remaining baseline slot with a drill. You end at a
results screen with a topic heatmap and separate accuracy and
calibration-quality scores.

The fixture covers fictional auth/billing/api modules — the questions are
toys, but the flow is exactly what real generated quizzes look like.

---

## Full setup with Claude Code

The end-to-end use case is: *generate a quiz from the diff I just merged,
take it, see what I don't actually understand.*

### Prerequisites

- Python **3.11 or newer**
- [Claude Code](https://claude.com/claude-code) installed and working
- [`code-review-graph`](https://github.com/tirth8205/code-review-graph) for
  indexing the target repo's call graph

### One-time install

```bash
# 1. Install Examen from PyPI.
python3.11 -m venv .venv && source .venv/bin/activate
pip install examen

# 2. Drop the Claude Code skill into ~/.claude/skills/examen.
examen skill install

# 3. Install code-review-graph (see its repo for current install command).
pip install code-review-graph
```

### Per-repo setup

In the repo you want to be quizzed on:

```bash
cd /path/to/your-project
code-review-graph build         # indexes the repo; one-time per repo
```

This creates `.code-review-graph/` (gitignored by Examen's gitignore template,
but if you're adding to an existing project add it yourself).

### Generate and take a quiz

Inside that project, open Claude Code and type one of:

```text
/examen:generate-diff                            # quiz on git diff vs main
/examen:generate-path src/auth                   # quiz on a directory
/examen:generate-repo                            # quiz on the whole repo
```

The skill writes a quiz artifact to `.examen/quizzes/quiz-<uuid>.json` in
the repo root and prints:

```text
Ready. Run: examen serve .examen/quizzes/quiz-<uuid>.json
```

Run that command (from any shell, doesn't need to be Claude Code), and the
browser opens at `http://127.0.0.1:8780/start`.

---

## How it works

Two halves, one JSON contract between them.

```text
┌─────────────────────────────────────────────────────────────────┐
│  Generator  (Claude Code skill: examen)                         │
│  - Reads code through code-review-graph MCP tools                │
│  - Claude authors questions grounded in graph facts              │
│  - Emits quiz-<uuid>.json (the artifact)                         │
└──────────────────────────┬──────────────────────────────────────┘
                           │ JSON file on disk
                           ▼
┌─────────────────────────────────────────────────────────────────┐
│  Runner     (Python, FastAPI + htmx + SQLite)                   │
│  - `examen serve <quiz.json>` opens browser at localhost         │
│  - Confidence committed alongside the answer (server-locked)     │
│  - Raw correctness drives accuracy, grades, trends, and ranking  │
│  - Brier-style calibration quality is tracked separately         │
│  - Two-channel heatmap: confidence × correctness                │
│  - Longitudinal history in ~/.examen/history.db                  │
│                                                                  │
│  HARD RULE: runner never calls an LLM, never makes network       │
│  requests beyond serving on 127.0.0.1.                           │
└─────────────────────────────────────────────────────────────────┘
```

The **artifact format is the contract.** Anyone could write a different
generator (different model, manual authoring, custom static analysis) and the
runner doesn't change.

### Why confidence-alongside-answer matters

A multiple-choice question with 4 options gives you a 25% chance of guessing
right. Without measuring confidence, "I got it right" doesn't distinguish
between *knowing* and *guessing*. With a confidence chip submitted alongside
each answer:

| Confidence | Correctness | Calibration quality | Interpretation |
|---|---|---|---|
| 5 (sure) | 1.0 | **1.00** | You knew it. |
| 5 (sure) | 0.0 | **0.00** | You were confidently wrong — the dangerous case. |
| 1 (no idea) | 0.0 | **1.00** | You said you didn't know, and you didn't. Calibrated. |
| 1 (no idea) | 1.0 | **0.00** | You got it right despite claiming no confidence. |
| 2 (guessing) | 1.0 | 0.44 | Got lucky, with some confidence. |
| 3 (medium) | 1.0 | 0.75 | Right with reasonable uncertainty. |

The "confidently wrong" cell is what the tool is hunting. Accuracy is the
weighted mean of raw correctness and determines grades and weak-topic ranking.
The Brier-style value measures **calibration quality**, while signed confidence
bias shows overconfidence or underconfidence. Keeping the three separate means
calibrated honesty can never be mislabeled as "% correct."

---

## CLI reference

```text
examen serve <artifact> [--port N] [--host ADDR] [--history-db PATH] [--no-browser]
```
Open the quiz UI in a browser for a given artifact. Default port `8780`;
picks the next free port if taken. `--no-browser` is useful for SSH/headless.
`--host` defaults to `127.0.0.1` (local-only); pass `0.0.0.0` to expose on all
interfaces, e.g. inside a container.

```text
examen validate <artifact>
```
Validate an artifact against the schema without serving it — schema
violations, unsupported `schema_version`, and topic cross-reference errors.
Exits non-zero on any problem. Useful for catching bad generator output early.

```text
examen history [--repo PATH]
```
List past sessions across all repos (5 most recent per repo), or filtered to
one repo's 20 most recent sessions with `--repo /path/to/repo`.

```text
examen trend --repo PATH [--topic LABEL]
```
ASCII bar chart of overall accuracy over time for one repo; pass `--topic` to
plot just that topic's trend ("am I getting better at auth?").

```text
examen revise --repo PATH [--limit N] [--questions N]
```
Surface a repo's weakest topics (lowest accuracy first, with confidence-bias flags)
and print a ready-to-paste Claude Code command that regenerates a quiz focused
on those topics' files. Closes the loop: quiz → see weak spots → re-quiz them.
The runner never calls an LLM; generation stays in the skill.

```text
examen prune --older-than 30d|6w|6mo|1y [--yes]
```
Delete completed and abandoned sessions older than the given duration.
Cascades through answers and topic snapshots, then removes orphaned topic and
repository rows. Confirms unless `--yes`.

```text
examen skill install [--dest PATH]
```
Copy the Claude Code skill bundle to `~/.claude/skills/examen` (or `--dest`).
Asks before overwriting an existing install.

```text
examen generate
```
Stub. Quiz generation lives in the Claude Code skill, not the CLI. Prints a
pointer and exits with code 2.

```text
examen --version
examen --help
```

---

## File locations

| Path | What |
|---|---|
| `~/.examen/history.db` | Longitudinal SQLite store: sessions, immutable quiz snapshots, selected answers, and topic aggregates. |
| `<repo>/.examen/quizzes/quiz-<uuid>.json` | Quiz artifacts the skill writes. Gitignored. |
| `<repo>/.code-review-graph/` | `code-review-graph` index. Gitignored (it's regeneratable). |
| `~/.claude/skills/examen/` | Where `examen skill install` drops the skill bundle. |

Nothing leaves your machine. The runner only binds `127.0.0.1`.

---

## Known limitations

Be honest with yourself before relying on this:

- **No staleness detection.** The runner serves the artifact as-is and never
  checks whether your code has changed since the quiz was generated. (The
  v0.1.0 codegraph-backed drift check was removed in v0.2.0 — it flagged
  every blast-radius question as drifted; see
  `docs/codegraph-schema-notes.md`.) Regenerate the quiz after meaningful
  changes.
- **`predict_output` question type is not implemented.** The schema rejects it
  until a safe execution sandbox exists.
- **Single-user, single-machine.** No multi-user, no team mode, no
  shared-history. The history db is local.
- **No git rename detection.** If you rename `auth/` to `authentication/`,
  every topic in that area gets a new ID and trend continuity breaks.
- **Cheat-resistant only by social contract.** Copying a question into another
  LLM defeats the tool. It's a self-honesty instrument, not an adversarial
  assessment.
- **Legacy unfinished sessions may not resume.** Current sessions persist an
  immutable artifact, ordered answers, and the displayed question so the
  adaptive state can be rebuilt after restart. An older session without that
  snapshot—or a corrupted history row—fails closed and asks you to start over.

---

## Development

```bash
git clone https://github.com/abhiksark/examen.git examen && cd examen
python3.11 -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"

# Sync the artifact mirrors, type-check, lint, and run the full test suite
make check

# Check the final diff for whitespace errors
git diff --check
```

Project layout:

```text
src/examen/
├── adaptive/      # scoring + state machine
├── artifact/      # JSON schema, loader, typed artifact structures
├── store/         # SQLite store (migrations, repos, sessions, topics)
├── static/        # vendored htmx + CSS
├── templates/     # Jinja2 templates and reusable page partials
├── cli.py         # Click CLI
├── codeview.py    # code-context loading + syntax highlighting
├── sessions.py    # in-memory live-session registry
├── viewmodel.py   # results/heatmap view models
└── serve.py       # FastAPI routes and application composition

skill/examen/      # Claude Code skill bundle (markdown prompts)
tests/             # unit and smoke tests
docs/              # design spec, implementation plan, codegraph notes
```

Conventions: every Python file starts with a `# path/to/file.py` comment;
ty type checking; Google Python Style; no Claude attribution in commits.

---

## Troubleshooting

**"Module 'examen.cli' has no attribute 'main'" after `pip install -e .`**
Re-run `pip install -e ".[dev]"` after switching branches — entry points are
re-registered at install time.

**`examen --version` says command not found**
The console script is registered via `pyproject.toml`'s `[project.scripts]`.
Make sure the venv is activated, or call directly: `.venv/bin/examen --version`.

**`/examen:generate-*` doesn't appear in Claude Code**
Verify the skill is installed: `ls ~/.claude/skills/examen/SKILL.md`. If
missing, run `examen skill install`. Restart Claude Code to pick up new
skills.

**The skill complains that `code-review-graph` isn't built**
Run `code-review-graph build` from your repo root. Indexing 1000+ files
takes a couple of minutes the first time.

**Quiz artifact fails to load: "Artifact fails schema validation"**
The skill version that wrote the artifact may be out of sync with the runner
version. Regenerate the quiz with the current skill, or downgrade the runner
to match.

**"Port 8780 taken"**
Examen picks the next free port and prints the new URL on stderr. Or pass
`--port N` explicitly.

**The browser doesn't open automatically**
Use `--no-browser` and copy the URL yourself. Examen prints it before
launching uvicorn so you can see it regardless.

---

## License

MIT. See `LICENSE`.
