Metadata-Version: 2.5
Name: kt-audit
Version: 0.2.0
Summary: Audit knowledge-tracing evaluation before you trust the AUC.
Project-URL: Homepage, https://github.com/summeriver13/kt-audit
Project-URL: Repository, https://github.com/summeriver13/kt-audit
Project-URL: Issues, https://github.com/summeriver13/kt-audit/issues
Author: Xia Jiang
License: MIT
License-File: LICENSE
Keywords: benchmark,education,evaluation,knowledge-tracing,leakage
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Requires-Dist: numpy>=1.26
Requires-Dist: pandas>=2.1
Requires-Dist: scikit-learn>=1.4
Provides-Extra: dev
Requires-Dist: basedpyright>=1.17; extra == 'dev'
Requires-Dist: pytest-cov>=5; extra == 'dev'
Requires-Dist: pytest>=8; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Description-Content-Type: text/markdown

# kt-audit

[![PyPI](https://img.shields.io/pypi/v/kt-audit)](https://pypi.org/project/kt-audit/)
[![CI](https://github.com/summeriver13/kt-audit/actions/workflows/ci.yml/badge.svg)](https://github.com/summeriver13/kt-audit/actions/workflows/ci.yml)
[![Python](https://img.shields.io/pypi/pyversions/kt-audit)](https://pypi.org/project/kt-audit/)

**Before you trust an AUC, audit the evaluation.**

`kt-audit` is a small, opinionated audit tool for knowledge-tracing (KT) experiments. It does not train models. It checks whether the evaluation setup behind a reported AUC is trustworthy.

Status: **alpha (0.1.x)**. The scope is intentionally narrow; see [docs/charter.md](https://github.com/summeriver13/kt-audit/blob/main/docs/charter.md) for the scope and non-goals.

## Why

KT papers and libraries report AUC, but the evaluation protocol is often under-specified:

- Are train events strictly before validation/test events for each user?
- Does the same item appear in train and test?
- When an item has multiple knowledge components (KCs), are some KCs train-only and others test-only?
- Do the reported metrics match a recomputation from the released predictions?

`kt-audit` turns those questions into a machine-readable report you can attach to a paper artifact, a PR, or a model release.

Two 2026 measurements motivated the v0.2 checks. In a tutoring benchmark, a depth-4 tree over seven
scalar features reproduced the gold diagnosis labels at 82.3 accuracy while the best LLM reached
42.0: the metric was measuring rule-following. In the same benchmark, a 3,000-token output budget
silently truncated 14.5% of samples and cost 7 points of score.

These are not hypothetical. In pyKT, a widely used KT benchmark library, the authors themselves warn that "the training and test data has overlap and the experimental results cannot be compared" ([pykt-team/pykt-toolkit#91](https://github.com/pykt-team/pykt-toolkit/issues/91)), and `data_process.py` points `valid` and `test` at the same file.

## Install

```bash
pip install kt-audit
```

Or with uv:

```bash
uv add kt-audit
```

From source:

```bash
git clone https://github.com/summeriver13/kt-audit
cd kt-audit
uv sync
uv run kt-audit --help
```

Requires Python 3.10+. Pin a specific version for reproducible paper artifacts.

## Quickstart

Download the synthetic example and run the audit:

```bash
curl -fsSLO https://raw.githubusercontent.com/summeriver13/kt-audit/main/examples/leaky.csv
kt-audit run leaky.csv --out report
```

Or, from a source checkout:

```bash
uv run kt-audit run examples/leaky.csv --out report
```

Output:

```text
verdict: fail
findings: 5 error, 0 warn
report: report/report.md
```

The command writes:

- `report.json` — stable, machine-readable.
- `report.md` — human-readable, with examples and limitations.

Exit codes: `0` pass/warn, `1` fail, `2` input/config error.

## What it checks

| Check | Catches | Default |
| --- | --- | --- |
| `temporal_split` | a user's train events after that user's evaluation events | error |
| `multi_kc` | same item with train-only and test-only KCs | error |
| `alignment` | missing/duplicate/out-of-range predictions; constant predictions; reported AUC mismatch | error / warn |
| `cheap_baseline` | reported predictions that a tree over past-event aggregates matches or beats | error / warn |
| `degenerate` | predictions too few, too narrow, on too small a sample, or with a rare minority class | warn / info |

Each finding has a `severity`, a `count`, and up to three examples.

### `cheap_baseline`

KT claims that long history adds information. This check tests that claim directly: it builds
eight features from **past events only** — prior count, prior accuracy, gap since the last event,
same-item history, item popularity, KC history, and sequence position — trains a depth-4 decision
tree on the training split, and compares its AUC with the reported predictions on the same rows.

- baseline AUC ≥ model AUC: `error`. The model adds nothing over cheap aggregates.
- model AUC − baseline AUC < `baseline_warn_margin` (0.02): `warn`.
- fewer than 50 labelled training rows, 30 evaluation rows, or one class: skipped.
- The comparison runs on the test split, or on valid when test is absent.

The numbers land in `report.json` under `baselines` and in a `## Baseline` table.

### `degenerate`

| Rule | Threshold | Why |
| --- | --- | --- |
| few distinct values | fewer than 10, at least 30 rows | predictions collapse to a few levels |
| narrow band | IQR below 0.02, at least 30 rows | no discrimination; AUC sits near 0.5 |
| small sample | fewer than 30 rows | interval too wide to compare (info, not a defect) |
| rare minority | minority share below 1% | AUC unstable |

Thresholds are module constants, not config keys. Change them with a release.

## Compare two runs

Two models that differ by 3 points on 200 rows may be indistinguishable. `compare` pairs the two
prediction files on their interaction keys and bootstraps the AUC delta on the same resampled rows:

```bash
kt-audit compare baseline.csv candidate.csv --out comparison
```

```text
verdict: b_better
delta: +0.4736 (95% CI [+0.3637, +0.5907])
report: comparison/comparison.md
```

- CI contains 0: `tie`. The sample cannot tell the two runs apart.
- Output: `comparison.json` and `comparison.md`.
- Defaults: 2000 rounds, seed 0, so the interval is reproducible.
- Inputs must share keys and agree on `y_true`; otherwise the command exits with code 2.

## Case study

`docs/case-study-assist09.md` runs the tool on ASSISTments 2009 with a per-user chronological split.
The v0.1 checks stay silent, which is correct; the cheap baseline reaches 0.7138, so a reported
0.771 passes and a reported 0.70 fails. Three severities changed as a result.

## Provenance

`report.json` records `tool_version` and `input.sha256`, so an artifact stays readable after the
input file is regenerated. The Markdown report prints both in its header.

## Input

One row per interaction or prediction. CSV or JSONL.

| Column | Required | Notes |
| --- | :--: | --- |
| `user_id` | yes | non-null |
| `timestamp` | yes | ISO8601 or numeric; comparable within a user |
| `split` | yes | `train` / `valid` / `test` |
| `kc_ids` | yes | pipe-separated KCs or JSON array |
| `item_id` | no | needed for multi-KC check |
| `y_true` | no | 0/1; required with `y_pred` |
| `y_pred` | no | [0,1]; omit to skip metric checks |
| `fold` | no | checks run per fold |
| `sequence_id`, `position` | no | reserved for sequence checks |

See [docs/data-contract.md](https://github.com/summeriver13/kt-audit/blob/main/docs/data-contract.md) for the full contract.

## Python API

```python
from kt_audit import AuditConfig, audit_path

report = audit_path("events.csv", AuditConfig(reported_auc=0.80))
print(report.verdict)
for finding in report.findings:
    print(finding.severity, finding.check, finding.count)
```

## Design constraints

- Runtime dependencies: `numpy`, `pandas`, `scikit-learn`.
- CLI uses `argparse`; config uses JSON.
- Each check is a pure function over a prepared `DataFrame`.
- Synthetic examples only; no dataset is bundled.
- Findings are heuristics, not causal proof.

## Development

```bash
uv sync --extra dev
uv run ruff check .
uv run ruff format --check .
uv run basedpyright
uv run pytest --cov=kt_audit --cov-report=term-missing
bash scripts/wheel-smoke.sh
```

## Release

See [docs/release.md](https://github.com/summeriver13/kt-audit/blob/main/docs/release.md) and [CHANGELOG.md](https://github.com/summeriver13/kt-audit/blob/main/CHANGELOG.md). TestPyPI and PyPI publish through GitHub Actions trusted publishing.

## Roadmap

- v0.2 (shipped): `cheap_baseline`, `degenerate`, `compare`, provenance fields. See `docs/rfc-0.2.md`.
- v0.3: pyKT adapter (covers the models wired in `init_model.py`, including the cold-start `cskt`), HTML report, offline `judge-audit` for saved LLM judgments, duplicate interactions, sequence truncation.

## AI disclosure

AI tools are used in development. The maintainer reviews every check definition, test, and release. Findings generated by the tool are deterministic and do not call an LLM.

## Contributing

See [CONTRIBUTING.md](https://github.com/summeriver13/kt-audit/blob/main/CONTRIBUTING.md). The v0.1 scope is frozen to the three checks above.

## License

MIT. See [LICENSE](https://github.com/summeriver13/kt-audit/blob/main/LICENSE).
