Metadata-Version: 2.5
Name: kt-audit
Version: 0.1.0
Summary: Audit knowledge-tracing evaluation before you trust the AUC.
Project-URL: Homepage, https://github.com/summeriver13/kt-audit
Project-URL: Repository, https://github.com/summeriver13/kt-audit
Project-URL: Issues, https://github.com/summeriver13/kt-audit/issues
Author: Xia Jiang
License: MIT
License-File: LICENSE
Keywords: benchmark,education,evaluation,knowledge-tracing,leakage
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Requires-Dist: numpy>=1.26
Requires-Dist: pandas>=2.1
Requires-Dist: scikit-learn>=1.4
Provides-Extra: dev
Requires-Dist: basedpyright>=1.17; extra == 'dev'
Requires-Dist: pytest-cov>=5; extra == 'dev'
Requires-Dist: pytest>=8; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Description-Content-Type: text/markdown

# kt-audit

**Before you trust an AUC, audit the evaluation.**

`kt-audit` is a small, opinionated audit tool for knowledge-tracing (KT) experiments. It does not train models. It checks whether the evaluation setup behind a reported AUC is trustworthy.

Status: **alpha (0.1.0)**. The scope is intentionally narrow; see [docs/charter.md](docs/charter.md) for the scope and non-goals.

## Why

KT papers and libraries report AUC, but the evaluation protocol is often under-specified:

- Are train events strictly before validation/test events for each user?
- Does the same item appear in train and test?
- When an item has multiple knowledge components (KCs), are some KCs train-only and others test-only?
- Do the reported metrics match a recomputation from the released predictions?

`kt-audit` turns those questions into a machine-readable report you can attach to a paper artifact, a PR, or a model release.

## Install

Not on PyPI yet. From source:

```bash
git clone https://github.com/summeriver13/kt-audit
cd kt-audit
uv sync
uv run kt-audit --help
```

Or with pip:

```bash
pip install -e .
```

Requires Python 3.10+.

## Quickstart

```bash
uv run kt-audit run examples/leaky.csv --out /tmp/kt-audit-report
```

Output:

```text
verdict: fail
findings: 6 error, 2 warn
report: /tmp/kt-audit-report/report.md
```

The command writes:

- `report.json` — stable, machine-readable.
- `report.md` — human-readable, with examples and limitations.

Exit codes: `0` pass/warn, `1` fail, `2` input/config error.

## What it checks

| Check | Catches | Default |
| --- | --- | --- |
| `temporal_split` | train timestamps after validation/test; users across splits | error / warn |
| `multi_kc` | same item with train-only and test-only KCs | error |
| `alignment` | missing/duplicate/out-of-range predictions; constant predictions; reported AUC mismatch | error / warn |

Each finding has a `severity`, a `count`, and up to three examples.

## Input

One row per interaction or prediction. CSV or JSONL.

| Column | Required | Notes |
| --- | :--: | --- |
| `user_id` | yes | non-null |
| `timestamp` | yes | ISO8601 or numeric; comparable within a user |
| `split` | yes | `train` / `valid` / `test` |
| `kc_ids` | yes | pipe-separated KCs or JSON array |
| `item_id` | no | needed for multi-KC check |
| `y_true` | no | 0/1; required with `y_pred` |
| `y_pred` | no | [0,1]; omit to skip metric checks |
| `fold` | no | checks run per fold |
| `sequence_id`, `position` | no | reserved for sequence checks |

See [docs/data-contract.md](docs/data-contract.md) for the full contract.

## Python API

```python
from kt_audit import AuditConfig, audit_path

report = audit_path("events.csv", AuditConfig(reported_auc=0.80))
print(report.verdict)
for finding in report.findings:
    print(finding.severity, finding.check, finding.count)
```

## Design constraints

- Runtime dependencies: `numpy`, `pandas`, `scikit-learn`.
- CLI uses `argparse`; config uses JSON.
- Each check is a pure function over a prepared `DataFrame`.
- Synthetic examples only; no dataset is bundled.
- Findings are heuristics, not causal proof.

## Development

```bash
uv sync --extra dev
uv run ruff check .
uv run ruff format --check .
uv run basedpyright
uv run pytest --cov=kt_audit --cov-report=term-missing
bash scripts/wheel-smoke.sh
```

## Release

See [docs/release.md](docs/release.md). TestPyPI and PyPI publish through GitHub Actions trusted publishing.

## Roadmap

- v0.2: pyKT adapter, bootstrap AUC intervals, HTML report.
- v0.3: duplicate interactions, cold-start, sequence truncation, label leakage.

## AI disclosure

AI tools are used in development. The maintainer reviews every check definition, test, and release. Findings generated by the tool are deterministic and do not call an LLM.

## Contributing

See [CONTRIBUTING.md](CONTRIBUTING.md). The v0.1 scope is frozen to the three checks above.

## License

MIT. See [LICENSE](LICENSE).
