Metadata-Version: 2.4
Name: tokenroi-metric
Version: 0.1.1
Summary: Difficulty-normalized value-per-token efficiency metric for AI-assisted coding
Author: Maya Lekhi
License: MIT License
        
        Copyright (c) 2026 Maya Lekhi
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        
Project-URL: Repository, https://github.com/mlekhi/tokens
Project-URL: Issues, https://github.com/mlekhi/tokens/issues
Keywords: software metrics,ai-assisted coding,efficiency,dea,iso 25010
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Software Development :: Quality Assurance
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy
Requires-Dist: scipy
Requires-Dist: scikit-learn<1.6,>=1.3
Requires-Dist: pandas
Requires-Dist: factor_analyzer
Requires-Dist: statsmodels
Requires-Dist: rich>=13
Provides-Extra: analysis
Requires-Dist: matplotlib; extra == "analysis"
Requires-Dist: semopy; extra == "analysis"
Provides-Extra: signals
Requires-Dist: radon; extra == "signals"
Requires-Dist: ruff; extra == "signals"
Requires-Dist: mypy; extra == "signals"
Requires-Dist: bandit; extra == "signals"
Provides-Extra: test
Requires-Dist: pytest; extra == "test"
Provides-Extra: bench
Requires-Dist: pytest; extra == "bench"
Requires-Dist: radon; extra == "bench"
Requires-Dist: ruff; extra == "bench"
Requires-Dist: mypy; extra == "bench"
Requires-Dist: bandit; extra == "bench"
Dynamic: license-file

# TokenROI
<img width="1072" height="499" alt="TokenROI" src="https://github.com/user-attachments/assets/373b9a32-0646-4ab1-8afd-033f4e430e6f" />

![tests](https://github.com/mlekhi/tokens/actions/workflows/ci.yml/badge.svg) ![license](https://img.shields.io/badge/license-MIT-blue)

A difficulty-normalized **value-per-token efficiency metric** for AI-assisted coding sessions.

Industry went from "tokenmaxxing" (reward token volume) to "tokenminning" (cap spend) without a
principled way to measure value-per-token. TokenROI scores how much measurable code value a session
produced per dollar of tokens, normalized for task difficulty, in a way that resists gaming by
spending more.

## Install

```bash
# recommended (isolated CLI install; modern macOS/Homebrew Python blocks bare pip — PEP 668):
pipx install "tokenroi-metric[bench]"     # brew install pipx first if needed
# or: uv tool install "tokenroi-metric[bench]"

# or in a venv (quotes matter — zsh eats bare [brackets]):
python3 -m venv .venv && . .venv/bin/activate && pip install "tokenroi-metric[bench]"

# or from source:
python3 -m venv .venv && . .venv/bin/activate && pip install -e .
```

The package imports as `tokenroi` and the CLI is `tokenroi`; `[bench]` adds what benchmark
participants need (pytest + the static-signal tools).

## Use the harness

```bash
# your local dashboard (also what bare `tokenroi` shows): recent sessions, deduped cost,
# cache share, per-project totals — the COST view (value scores come from bench mode)
tokenroi sessions
tokenroi sessions --by-project --since 7d

# score your own session logs (csv with token counts + the iso signal columns)
tokenroi score --input sessions.csv --output scored.csv

# or generate synthetic sessions with known ground truth and score them
tokenroi demo --tasks 40

# or ingest real Claude Code session logs -> tokens + cache-aware cost
tokenroi ingest --logs "~/.claude/projects/*/*.jsonl"

# or ingest ANY tool's usage export (Claude Code, Codex, Cursor, ...) on one scale
# csv columns: session_id, model, in_tokens, out_tokens
tokenroi ingest-usage --input usage.csv

# developer breakdown: WHY a session got its score + how to improve (not semantic — vs band peers)
tokenroi explain --input sessions.csv --session <id>
```

## Benchmark your friends (bench mode — real data, real value)

Same-task cohort benchmarking: N people solve one task pack with their coding agent, each
scores their own run locally (no transcript leaves the machine), the organizer ranks rows.
Same task = same difficulty band; the pack's gold tests = real functional value; a fresh
work dir = diff-scoped maintainability signals. This is TokenROI end-to-end on real data.

```bash
# each participant:
pipx install "tokenroi-metric[bench]"   # see Install above if you don't have pipx
tokenroi bench init logparse ~/bench-logparse
cd ~/bench-logparse            # do the task with your agent, any way you like
tokenroi bench submit --name alice   # -> alice.row.csv (cost + value signals only)

# organizer:
tokenroi bench score rows/*.csv      # -> leaderboard + honesty notes
```

Visible tests are in the work dir; hidden edge-case tests run at submit time from the
installed package (honor system in v1). Cost is deduped and cache-aware, labeled
API-equivalent — not what anyone paid.

## Reproduce the paper's numbers

```bash
pip install ".[test,analysis,signals]" tabulate
python -m pytest -q                  # tests: axioms, dea worked example, e2e ground-truth recovery
python analysis/reproduce_all.py     # regenerate EVERY result in one command -> analysis/results/

# or run individual analyses:
python analysis/run_analysis.py     # full study -> analysis/results/
python analysis/robustness.py       # multi-seed robustness (25 seeds) -> mean +/- CI
python analysis/weighted_baseline.py # the foil: when is the weighted ratio arbitrary?
python analysis/padding_attack.py   # gaming resistance: value-padding attack
pip install ".[analysis]"            # matplotlib + semopy for figures and CFA
python analysis/cfa.py              # confirmatory factor analysis of the ISO structure
```

## Paper

The IEEE-format manuscript (IEEEtran) is in [`paper/tokenroi.tex`](paper/tokenroi.tex) with
[`paper/references.bib`](paper/references.bib) and figures in `paper/figures/` — drop into Overleaf
to build. A markdown draft lives in [`docs/paper-draft.md`](docs/paper-draft.md).

## How the metric works

```
raw signals -> orient + normalize per band -> value vector
cost = tokens priced to dollars (the single input; process waste lands here automatically)

lean (headline):   R_s = (mean(value)/cost) / max_band(mean(value)/cost)      in (0,1]
work (wild logs):  value *= W_eff = lines x entropy x survival   (scope varies -> extensive value;
                   padding is entropy-crushed, deleted junk is survival-crushed)
dea (extension):   factor-analyze ~15 signals -> ~3 factors (+ W_eff) -> CCR + cross-efficiency, per band
```

- **Value** is the objective, tool-measurable ISO/IEC 25010 battery — no LLM/semantic judging.
- **Difficulty** is handled by competing only within difficulty bands.
- **Weights** in the DEA extension are derived by a linear program, not hand-picked.

## Results (synthetic study, 160 sessions/dataset, replicated across 25 seeds)

| claim | result (mean, 95% range across seeds) |
|---|---|
| recovers ground-truth ROI (H4) | Spearman 0.985 [0.973, 0.992] within band |
| resists gaming (H3) | frugal > wasteful in 100% of tasks, every seed |
| separates equal-quality sessions by cost (H2) | significant (p<0.05) in 100% of seeds |
| difficulty normalization works (H5) | \|corr w/ difficulty\| 0.08 stratified (vs ~0.37 unstratified) |
| lean ≈ DEA (H6) | Kendall τ 0.925 [0.888, 0.960] (lean often suffices) |
| factors recover ISO groups (EFA) | F1 maintainability, F2 functional/reliability, F3 security; CFA fit moderate (CFI 0.83) |
| value-padding resistance | rank gain +0.001 vs +0.051 for a coverage-trusting metric |

## Layout

```
src/tokenroi/      metric: cost, signals, lean (+ economic_roi), dea, reduction, bootstrap, stats,
                   synth, cli; adapters (real Claude Code logs), code_signals (static ISO signals)
tests/             axiom + e2e + coverage tests (90% coverage)
analysis/          run_analysis, robustness (25-seed), weighted_baseline, padding_attack, cfa
                   -> results/ (tables, figures, hypotheses.json)
paper/             IEEEtran manuscript + references.bib + figures
docs/              design spec + paper draft
LICENSE            MIT
```

## Limitations

DEA scores are relative to the observed cohort (not absolute); the value model is blind to semantic
quality by design; validation here is on synthetic data with known ground truth — the harness ingests
real agent logs through the same interface for the live study. See `docs/` for the full list.
