Metadata-Version: 2.5
Name: f1-temporal-bench
Version: 0.1.0
Summary: A benchmark for evaluating temporal knowledge and hallucination in language models, using Formula 1 as a fast-changing ground truth domain.
Project-URL: Homepage, https://github.com/YOUR_USERNAME/f1-temporal-bench
Project-URL: Issues, https://github.com/YOUR_USERNAME/f1-temporal-bench/issues
Author: Your Name
License: MIT
License-File: LICENSE
Keywords: benchmark,formula1,huggingface,llm-evaluation,nlp
Requires-Python: >=3.10
Requires-Dist: click>=8.1
Requires-Dist: huggingface-hub>=0.24
Requires-Dist: requests>=2.31
Requires-Dist: rich>=13.7
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff>=0.5; extra == 'dev'
Provides-Extra: local-models
Requires-Dist: torch>=2.2; extra == 'local-models'
Requires-Dist: transformers>=4.40; extra == 'local-models'
Description-Content-Type: text/markdown

# F1 Temporal Knowledge Benchmark

A benchmark for measuring how well language models track fast-changing,
verifiable facts — using Formula 1 standings, results, and records as
ground truth. F1 facts change every ~2 weeks during the season, making
this domain a natural, continuously-refreshed test of temporal knowledge
and hallucination in LLMs.

## Why F1?

Most knowledge benchmarks go stale the moment a model is trained. F1
gives us a domain where "true as of last week" and "true as of last year"
are meaningfully different — a clean way to measure whether a model
knows what it knows, or confidently guesses.

## Install

\`\`\`bash
pip install f1-temporal-bench
\`\`\`

## Usage

\`\`\`bash
f1-temporal-bench validate
f1-temporal-bench run --model meta-llama/Llama-3.2-1B-Instruct
\`\`\`

## Metrics

- **Accuracy** — exact/alias match against ground truth
- **Confidently wrong rate** — model gives a specific, wrong answer
  instead of hedging
- **Refusal rate** — model declines to answer / says it doesn't know

## Contributing questions

Add new rows to \`data/questions.jsonl\` after each race weekend. Run
\`f1-temporal-bench validate\` before committing.

## Roadmap

- [ ] Automated dataset updates via GitHub Actions after each race
- [ ] Multi-model leaderboard published to GitHub Pages
- [ ] Dataset published on the Hugging Face Hub with versioned season snapshots