Metadata-Version: 2.4
Name: hmft
Version: 0.2.1
Summary: Estimate how long an LLM call will take before you send it, plus a live ETA while it streams. Fully client-side.
License-Expression: Apache-2.0
Project-URL: Homepage, https://hmft.dev
Project-URL: Repository, https://github.com/hmft-dev/hmft
Project-URL: Changelog, https://github.com/hmft-dev/hmft/blob/main/CHANGELOG.md
Project-URL: Issues, https://github.com/hmft-dev/hmft/issues
Keywords: llm,latency,estimate,eta,streaming,tokens,tokenizer,openai,anthropic,benchmark
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: tiktoken>=0.7
Requires-Dist: httpx>=0.25
Provides-Extra: openai
Requires-Dist: openai>=1.30; extra == "openai"
Provides-Extra: anthropic
Requires-Dist: anthropic>=0.30; extra == "anthropic"
Provides-Extra: dev
Requires-Dist: pytest>=8; extra == "dev"
Requires-Dist: openai>=1.30; extra == "dev"
Requires-Dist: anthropic>=0.30; extra == "dev"
Dynamic: license-file

# hmft

Estimate how long an LLM call will take **before you send it**, and get a live ETA while it
streams. Runs entirely on your machine: no prompt or response text ever leaves the process.

```bash
pip install hmft                      # from PyPI
pip install "hmft[openai,anthropic]"  # with the SDK wrappers
pip install -e ".[dev]"               # from a clone, with the test extras
```

## Quickstart

No API key, no network call: the estimate is computed locally from the prompt.

```python
import hmft

est = hmft.estimate(
    [{"role": "user", "content": "Explain how bicycles stay upright in two paragraphs."}],
    model="gpt-4.1-mini", provider="openai",
)
print(est)                                      # ~2.3s (p90 4.6s): ttft 0.69s, ~220 tokens [measured]
print(est.total_s.p50, est.total_s.p90)         # 2.27 4.58
print(est.output_tokens.p50, est.output_tokens.p90)  # 220 352
```

Every estimate is a **band, not a point**: `p50` is the typical case, `p90` is the slow-but-likely
case you should plan around.

## Accuracy on held-out prompts

186 rows from prompt sets A to D, four models, re-scored under the current code with
`scripts/compare_runs.py`. "Coverage" is the share of rows whose actual landed at or under
the estimated p90; the target is about 90%.

**What "held out" means here, precisely.** These 186 rows are held out from **v2** constant
fitting: `compare_runs.py` marks sets A to D held-out and fits only on the later sets (E, F).
They are *not* untouched by v1. Set A is the v1 calibration set, and four v1.1 constants were
fitted or corrected using rows that are in this table — `TOKENS_PER_LIST_ITEM` (set B
actuals), `TOKENS_PER_ONE_LINE_ITEM_P50/P90` (A, B and C), the spoken-duration cue
`490 + 59 × minutes` (C and D), and the `code_large` band (the `long_code` actuals in A, B
and C). Read the table below as a standing regression score, not as a clean out-of-sample
result. The cleanest out-of-sample evidence is set E, measured on different subjects, where
the duration cue covered 29 of 30 rows.

| model | held-out rows | token p90 coverage | time p90 coverage |
|---|---|---|---|
| gpt-4.1-mini | 46 | 100% | 98% |
| claude-haiku-4-5 | 47 | 96% | 100% |
| claude-sonnet-4-5 | 47 | 100% | 94% |
| gpt-5@low | 46 | 100% † | 100% † |
| **all non-reasoning** | **140** | **99%** | **97%** |

† **gpt-5@low passes neither figure, despite the numbers, because all three of its table
constants are wrong.** The row assumes 1,312 hidden reasoning tokens, where the split runs
measured medians of 320 (set E) and 96 (set F); a 9.5 s first token, where the runs measured
per-run medians of 4.1 to 7.2 s; and 123 tokens/s decode, where the runs measured 34.9 to
108.6 tokens/s (per-run medians 58 to 81). So hmft over-estimates time for gpt-5@low (set F
median actual/estimate 0.55) and under-estimates visible output on long-form prompts (set E
1.27 and 1.34). The 100% coverage above is those two errors cancelling, not a pass. The
rebuild ships in 0.3.0. See `RESULTS.md`.

All four time-coverage breaches across the 140 non-reasoning rows were caused by time to
first token, not by the length heuristics; the token estimate was inside p90 in every one.

## Supported models

| model | provider | status | basis |
|---|---|---|---|
| `gpt-4.1-mini` | openai | measured | 10 benchmark samples; 46 held-out rows |
| `claude-haiku-4-5` | anthropic | measured | 10 benchmark samples; 47 held-out rows |
| `claude-sonnet-4-5` | anthropic | measured | 40 benchmark samples; 47 held-out rows |
| `gpt-5@low` | openai | experimental | 10 benchmark samples; hidden tokens, first token and decode rate all mis-set (above): over-estimates time, under-estimates tokens |
| `gpt-5@medium` | openai | experimental | 10 benchmark samples; no held-out rows |
| `gpt-5-mini` | openai | experimental | 10 benchmark samples; no held-out rows |
| `gpt-4.1` | openai | experimental | 10 benchmark samples; no held-out rows |
| `openai/gpt-4.1-mini`, `openai/gpt-4.1`, `openai/gpt-5-mini`, `anthropic/claude-sonnet-4.5`, `anthropic/claude-haiku-4.5` | openrouter | experimental | 6 to 10 benchmark samples each; no held-out rows |
| anything else | any | not measured | `basis="unmeasured"`: token band only, no timing |

**measured** means the throughput row comes from a real benchmark run *and* the model has
held-out coverage above. **experimental** means the throughput row is real but the estimate
has not been scored against held-out prompts, or was scored and did not pass. **not
measured** means hmft refuses to invent a number: you get an output-token band and
`basis="unmeasured"`, never a fabricated time.

A bare `gpt-5` call with no effort set maps to the `gpt-5@medium` row, the provider default,
and `basis_key` says so. OpenRouter has no provider-wide fallback row, because it routes
across model families; an unbenchmarked slug is `unmeasured`.

## Privacy

**No prompt text and no response text is ever stored, logged, printed or transmitted.**
Nothing is uploaded anywhere, by any code path, in v1. There is no server.

After each wrapped call hmft appends one JSON line to `~/.hmft/telemetry.jsonl`, created
`0600`, so you can score your own accuracy offline. The row is an allow-list of integers,
floats and short enum words — exactly these 24 fields and nothing else:

`ts`, `hmft_version`, `model`, `provider`, `prompt_tokens`, `output_tokens`, `ttft_ms`,
`total_ms`, `est_output_p50`, `est_output_p90`, `est_ttft_p50_s`, `est_ttft_p90_s`,
`est_total_p50_s`, `est_total_p90_s`, `basis`, `reasoning_effort`, `effort_source`,
`reasoning_tokens`, `finish_reason`, `visible_chunks`, `first_second_visible_chunks`,
`last_visible_ms`, `first_event_ms`, `max_gap_ms`.

No prompt, no response, no request ids, no user ids. `reasoning_effort`, `effort_source`,
`basis` and `finish_reason` are validated against fixed sets, so free text is dropped rather
than written. The write is enforced by an assertion against the allow-list in
`hmft/telemetry.py`, and `tests/test_telemetry.py` checks it.

Opt out with one environment variable:

```bash
HMFT_TELEMETRY=off                     # write nothing
HMFT_TELEMETRY=/path/to/file.jsonl     # or move it somewhere else
```

## The name

HMFT is short for **"how much fucking time"** — the question you actually ask when a model has
been streaming for forty seconds and you have no idea whether it is nearly done or has barely
started. The package, the import and the docs are all just `hmft`; the long form lives here and
nowhere else.

## How estimates work

```
total_time ≈ TTFT(prompt_tokens, model) + output_tokens / tokens_per_second(model)
```

1. **Output length comes from cues in the prompt** (`hmft/length.py`), in priority order:
   an explicit count ("in one sentence", "about 500 words", "8 questions", "one per line"),
   a stated speaking duration ("a 10-minute keynote" → `490 + 59 × minutes` tokens, validated
   2 to 45 minutes), a forced format (JSON mode, forced tool call, `max_tokens`), and finally
   a task type guessed from keywords (code, list, JSON, yes/no, translate, summarise, essay,
   explain, chat), each with its own base band.
2. **TTFT and tokens/second are measured constants**, per model and provider, in
   `hmft/data/throughput.json`. They are only ever written by `scripts/benchmark.py` from real
   runs — never estimated, never hand-edited. Unmeasured rows stay `null` and the estimator
   reports `basis="unmeasured"` instead of guessing.
3. **The band comes from both.** The p50 time uses the throughput `p50`; the p90 time uses the
   throughput **`p10`**, because for a rate the slow side is the low percentile. The length
   heuristic contributes its own p50/p90, and the two compose into `total_s`.

Measured throughput, as of the `updated_at` in the table:

| model | TTFT p50 / p90 | tokens/s p50 / p10 | samples |
|---|---|---|---|
| gpt-4.1-mini | 0.69 s / 1.622 s | 139.2 / 119.1 | 10 |
| gpt-4.1 | 0.76 s / 2.229 s | 111.3 / 90.5 | 10 |
| claude-haiku-4-5 | 0.67 s / 0.686 s | 83.4 / 79.5 | 10 |
| claude-sonnet-4-5 | 0.86 s / 1.355 s | 36.8 / 34.6 | 40 |
| gpt-5@low | 9.5 s / 17.52 s | 123.0 / 69.4 | 10 |

On reasoning models the TTFT column absorbs hidden thinking time, which is why it is large and
why those rows are experimental.

## Streaming ETA

```python
from openai import OpenAI
import hmft

client = hmft.wrap_openai(OpenAI())
stream = client.chat.completions.create(
    model="gpt-4.1-mini", stream=True,
    messages=[{"role": "user", "content": "Explain how bicycles stay upright in two paragraphs."}],
)
print(stream.estimate)             # pre-call band
for chunk in stream:
    print(stream.eta(), end="\r")  # live: blends the observed rate in after a few tokens
print(stream.result)               # actual vs estimated
```

`hmft.wrap_anthropic(Anthropic())` is the same shape around `client.messages.stream(...)`.
Wrappers are sync-only in v1; async is a v1.x item.

## Filling the throughput table

```bash
export OPENAI_API_KEY=...       # and/or ANTHROPIC_API_KEY, OPENROUTER_API_KEY
python scripts/benchmark.py --all --dry-run    # print the plan, make no calls
python scripts/benchmark.py --provider anthropic --models claude-haiku-4-5 --runs 3
```

Each run sends two synthetic prompts and records only token counts and timings.
`name@effort` (for example `gpt-5@low`) sets `reasoning_effort` and gets its own row.

## Known limits

- Output-length heuristics are rules, not a model. Expect the p50 to be off by 2x on
  open-ended prompts; the p90 band is what to plan around.
- Reasoning models spend most of their wall time on hidden tokens the heuristics cannot see.
- Non-OpenAI token counts use the o200k tokenizer as a proxy, typically within 10 to 20%.
- Image and tool-result content blocks are not counted.
- Talk-duration cues are validated 2 to 45 minutes; above 45 is extrapolation. The p90 held on
  29 of 30 measured talks, but the p50 is biased per model: gpt-4.1-mini writes far less than
  the estimate on long talks, Claude models somewhat more.
- Provider latency drifts hour to hour; `measured_at` tells you how stale a row is.

## Tests

```bash
pip install -e ".[dev]"
python -m pytest
```

All tests use synthetic prompts and fake clients, and the suite makes no network calls —
`tests/test_no_network.py` enforces that. See `CONTRIBUTING.md` for the held-out rule and the
version-bump rule. `BASELINE.md` is the frozen v1 reference and `RESULTS.md` records every
measurement since.

## License

Apache-2.0. See `LICENSE`.
