Metadata-Version: 2.4
Name: edgefit
Version: 0.1.1
Summary: Neutral, reproducible measurement of on-device AI inference, across silicon vendors
Project-URL: Homepage, https://ahmtox.github.io/edgefit
Project-URL: Atlas, https://ahmtox.github.io/edgefit
Project-URL: Source, https://github.com/ahmtox/edgefit
Project-URL: Issues, https://github.com/ahmtox/edgefit/issues
Author: Alan Morelos
License-Expression: Apache-2.0
License-File: LICENSE
License-File: NOTICE
Keywords: benchmark,coreml,edge-ai,inference,latency,npu,on-device,onnxruntime,qualcomm,quantization
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: MacOS
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: System :: Benchmark
Requires-Python: <3.14,>=3.11
Requires-Dist: duckdb>=1.0
Requires-Dist: numpy<3,>=1.26
Requires-Dist: onnx>=1.16
Requires-Dist: onnxconverter-common>=1.14
Requires-Dist: onnxruntime>=1.19
Requires-Dist: psutil>=5.9
Requires-Dist: pyarrow>=16
Requires-Dist: pydantic>=2.7
Requires-Dist: pyobjc-framework-cocoa>=10; sys_platform == 'darwin'
Requires-Dist: pytz>=2024.1
Requires-Dist: pyyaml>=6
Requires-Dist: rich>=13
Requires-Dist: typer>=0.12
Provides-Extra: export
Requires-Dist: torch>=2.4; extra == 'export'
Requires-Dist: transformers>=4.44; extra == 'export'
Provides-Extra: qai-hub
Requires-Dist: qai-hub>=0.54; extra == 'qai-hub'
Description-Content-Type: text/markdown

# EdgeFit

Neutral, reproducible measurement of on-device AI inference — across silicon vendors,
including the failures.

Longer term this is a deployment compiler: give it a model, target devices and
constraints; it searches deployment recipes, measures the promising ones on real
hardware, and returns the best one plus the artifact plus a proof it meets budget. That
comes later and deliberately so. **Measurement first, because everything downstream is
worthless if the numbers are wrong.**

What is built, what is not, and what we know to be shaky is in
[the finding](docs/silent-fallback.md) and in *What is not true yet* below.
Product strategy and roadmap are not published.

---

## The finding

**[Your accelerator probably isn't running your model](docs/silent-fallback.md)**

Five models — three text, two vision — each exported once to fp32 ONNX and profiled on
up to eleven mobile SoCs from Qualcomm, Google and Samsung.

**Every device is either fully accelerated on every model, or runs every node on the
CPU.** Not one mixed case in 45 measurements. Three SoCs accelerate; eight do not — no
error, no warning, correct results throughout. Fastest 6.26 ms, slowest 820.37 ms:
**131× on the same file.**

Two 2024 flagships, same artifacts: Galaxy S24 vs Pixel 9 runs **15× to 40× apart**
depending on the model.

And the mirror image on Apple: ONNX Runtime's CoreML provider makes **four of six
models slower** than plain CPU, also silently.

## What works today

```bash
uv sync --extra export --group dev

uv run edgefit doctor                  # is this host fit to measure on?
uv run edgefit probe --model hf:...    # how would this model be measured?
uv run edgefit measure --model hf:sentence-transformers/all-MiniLM-L6-v2 \
                       --recipe recipes/ort_coreml_fp32.yaml
uv run edgefit sweep                   # models × recipes, locally
uv run edgefit sweep-remote            # models × hosted phones
uv run edgefit atlas build             # the corpus as a static site
uv run edgefit corpus export           # Parquet + CSV
uv run edgefit verify                  # golden fixtures — the gate for everything after
```

Corpus today: **110 measurements** over 7 models and 16 devices — 12 SoCs from Apple,
Qualcomm, Google and Samsung — of which **22 rows are recorded failures**.

| | |
|---|---|
| **Backends** | ONNX Runtime — CPU and CoreML providers, locally; Qualcomm AI Hub for hosted phones |
| **Models** | Any HuggingFace repo id. Specs are inferred from `config.json`; the registry holds overrides for what inference cannot get right |
| **Recipes** | fp32 · fp16 · int8 dynamic (per-tensor and per-channel) · static vs dynamic shapes · provider and vendor flags |
| **Workloads** | Encoders, classifiers, vision, and decoder-only generation with KV-cache I/O (TTFT and decode reported separately, never averaged) |
| **Analysis** | Per-node accelerator placement three ways · static FLOP estimation · graph fingerprint · duplicate-weight detection |
| **Output** | Insert-only DuckDB corpus, Parquet/CSV export, and a static atlas with a reproduction command on every row |

## Why the harness refuses to run

Measurement trust is the whole asset. One hallucinated number compromises every model
trained on the corpus, so this is built to fail loudly rather than produce a plausible
wrong answer:

- **A preflight gate** checks AC power, low-power mode, thermal state and free memory,
  and refuses if any fails. On a laptop with a browser open it refuses — correctly.
- **A measured throttle probe** times a fixed kernel against the host's own recorded
  healthy throughput, because Apple Silicon exposes no unprivileged temperature and
  inventing one is worse than admitting it.
- **Variance is mandatory and structural.** `RunStats` is constructible only from raw
  samples and revalidates its own aggregates, so a fabricated standard deviation cannot
  be represented.
- **The corpus is insert-only.** No update, no delete, anywhere. A re-measurement is a
  new row carrying a new harness version.
- **Unavailable values are null plus a written reason**, never a placeholder.
- **Both cascade tiers run out of process**, so a delegate that aborts the interpreter
  becomes a recorded failure instead of a dead sweep.
- **Third-party rows never impersonate ours.** Hosted measurements are marked
  throughout, with their thermal state recorded as unknown rather than assumed clean.
- **A model we cannot place is refused, not approximated.** The wrong input harness does
  not error — it returns a plausible number for a workload nobody asked about.

## What is not true yet

Stated here rather than buried, because the gaps are the reason to trust the rest:

- **No two-unit test.** Four physical devices of one SoC agree to 0.87%, which is the
  closest substitute, but they are different products — a disagreement could have been
  real rather than methodological.
- **Apple numbers are dev-grade.** One laptop-class machine, no second unit.
- **No quantized path on hosted devices.** AI Hub compile jobs are rejected server-side,
  so hosted rows are fp32 only — and "does int8 recover those eight devices?" is
  therefore the one question our headline finding raises that we cannot answer.
- **No power instrumentation, no thermal soak, no accuracy tier.** All null with
  recorded reasons rather than estimated.

## A note on the `§` references

The source cites `PROJECT.md §N` in about 85 places. That design document is not
published — it is product strategy — so those are pointers you cannot follow, and the
honest thing is to say so rather than let you hunt for a missing file. Where a reference
is load-bearing for understanding *why the code does something*, the reasoning is
restated inline next to it.

## Layout

```
src/edgefit/
  schema/     recipe, measurement, fingerprint, host records
  corpus/     insert-only DuckDB store + Parquet export
  harness/    host probes, preflight gate, run protocol, hosted measurement
  backends/   ONNX Runtime, export, quantization, graph/FLOP/placement analysis
  models/     spec inference + registry overrides
  atlas/      static site generator
  devices/    device inventory and fleet resolution
  cli/        typer entry point
tests/golden/ known-answer fixtures (marked `device`)
```

```bash
uv run pytest        # fast suite, no hardware
uv run ruff check .
```

## Hard rules

Enforced mechanically where possible, not by discipline:

1. Never estimate, extrapolate or synthesize a measurement value. A failed run is
   recorded as a failure; an unavailable field is null plus a reason.
2. Every measurement needs n≥5 runs and reported variance, or the record is invalid.
3. Measurements are immutable. Never `UPDATE`.
4. Measure end-to-end, including framework overhead and lowering time.
5. Open-source the harness. Every published number independently reproducible.
