Metadata-Version: 2.4
Name: eleanity
Version: 1.2.0
Summary: Same model. Same input. Find the first divergence.
Project-URL: Homepage, https://github.com/TheusHen/eleanity
Project-URL: Documentation, https://github.com/TheusHen/eleanity/tree/main/docs
Project-URL: Issues, https://github.com/TheusHen/eleanity/issues
Project-URL: Repository, https://github.com/TheusHen/eleanity
Author: Eleanity Contributors
License: Apache-2.0
License-File: LICENSE
License-File: NOTICE
Keywords: diagnostics,llama.cpp,llm,parity,transformers,vllm
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.11
Requires-Dist: httpx>=0.27
Requires-Dist: jinja2>=3.1
Requires-Dist: jsonschema>=4.22
Requires-Dist: packaging>=24
Requires-Dist: pydantic<3,>=2.7
Requires-Dist: pyyaml>=6.0
Requires-Dist: rich>=13
Requires-Dist: typer>=0.12
Provides-Extra: all
Requires-Dist: accelerate>=0.33; extra == 'all'
Requires-Dist: safetensors>=0.4; extra == 'all'
Requires-Dist: tokenizers>=0.20; extra == 'all'
Requires-Dist: torch>=2.11; extra == 'all'
Requires-Dist: transformers>=4.45; extra == 'all'
Provides-Extra: dev
Requires-Dist: mypy>=1.11; extra == 'dev'
Requires-Dist: pip-audit>=2.9; extra == 'dev'
Requires-Dist: pytest-cov>=5; extra == 'dev'
Requires-Dist: pytest>=8; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Requires-Dist: types-jsonschema>=4.23; extra == 'dev'
Requires-Dist: types-pyyaml>=6.0; extra == 'dev'
Provides-Extra: llamacpp
Provides-Extra: transformers
Requires-Dist: accelerate>=0.33; extra == 'transformers'
Requires-Dist: safetensors>=0.4; extra == 'transformers'
Requires-Dist: tokenizers>=0.20; extra == 'transformers'
Requires-Dist: torch>=2.11; extra == 'transformers'
Requires-Dist: transformers>=4.45; extra == 'transformers'
Provides-Extra: vllm
Description-Content-Type: text/markdown

# Eleanity

**Same model. Same input. Find the first causal divergence.**

[![CI](https://github.com/TheusHen/eleanity/actions/workflows/ci.yml/badge.svg)](https://github.com/TheusHen/eleanity/actions/workflows/ci.yml)
[![Local AI parity](https://github.com/TheusHen/eleanity/actions/workflows/parity-local-ai.yml/badge.svg)](https://github.com/TheusHen/eleanity/actions/workflows/parity-local-ai.yml)
[![Python 3.11+](https://img.shields.io/badge/python-3.11%2B-blue.svg)](https://www.python.org/downloads/)
[![License](https://img.shields.io/badge/license-Apache%202.0-green.svg)](LICENSE)
[![Status](https://img.shields.io/badge/status-stable-brightgreen.svg)](ROADMAP.md)

CLI-first tool for **LLM runtime parity**. It compares inference stacks on the same scenario and reports:

- **status** (`PASS`, `PASS_WITH_TOLERANCE`, `PASS_WITH_LIMITED_COVERAGE`, `INCONCLUSIVE`, `DIVERGENT`, `ERROR`)
- **first divergent layer** (when present)
- **coverage** of required layers and **diagnosis confidence**
- **functional impact** (separate from internal parity)
- a **reproduction command**

It does **not** score model quality (no MMLU). It checks whether two runtimes still share the same causal path: artifact → template → special tokens → token IDs → generation / API.

**Product surfaces:** CLI (`text` / `json` / `quiet` / `sarif`) and a **Python API** for embedding without subprocess — see [docs/api.md](docs/api.md).

## Try the core idea offline

```bash
uvx eleanity demo
```

```text
status:            DIVERGENT
first_divergence:  template
baseline_template: 'user: Hello\nassistant:'
candidate_template:'user: Hello'
probable_cause:    CHAT_TEMPLATE_DIFFERENT (confidence=0.92)
```

No model download, GPU, server, account, or network endpoint is required.

> **Hosting note:** the repository currently lives at
> [`TheusHen/eleanity`](https://github.com/TheusHen/eleanity).
> Transfer to `eleanity/eleanity` is planned when the org is available ([ROADMAP.md](ROADMAP.md)).

---

## Install (clean machine)

### From Git (available now)

```bash
# one-shot CLI without a permanent install
uvx --from git+https://github.com/TheusHen/eleanity.git eleanity --help

# or install into an environment
pip install "git+https://github.com/TheusHen/eleanity.git"
# optional HF backend (PEP 508 direct reference + extras)
pip install "eleanity[transformers] @ git+https://github.com/TheusHen/eleanity.git"
```

```bash
uv add git+https://github.com/TheusHen/eleanity.git
# or with transformers extra:
uv add "eleanity[transformers] @ git+https://github.com/TheusHen/eleanity.git"
```

### From a local checkout

```bash
git clone https://github.com/TheusHen/eleanity.git
cd eleanity
uv sync --group dev
uv run eleanity --help
```

### PyPI

```bash
pip install eleanity
# pin the stable API line:
pip install "eleanity==1.0.0"
```

When a reviewed version change reaches `main`, [`release.yml`](.github/workflows/release.yml)
runs the complete quality suite, publishes that exact version to PyPI, then
creates an immutable `vX.Y.Z` tag and GitHub Release with the wheel, sdist, and
SHA256 checksums. The workflow is idempotent and uses the built-in GitHub token;
it never writes version commits directly to the protected branch.

```bash
# Required repository secret (once):
gh secret set PYPI_API_TOKEN -R TheusHen/eleanity
```

The job verifies wheel/sdist metadata and writes **SHA256SUMS**.

Requires **Python 3.11+**.

---

## Offline parity check

```bash
uv run eleanity compare --model demo --backends fake,fake \
  --format quiet --no-parallel --no-gates
```

Exact output from a live run:

```text
status=PASS impact=NONE coverage=100.0 confidence=0.85 first_divergence=none gates=True run_id=16c3f05b-70f9-4c6a-9170-3e5942f91bd6
```

---

## Real model self-parity (PASS)

Model: `HuggingFaceTB/SmolLM2-135M-Instruct` (CPU).

### Commands

```bash
uv sync --group dev --extra transformers

uv run eleanity pull HuggingFaceTB/SmolLM2-135M-Instruct --tokenizer-only

uv run eleanity compare \
  --model HuggingFaceTB/SmolLM2-135M-Instruct \
  --backends transformers,transformers \
  --format quiet --no-parallel --no-gates \
  --observe artifact,template,special_tokens,tokens,generation
```

### Exact quiet output (live run)

```text
status=PASS impact=NONE coverage=100.0 confidence=0.85 first_divergence=none gates=True run_id=0aba046a-5846-45b2-94d9-4584bb4fe98a
```

| Metric | Value |
| --- | --- |
| Engine total | ~10276 ms |
| First observation | ~9259 ms |
| Second (warm) | ~1017 ms |
| Token match | 31 / 31 |
| Required coverage | 100% (min 75%) |

```bash
uv run eleanity report 0aba046a-5846-45b2-94d9-4584bb4fe98a --format text
```

Layer table excerpt:

```text
Layer            Baseline obs   Candidate obs   Compare
artifact         OBSERVED       OBSERVED        PASS
special_tokens   OBSERVED       OBSERVED        PASS
template         OBSERVED       OBSERVED        PASS
tokens           OBSERVED       OBSERVED        PASS
generation       OBSERVED       OBSERVED        PASS

Template diff: PASS
Token diff:    PASS (count: 31)
```

This validates the **engine + Transformers path**.

---

## Cross-runtime DIVERGENT (Transformers × real HF OpenAI-compat HTTP)

**Both sides run real Hugging Face weights** (SmolLM2-135M-Instruct):

| Side | Runtime path |
| --- | --- |
| Baseline | `transformers` adapter (in-process) |
| Candidate | `vllm` adapter → local OpenAI-compat server that **loads the same HF model** via Transformers |

The HTTP server does **not** expose template/tokenize endpoints (same honesty profile as many LM Studio / gateway setups). By default it applies the chat template **without** the assistant generation prompt — a common production bug — so generation diverges for a causal reason, not a hard-coded string.

### Commands (exact)

```bash
uv sync --group dev --extra transformers

# starts real HF server, runs doctor + compare, stops server
uv run python scripts/examples/run_cross_runtime_demo.py
```

Under the hood:

```bash
# terminal A
uv run python scripts/examples/hf_openai_server.py \
  --port 8765 --preload --omit-generation-prompt \
  --model HuggingFaceTB/SmolLM2-135M-Instruct

# terminal B
export ELEANITY_VLLM_URL=http://127.0.0.1:8765
uv run eleanity doctor --check-backends --backends vllm --format json
uv run eleanity compare \
  --model HuggingFaceTB/SmolLM2-135M-Instruct \
  --backends transformers,vllm \
  --backend-url vllm=http://127.0.0.1:8765 \
  --policy quantized \
  --format quiet --no-parallel --no-gates \
  --observe artifact,template,tokens,generation,api
```

Parity attempt (server uses full AGP): `ELEANITY_DEMO_MATCH=1 uv run python scripts/examples/run_cross_runtime_demo.py`

Point the same commands at LM Studio / vLLM serve by changing `--backend-url`. Offline fixed-string stub (CI without GPU weights): `scripts/examples/mock_openai_diverge.py`.

### Exact quiet output (live run)

```text
status=DIVERGENT impact=HIGH coverage=50.0 confidence=0.762 first_divergence=generation gates=False run_id=90028893-8848-463f-9331-daf5268f60b5
```

| Field | Value |
| --- | --- |
| Baseline | transformers · `HuggingFaceTB/SmolLM2-135M-Instruct` (HF weights, in-process) |
| Candidate | vllm adapter → `hf_openai_server.py` (same HF weights over HTTP, `--omit-generation-prompt`) |
| Policy | `quantized` |
| Status | **DIVERGENT** |
| First divergence | **generation** |
| Coverage | **50%** required layers (template/tokens not mutually observed on HTTP) |
| Verified | artifact, generation |
| Not verified | template (`NOT_SUPPORTED` on HTTP), tokens (`NOT_EXPOSED` on HTTP) |
| Generation texts | transformers: `Hello! How can I help you today?` · HTTP: `assistant\nHello! How can I help you today?` |
| Engine total | ~8287 ms |

Reproduction command stored on the run:

```text
eleanity compare --model HuggingFaceTB/SmolLM2-135M-Instruct --backends transformers,vllm \
  --baseline transformers --policy quantized \
  --observe artifact,template,tokens,generation,api --no-gates \
  --backend-url vllm=http://127.0.0.1:8765 --format text
```

This is the cross-stack claim: **two real runtimes, same model weights, localized first divergence, honest missing layers**.

---

## Template first-divergence (character-level)

Classic chat-template bug (missing assistant generation prompt), fully localized:

```bash
uv run python scripts/examples/demo_template_divergence.py
```

Exact output (live run):

```text
status:            DIVERGENT
first_divergence:  template
character:         11
byte:              11
baseline template: 'user: Hello\nassistant:'
candidate template:'user: Hello'
probable_cause:    [CHAT_TEMPLATE_DIFFERENT] conf=0.92
```

More write-ups: [docs/examples/first-divergence.md](docs/examples/first-divergence.md).

---

## Compared to common alternatives

| Approach | Result type | Causal first layer | CI exit contract | Missing-data honesty |
| --- | --- | --- | --- | --- |
| Manual print / eyeball | vibes | no | no | no |
| String golden tests | brittle text diff | rare | sometimes | often silent |
| lm-eval / quality benches | capability scores | no | yes | n/a |
| API latency dashboards | ops metrics | no | maybe | n/a |
| **Eleanity** | parity + location | **yes** | **0/1/2** | **yes (coverage)** |

---

## Adapter capability matrix (summary)

Full table: [docs/adapter-capabilities.md](docs/adapter-capabilities.md).

`vllm` and `llamacpp` are HTTP-client adapters. Run the corresponding server
runtime in its own restricted environment; Eleanity does not install either
server runtime as an extra.

| Adapter | template | tokens | logits | generation | streaming | API |
| --- | --- | --- | --- | --- | --- | --- |
| fake | yes | yes | no | yes | yes | partial |
| transformers | yes | yes | partial | yes | no | no |
| vllm HTTP | partial | partial | no | yes | partial | yes |
| llamacpp HTTP | partial | partial | no | yes | partial | yes |
| ollama | no | no | no | yes | partial | yes |
| sglang / tgi / openai | partial/no | partial/no | no | yes | partial | yes |

---

## Everyday CLI

```bash
eleanity doctor --check-backends
eleanity compare --backends transformers,vllm --format text
eleanity test fixtures/public/tokenizer-edge.yaml --backends fake,fake --format quiet
eleanity report <run-id> --format text
eleanity replay <run-id>
eleanity stabilize --backend fake --repetitions 3 --format quiet
eleanity policy-spec --policy quantized
```

Exit codes: `0` pass family · `1` divergent / gate fail · `2` config / dependency error.

Full reference: [docs/cli.md](docs/cli.md)

---

## Python API (no subprocess)

```python
from eleanity import Eleanity

client = Eleanity()  # or Eleanity.from_yaml("eleanity.yaml")
result = client.compare(model="demo", backends=["fake", "fake"], no_gates=True)
print(result.status, result.first_divergence, result.coverage)
raise SystemExit(result.exit_code)
```

Low-level observe/compare: `from eleanity.api import observe_backend, compare_traces, make_scenario`.  
Full reference: [docs/api.md](docs/api.md).

---

## CI

| Workflow | Role |
| --- | --- |
| `ci.yml` | **Required quality:** actionlint + ruff + mypy + unit/contract/integration + CLI smoke |
| `ci.yml` typecheck job | Required mypy gate |
| `parity-local-ai.yml` | Downloads SmolLM2-135M and runs real Transformers self-parity |
| `parity-public-fixtures.yml` | Public fixture suites |
| `release.yml` | Idempotent PyPI publish, immutable tag, and GitHub Release for reviewed versions |
| `eleanity.yml` | Reusable monorepo gate |

Local:

```bash
uv run ruff check src tests
uv run ruff format --check src tests
uv run pytest -q
```

---

## Project docs

| Doc | Purpose |
| --- | --- |
| [docs/cli.md](docs/cli.md) | CLI reference |
| [docs/api.md](docs/api.md) | Python client + low-level API |
| [docs/parity-specification.md](docs/parity-specification.md) | Status + comparator tables |
| [docs/adapter-capabilities.md](docs/adapter-capabilities.md) | Honesty matrix |
| [docs/trace-specification.md](docs/trace-specification.md) | Trace Spec v1 |
| [ROADMAP.md](ROADMAP.md) | Stable roadmap and known boundaries |
| [SUPPORT.md](SUPPORT.md) | Where to get help |
| [SECURITY.md](SECURITY.md) | Vulnerability reporting |

---

## License

Apache License 2.0 — full text in [LICENSE](LICENSE).
