Metadata-Version: 2.3
Name: screamingface
Version: 0.1.1
Summary: Compose model ensembles, benchmark them, and get a score you can reproduce
Project-URL: Homepage, https://screamingface.ai
Project-URL: Repository, https://github.com/OpenMined/screamingface
Project-URL: Issues, https://github.com/OpenMined/screamingface/issues
License-File: LICENSE
Keywords: benchmarks,ensembles,evaluation,leaderboard,notebook,openrouter,reproducibility,url4
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Typing :: Typed
Requires-Python: >=3.12
Requires-Dist: httpx>=0.27
Requires-Dist: pynacl>=1.5
Requires-Dist: pyyaml>=6.0
Requires-Dist: url4>=1.3.0
Requires-Dist: websockets<17,>=16.1
Provides-Extra: notebook
Requires-Dist: ipykernel>=6.29; extra == 'notebook'
Requires-Dist: ipywidgets>=8.1; extra == 'notebook'
Requires-Dist: jupyterlab>=4.6.2; extra == 'notebook'
Requires-Dist: nbclient>=0.10; extra == 'notebook'
Requires-Dist: nbformat>=5.10; extra == 'notebook'
Description-Content-Type: text/markdown

# 😱 ScreamingFace

> **Build model fusions, measure them honestly, and reproduce any run from a single line of text.**

`screamingface` is the Python client. You compose a **fusion** — several models answering the same
question, reduced to a single answer — score it against a real benchmark, and get back both the
number and the exact expression that produced it.

We built this because the same pattern kept showing up: a fusion beat the best single model inside
it, and the result was nearly impossible to reproduce or check. Two datapoints: reproducing DRACO,
our best fusion scored **68.6%** vs **60.2%** for the top single model (**+8.4**,
[write-up](https://andrewtrask.substack.com/p/6-weeks-ago-frontier-ai-labs-lost)); and small-model
ensembles beat their best member in _Beyond Leaderboards_ (Skurikhin et al., Los Alamos).

It is early, and rough in places. If you find something wrong, good: every result here is meant to
be rerun and picked apart.

📚 [docs.screamingface.ai](https://docs.screamingface.ai) ·
🏆 [leaderboard.screamingface.ai](https://leaderboard.screamingface.ai) ·
⚖️ [Apache-2.0](https://github.com/OpenMined/screamingface/blob/main/LICENSE)

## Install

```bash
pip install "screamingface[notebook]"   # drop the extra if you are not in Jupyter
```

Python 3.12 or newer. You also need an **engine**: the runtime that actually executes a run and
holds your provider keys. Run your own, or point at a hosted one — same client code either way,
only the URL changes.

## Quickstart

```python
import screamingface as sf

sf.configure(engine_url="http://127.0.0.1:9108")   # your own engine, or a hosted one
sf.connect()                                       # bring your keys; the client stores none

gpt = sf.Model("openrouter/openai/gpt-5.5")
opus = sf.Model("openrouter/anthropic/claude-opus-4.8")
fusion = sf.Fusion([gpt, opus], name="frontier-pair", synthesizer=opus)

# score the solo models beside the fusion, on the same cases
report = sf.evaluate([gpt, opus, fusion], benchmark="draco", limit=1)
{c.name: c.score for c in report.candidates}
```

A fusion always names its `synthesizer=` — the thing that turns several answers into one. Nothing
is guessed on your behalf. `draco` is the full 100-task benchmark and `limit=1` runs one case of
it; `draco/lite` and `draco/smoke` are cheap development protocols whose scores are deliberately
_not_ comparable to the real thing. `sf.benchmarks.list()` shows what your engine has installed.

## What it gives you

- **One interface, every provider.** Compose across open and closed models, API and local. The
  client never calls a provider directly; your keys go to the engine, which validates and stores
  them encrypted.
- **Evaluation you can defend.** The benchmark is pinned and lives engine-side — the cases, the
  judge, the rubric, the answer keys. Your candidate only ever sees the prompt. That separation is
  the whole reason a "verified gain" means anything.
- **Start from the frontier.** Pull a published run's `url4`, change one thing, and measure the
  difference, so the next person starts where the last one finished.
- **Real costs.** Tokens and USD per candidate and in aggregate, not an estimate.

Right now this is single-turn evaluation. No multi-turn or tool-using agent loops yet.

## Compose

Three immutable, network-free values. Each takes one input and returns one answer, which is why
they nest.

```python
draft = sf.Model("openrouter/openai/gpt-5.5")
review = sf.Model("openrouter/anthropic/claude-opus-4.8", prompt="Correct the previous answer.")
polish = sf.Model("openrouter/openai/gpt-5.5", prompt="Tighten it. Add nothing new.")

chain = sf.Pipeline([draft, review, polish])
chain = draft.then(review).then(polish)            # identical, just shorthand

candidate = sf.Fusion([chain, draft], synthesizer=polish)
```

Prompts and sampling parameters belong to the candidate, never to the benchmark:

```python
careful = sf.Model(
    "openrouter/openai/gpt-5.5",
    prompt="Answer from primary evidence and follow every output constraint.",
    params={"reasoning_effort": "high"},
)
```

Unknown routes, invalid parameters, and shape mismatches all fail before anything is spent.

## Read the report

```python
result = report.candidates["frontier-pair"]

result.score
result.usage.cost_usd
result.url4                    # the exact expression that ran
result.operations              # every call underneath it
report.export("runs/draco.json")
```

## Reproduce

Sharing a result is sharing its `url4`. Replay it, or turn it back into editable Python.

```python
score = sf.leaderboards.get_score(score_id)

score.url4.to_python()      # local and free: hands back the Model/Fusion/Pipeline
sf.evaluate(score.url4)     # replay it as a fresh run
```

Identical `url4` does not guarantee identical model output — but it does guarantee everyone is
measuring the same thing.

## More

- **Docs** (quickstart, guides, concepts) → [docs.screamingface.ai](https://docs.screamingface.ai)
- **Notebooks** →
  [`examples/`](https://github.com/OpenMined/screamingface/tree/main/packages/screamingface/examples),
  starting with `00_quickstart` and the no-spend `01_client_tour`
- **Source and issues** → [github.com/OpenMined/screamingface](https://github.com/OpenMined/screamingface)

Everything synchronous has an asynchronous twin: `sf.AsyncClient` uses the same names with `await`.

## License

[Apache-2.0](https://github.com/OpenMined/screamingface/blob/main/LICENSE). Open to whoever shows
up to measure the next slice.
