Metadata-Version: 2.4
Name: astraea-atomic
Version: 0.2.0
Summary: Procedural construction benchmarks with exact verification and experimental ratings
Author: Astraea
Project-URL: Homepage, https://astraea.mx
Classifier: Development Status :: 4 - Beta
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: MacOS
Classifier: Operating System :: POSIX :: Linux
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown

# Astraea Atomic

**Developed by Astraea.** Atomic tests AI models on procedurally generated construction problems with exact solution checking.

The terminal evaluator runs a selected built-in or custom model against a fixed Claude reference configuration and reports an **Atomic Constraint rating (experimental)** with a 95% confidence interval. It also includes an offline SDK for trying individual challenges.

## Run a model evaluation

You need Python 3.10 or newer, [uv](https://docs.astral.sh/uv/getting-started/installation/), Docker installed and running, and an Anthropic API key with credit. The controlled Python evaluator requires macOS or Linux and a local Docker Unix socket. Native Windows evaluation is not supported in this release. Local release validation uses macOS ARM64 and Python 3.12; the Python version declaration is not a claim that every platform/version combination has been tested.

Start the evaluator:

```sh
uvx --from astraea-atomic==0.2.0 atomic evaluate
```

The terminal shows Atomic branding and guides you through selecting a built-in or custom model, entering your API key without echoing it, choosing the number of puzzles and setting a spending limit. It prepares the isolated Python runtime, shows progress, and prints your result. No Python file or Astraea account is needed.

An existing `ANTHROPIC_API_KEY` environment variable is reused. Run evidence is saved locally under `./atomic-results/` by default; `--output PATH` chooses a new directory.

Recheck a saved run without making API calls:

```sh
atomic report PATH_TO_RUN_DIRECTORY
```

The report checks the saved puzzle generation, trial records, accounting and score. This checks local consistency, not independent authenticity. Generation seeds stay in a private evaluator file and are never given to either model.

For built-in Claude candidates, both models make paid API calls using your Anthropic key. A custom candidate uses its own connection and declared prices; the reference still uses Anthropic credit. The default run uses **32 fresh puzzle pairs: 64 model trials**, with Sonnet 5.5 at high effort anchored at 1,000 points. The default local accounting limit is **$31**, covering the conservative reservation of approximately $30.07 for all 32 pairs. This reservation is not an expected bill. Historical Sonnet runs suggest approximately $4.48 in returned usage for 32 pairs, but different models, responses, provider prices and errors can change the cost. The local limit is not an account-wide provider billing cap.

The evaluator downloads an official Python 3.12 container image when needed and records the local immutable image ID used for the run. Model-written code executes in restricted Docker containers; it has no network or host-folder access. API requests are made by the evaluator outside those containers.

Alternatively, install into a virtual environment on macOS/Linux:

```sh
python3 -m venv .venv
source .venv/bin/activate
python -m pip install astraea-atomic==0.2.0
atomic evaluate
```

See [the complete terminal walkthrough](docs/QUICKSTART.md).

## Understand your result

The experimental rating compares candidate-only successes with reference-only successes on the same fresh 384-variable Constraint Lock puzzles, using the versioned controlled Python protocol. The reference is assigned 1,000 by convention; it is not a measure of absolute intelligence.

When both counts are nonzero, the score is `1000 + 400 × log10(candidate-only successes / reference-only successes)`. Its 95% interval comes from an exact conditional binomial interval for the direction of the discordant pairs. Matching outcomes do not identify the rating difference. The numerical scale is uncapped, but the fixed 384-variable release is not proven to remain informative for arbitrarily strong future models.

For a complete run, the evaluator reports the observed pass rates and one of:

- **Estimate and 95% interval:** both models have at least one unique success.
- **An upper or lower bound:** unique successes occurred in only one direction, so a finite point estimate is unavailable.
- **Unidentified:** all paired outcomes matched; this does not establish equal ability.

An interrupted or otherwise incomplete run is identified as incomplete and does not receive a rating from its completed prefix. A run with few puzzles can produce a wide interval or only a bound. More puzzles can improve precision, but do not guarantee a finite estimate.

The uncertainty calculation is conditional on the comparison's statistical assumptions. It does not include every possible source of uncertainty, establish that all model families behave alike, or make the rating a general intelligence score. Results include skill at writing code, using tools and staying within the declared token and execution budgets. Self-run results are not an independently certified leaderboard entry.

Earlier research required a 95% interval no wider than 200 points. The completed study failed that target. This release exposes experimental estimates with their uncertainty rather than withholding every imprecise estimate; it does not retroactively mark that study as passing. Its results and amendments remain in the research record.

## Try the free offline demo

No API key, Docker or model calls are needed for the tutorials:

```sh
uvx --from astraea-atomic==0.2.0 atomic demo
```

The demo verifies two supplied example answers. It produces an **unrated practice scorecard**, not an AI evaluation or experimental rating.

For local development:

```sh
python3 -m venv .venv
source .venv/bin/activate
python -m pip install .
atomic demo
atomic evaluate
```

## Use the practice SDK

```python
from astraea_atomic import Atomic
from astraea_atomic.examples import switchyard_solution

atomic = Atomic()
env = atomic.make("switchyard-tutorial", render_mode="terminal")
env.submit(switchyard_solution())
print(atomic.get_scorecard())
```

The supplied solution demonstrates the interface. Practice scorecards retain `status: "unrated"` and `atomic_score: null`; calling the SDK does not enforce an evaluation budget or certify a model score.

Generate and inspect a reproducible practice problem:

```sh
atomic make constraint-research --seed 42 --level 1 --output challenge.json
atomic verify constraint-research solution.json --seed 42 --level 1
```

For Constraint Lock, submit `{"assignment": [...]}` with one integer `0` or `1` per variable. Any satisfying assignment is accepted. Verification scans the clauses without searching for a solution. Levels select structural sizes, not calibrated difficulty ratings.

The SDK also includes Switchyard and Counterexample Forge tutorials and research generators. They are not combined into the experimental Constraint rating. Larger problems and an uncapped numerical formula do not establish that the benchmark remains informative indefinitely.

## Research and development

The earlier internal README is preserved in [the research history](docs/RESEARCH-HISTORY.md) in the source workspace. Historical studies, provider records, private answers, credentials and local budget ledgers are excluded from the public package.

```sh
python -m pip install -e .
python -m unittest discover -s tests -v
```

Development tests belong to the full development workspace; they are not included in the public source distribution. The installed Python package has no third-party Python runtime dependencies. Model evaluation separately requires Docker, network access to Anthropic and paid API access.

## Private and local models

Create a Python adapter without adding your model to a whitelist:

```sh
uvx --from astraea-atomic==0.2.0 atomic init-adapter ./my-model
```

Implement the generated adapter and fill in its model, interpreter and price configuration. Then run:

```sh
uvx --from astraea-atomic==0.2.0 atomic evaluate --adapter ./my-model/adapter.json
```

The adapter connects your private API or local inference service. Atomic controls the same puzzles, Python workspace, limits and scoring. The fixed reference still requires Anthropic credit. Custom token usage and model identity are researcher-reported; cross-provider calibration is not established. The generated `my-model/README.md` contains the complete adapter contract and usage guide.
