Metadata-Version: 2.4
Name: aegis-eval
Version: 0.1.1
Summary: The Adaptive Intelligence Layer for AI Agents — eval, train, memory, environments.
Project-URL: Homepage, https://metronis.space/
Project-URL: Documentation, https://github.com/metronis-space/aegis#readme
Project-URL: Repository, https://github.com/metronis-space/aegis
Project-URL: Issues, https://github.com/metronis-space/aegis/issues
Project-URL: Changelog, https://github.com/metronis-space/aegis/blob/main/CHANGELOG.md
Project-URL: Security, https://github.com/metronis-space/aegis/blob/main/SECURITY.md
Project-URL: Support, https://github.com/metronis-space/aegis/blob/main/SUPPORT.md
Author-email: "Metronis, Inc." <support@metronis.space>
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: aegis,agents,ai,benchmarks,eval,evaluation,grpo,llm,memory,metronis,observability,reinforcement-learning,rl,training
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.11
Requires-Dist: httpx<1.0,>=0.27
Requires-Dist: pydantic-settings<3.0,>=2.0
Requires-Dist: pydantic<3.0,>=2.0
Requires-Dist: python-dotenv<2.0,>=1.0
Requires-Dist: pyyaml<7.0,>=6.0
Requires-Dist: rich<15.0,>=13.0
Requires-Dist: tenacity<10.0,>=8.0
Requires-Dist: typer<1.0,>=0.9
Provides-Extra: all
Requires-Dist: alembic<2.0,>=1.13; extra == 'all'
Requires-Dist: cachetools>=7.0.5; extra == 'all'
Requires-Dist: datasets<5.0,>=2.0; extra == 'all'
Requires-Dist: diskcache<6.0,>=5.6; extra == 'all'
Requires-Dist: docling<3.0,>=2.0; extra == 'all'
Requires-Dist: fastapi<1.0,>=0.115; extra == 'all'
Requires-Dist: huggingface-hub<2.0,>=0.20; extra == 'all'
Requires-Dist: instructor<2.0,>=1.0; extra == 'all'
Requires-Dist: neo4j<7.0,>=5.0; extra == 'all'
Requires-Dist: numpy<3.0,>=1.26; extra == 'all'
Requires-Dist: peft>=0.11; extra == 'all'
Requires-Dist: pgvector<1.0,>=0.3; extra == 'all'
Requires-Dist: playwright<2.0,>=1.40; extra == 'all'
Requires-Dist: psycopg[binary]<4.0,>=3.1; extra == 'all'
Requires-Dist: python-multipart<1.0,>=0.0.9; extra == 'all'
Requires-Dist: redis<6.0,>=5.0; extra == 'all'
Requires-Dist: scikit-learn<2.0,>=1.5; extra == 'all'
Requires-Dist: sentence-transformers<6.0,>=3.0; extra == 'all'
Requires-Dist: sqlalchemy<3.0,>=2.0; extra == 'all'
Requires-Dist: torch>=2.0; extra == 'all'
Requires-Dist: transformers>=4.40; extra == 'all'
Requires-Dist: uvicorn[standard]<1.0,>=0.30; extra == 'all'
Requires-Dist: verl>=0.7; extra == 'all'
Requires-Dist: websockets<14.0,>=12.0; extra == 'all'
Provides-Extra: api
Requires-Dist: fastapi<1.0,>=0.115; extra == 'api'
Requires-Dist: python-multipart<1.0,>=0.0.9; extra == 'api'
Requires-Dist: uvicorn[standard]<1.0,>=0.30; extra == 'api'
Requires-Dist: websockets<14.0,>=12.0; extra == 'api'
Provides-Extra: browser
Requires-Dist: playwright<2.0,>=1.40; extra == 'browser'
Provides-Extra: data
Requires-Dist: datasets<5.0,>=2.0; extra == 'data'
Requires-Dist: huggingface-hub<2.0,>=0.20; extra == 'data'
Provides-Extra: db
Requires-Dist: alembic<2.0,>=1.13; extra == 'db'
Requires-Dist: neo4j<7.0,>=5.0; extra == 'db'
Requires-Dist: pgvector<1.0,>=0.3; extra == 'db'
Requires-Dist: psycopg[binary]<4.0,>=3.1; extra == 'db'
Requires-Dist: redis<6.0,>=5.0; extra == 'db'
Requires-Dist: sqlalchemy<3.0,>=2.0; extra == 'db'
Provides-Extra: dev
Requires-Dist: mypy<2.0,>=1.13; extra == 'dev'
Requires-Dist: pre-commit<5.0,>=4.0; extra == 'dev'
Requires-Dist: pytest-asyncio<2.0,>=0.24; extra == 'dev'
Requires-Dist: pytest-cov<8.0,>=5.0; extra == 'dev'
Requires-Dist: pytest<10.0,>=8.0; extra == 'dev'
Requires-Dist: ruff<1.0,>=0.8; extra == 'dev'
Requires-Dist: twine<7.0,>=5.0; extra == 'dev'
Provides-Extra: docs
Requires-Dist: mkdocs-material<10.0,>=9.0; extra == 'docs'
Requires-Dist: mkdocstrings[python]<2.0,>=0.27; extra == 'docs'
Provides-Extra: full
Requires-Dist: alembic<2.0,>=1.13; extra == 'full'
Requires-Dist: cachetools>=7.0.5; extra == 'full'
Requires-Dist: datasets<5.0,>=2.0; extra == 'full'
Requires-Dist: diskcache<6.0,>=5.6; extra == 'full'
Requires-Dist: docling<3.0,>=2.0; extra == 'full'
Requires-Dist: fastapi<1.0,>=0.115; extra == 'full'
Requires-Dist: huggingface-hub<2.0,>=0.20; extra == 'full'
Requires-Dist: instructor<2.0,>=1.0; extra == 'full'
Requires-Dist: neo4j<7.0,>=5.0; extra == 'full'
Requires-Dist: numpy<3.0,>=1.26; extra == 'full'
Requires-Dist: peft>=0.11; extra == 'full'
Requires-Dist: pgvector<1.0,>=0.3; extra == 'full'
Requires-Dist: playwright<2.0,>=1.40; extra == 'full'
Requires-Dist: psycopg[binary]<4.0,>=3.1; extra == 'full'
Requires-Dist: python-multipart<1.0,>=0.0.9; extra == 'full'
Requires-Dist: redis<6.0,>=5.0; extra == 'full'
Requires-Dist: scikit-learn<2.0,>=1.5; extra == 'full'
Requires-Dist: sentence-transformers<6.0,>=3.0; extra == 'full'
Requires-Dist: sqlalchemy<3.0,>=2.0; extra == 'full'
Requires-Dist: torch>=2.0; extra == 'full'
Requires-Dist: transformers>=4.40; extra == 'full'
Requires-Dist: uvicorn[standard]<1.0,>=0.30; extra == 'full'
Requires-Dist: verl>=0.7; extra == 'full'
Requires-Dist: websockets<14.0,>=12.0; extra == 'full'
Provides-Extra: gpu
Requires-Dist: cachetools>=7.0.5; extra == 'gpu'
Requires-Dist: diskcache<6.0,>=5.6; extra == 'gpu'
Requires-Dist: instructor<2.0,>=1.0; extra == 'gpu'
Requires-Dist: numpy<3.0,>=1.26; extra == 'gpu'
Requires-Dist: peft>=0.11; extra == 'gpu'
Requires-Dist: scikit-learn<2.0,>=1.5; extra == 'gpu'
Requires-Dist: sentence-transformers<6.0,>=3.0; extra == 'gpu'
Requires-Dist: torch>=2.0; extra == 'gpu'
Requires-Dist: transformers>=4.40; extra == 'gpu'
Requires-Dist: verl>=0.7; extra == 'gpu'
Provides-Extra: ingestion
Requires-Dist: docling<3.0,>=2.0; extra == 'ingestion'
Provides-Extra: launch
Requires-Dist: cachetools>=7.0.5; extra == 'launch'
Requires-Dist: datasets<5.0,>=2.0; extra == 'launch'
Requires-Dist: diskcache<6.0,>=5.6; extra == 'launch'
Requires-Dist: fastapi<1.0,>=0.115; extra == 'launch'
Requires-Dist: huggingface-hub<2.0,>=0.20; extra == 'launch'
Requires-Dist: instructor<2.0,>=1.0; extra == 'launch'
Requires-Dist: numpy<3.0,>=1.26; extra == 'launch'
Requires-Dist: peft>=0.11; extra == 'launch'
Requires-Dist: python-multipart<1.0,>=0.0.9; extra == 'launch'
Requires-Dist: scikit-learn<2.0,>=1.5; extra == 'launch'
Requires-Dist: sentence-transformers<6.0,>=3.0; extra == 'launch'
Requires-Dist: torch>=2.0; extra == 'launch'
Requires-Dist: transformers>=4.40; extra == 'launch'
Requires-Dist: uvicorn[standard]<1.0,>=0.30; extra == 'launch'
Requires-Dist: verl>=0.7; extra == 'launch'
Requires-Dist: websockets<14.0,>=12.0; extra == 'launch'
Provides-Extra: observability
Requires-Dist: opentelemetry-api<2.0,>=1.20; extra == 'observability'
Requires-Dist: opentelemetry-exporter-otlp-proto-grpc<2.0,>=1.20; extra == 'observability'
Requires-Dist: opentelemetry-exporter-otlp-proto-http<2.0,>=1.20; extra == 'observability'
Requires-Dist: opentelemetry-instrumentation-httpx<1.0,>=0.44; extra == 'observability'
Requires-Dist: opentelemetry-sdk<2.0,>=1.20; extra == 'observability'
Provides-Extra: operator
Requires-Dist: fastapi<1.0,>=0.115; extra == 'operator'
Requires-Dist: python-multipart<1.0,>=0.0.9; extra == 'operator'
Requires-Dist: uvicorn[standard]<1.0,>=0.30; extra == 'operator'
Requires-Dist: websockets<14.0,>=12.0; extra == 'operator'
Provides-Extra: research
Requires-Dist: datasets<5.0,>=2.0; extra == 'research'
Requires-Dist: diskcache<6.0,>=5.6; extra == 'research'
Requires-Dist: fastapi<1.0,>=0.115; extra == 'research'
Requires-Dist: huggingface-hub<2.0,>=0.20; extra == 'research'
Requires-Dist: instructor<2.0,>=1.0; extra == 'research'
Requires-Dist: numpy<3.0,>=1.26; extra == 'research'
Requires-Dist: python-multipart<1.0,>=0.0.9; extra == 'research'
Requires-Dist: scikit-learn<2.0,>=1.5; extra == 'research'
Requires-Dist: sentence-transformers<6.0,>=3.0; extra == 'research'
Requires-Dist: uvicorn[standard]<1.0,>=0.30; extra == 'research'
Requires-Dist: websockets<14.0,>=12.0; extra == 'research'
Provides-Extra: sandbox
Requires-Dist: docker<8.0,>=7.0; extra == 'sandbox'
Provides-Extra: scoring
Requires-Dist: diskcache<6.0,>=5.6; extra == 'scoring'
Requires-Dist: instructor<2.0,>=1.0; extra == 'scoring'
Requires-Dist: numpy<3.0,>=1.26; extra == 'scoring'
Requires-Dist: scikit-learn<2.0,>=1.5; extra == 'scoring'
Requires-Dist: sentence-transformers<6.0,>=3.0; extra == 'scoring'
Provides-Extra: training
Requires-Dist: cachetools>=7.0.5; extra == 'training'
Requires-Dist: peft>=0.11; extra == 'training'
Requires-Dist: torch>=2.0; extra == 'training'
Requires-Dist: transformers>=4.40; extra == 'training'
Requires-Dist: verl>=0.7; extra == 'training'
Description-Content-Type: text/markdown

# Aegis

[![CI](https://github.com/metronis-space/aegis/actions/workflows/ci.yml/badge.svg)](https://github.com/metronis-space/aegis/actions/workflows/ci.yml)
[![Repo Hygiene](https://github.com/metronis-space/aegis/actions/workflows/repo-hygiene.yml/badge.svg)](https://github.com/metronis-space/aegis/actions/workflows/repo-hygiene.yml)
[![Docs](https://github.com/metronis-space/aegis/actions/workflows/docs.yml/badge.svg)](https://github.com/metronis-space/aegis/actions/workflows/docs.yml)
[![Link Check](https://github.com/metronis-space/aegis/actions/workflows/link-check.yml/badge.svg)](https://github.com/metronis-space/aegis/actions/workflows/link-check.yml)
[![CodeQL](https://github.com/metronis-space/aegis/actions/workflows/codeql.yml/badge.svg)](https://github.com/metronis-space/aegis/actions/workflows/codeql.yml)
[![Python 3.11+](https://img.shields.io/badge/python-3.11%2B-blue.svg)](https://www.python.org/downloads/)
[![License: Apache 2.0](https://img.shields.io/badge/License-Apache_2.0-green.svg)](https://opensource.org/licenses/Apache-2.0)

**The Closed-Loop Intelligence Engine for AI Agents** — eval, train, and deploy smarter agents.

Aegis by [Metronis, Inc.](https://metronis.space/) is an open-source framework that evaluates AI agents across 100+ dimensions, identifies weaknesses, trains improved models via RL, and proves the improvement with rigorous before/after evaluation.

Quick links: [Quick Start](docs/quickstart.md) | [Examples](docs/examples.md) | [Community Guide](docs/community.md) | [Security Model](docs/security-model.md) | [Maintainer Triage](docs/triage.md) | [Contributing](CONTRIBUTING.md) | [Governance](GOVERNANCE.md) | [Releasing](docs/releasing.md) | [Support](SUPPORT.md)

## Benchmark Evidence Tiers

Aegis exposes three different benchmark evidence modes and they are not interchangeable:

- `internal proxy`: built-in `*-proxy` suites are for internal regression tracking and ablations.
- `public proxy`: built-in `*-heldout` suites use held-out slices and benchmark-native metrics, but they are still not official leaderboard-grade evidence.
- `claim-grade`: only manifest-backed claim suites are intended for externally claimable benchmark results.

The built-in `legal` alias currently resolves to `legal-heldout`, which is a held-out proxy suite, not an official claim-grade legal benchmark. Treat strict built-in runs as honest proxy evidence unless the suite contract explicitly reports claim eligibility.
The repo does not bundle claim-grade legal suite JSON by default; install real
manifests under `benchmarks/claim_suites/` or set `AEGIS_BENCHMARK_SUITE_DIR`
to an external manifest directory.

## The Pipeline

```
Traces In → Eval → Diagnose Weaknesses → Spin RL Environments →
Train (GRPO + Continuous Rewards) → Store Results to Memory →
Train with Tools + Self-Managed Memory → Final Eval (Prove Improvement)
```

```mermaid
flowchart LR
    A["Agent Traces"] --> B["Aegis Eval\n100+ dimensions"]
    B --> C["Weakness\nDiagnosis"]
    C --> D["RL Environments\nauto-generated"]
    D --> E["GRPO Training\ncontinuous rewards"]
    E --> F["Aegis Memory\nextract strategies"]
    F --> G["Tool + Memory\nTraining"]
    G --> H["Final Eval\nprove improvement"]
    H --> A
```

## Three Products

| Product | What it does | Status |
|---------|-------------|--------|
| **Aegis Eval** | 100+ dimensions across 7 tiers, triangulated scoring (rule + semantic + LLM judge), legal, finance, and safety plugins | Working |
| **Aegis Train** | GRPO-based RL with continuous reward functions, environment factory, Observatory monitoring | Building |
| **Aegis Memory** | 12 operations, knowledge graph, vector store, provenance tracking, 7 time horizons | Working |

## Installation

```bash
pip install -e ".[dev,operator]"
```

Install profiles:

- `.[dev,operator]` — lean local dev path for the CLI, API, and hosted control plane work.
- `.[dev,research]` — adds scoring and dataset tooling for eval-focused research work.
- `.[dev,launch]` — adds the strict launch path dependencies for real scoring and training.
- `.[dev,all]` — the heaviest internal profile; only use it when you explicitly need every optional integration.

If you are working specifically on the launch-critical strict path, prefer:

```bash
pip install -e ".[dev,launch]"
```

For the exact launch checklist, see [`docs/launch-readiness.md`](docs/launch-readiness.md)
and [`docs/launch-gap-audit.md`](docs/launch-gap-audit.md).

For a fast public-repo sanity pass before you push, run:

```bash
make oss-check
```

That check now also verifies that GitHub workflow actions are pinned to
immutable commit SHAs instead of moving tags or branches.

To discover the main local workflows, run:

```bash
make help
```

To sync the repo's canonical GitHub labels from the checked-in taxonomy, run:

```bash
make labels-sync
```

To audit the live GitHub repository settings against the launch policy, run:

```bash
make github-launch-audit
```

To preview the repo-level GitHub settings patch we want before public launch, run:

```bash
make github-launch-apply
```

## Quick Start

### Evaluate an agent

```python
from aegis import Evaluator, EvalConfig

evaluator = Evaluator(config=EvalConfig(dimensions="all"))
result = evaluator.run()

print(f"Overall score: {result.overall_score:.2%}")
for tier_name, tier_score in result.tier_scores.items():
    print(f"  {tier_name}: {tier_score:.2%}")
```

### CLI

```bash
aegis guide                           # Show the recommended launch-grade workflows
aegis signup --name "Your Name"       # Create a hosted account and save the CLI session
aegis onboard --domain legal          # Run the hosted readiness flow and get the next commands
aegis login --json                    # Save a hosted session and emit machine-readable output
aegis config show                     # Inspect resolved URLs, state paths, and saved session metadata
aegis doctor --json                   # Run launch-path health checks for CI and scripts
aegis completion show --shell zsh     # Print an explicit completion script for your shell
aegis completion install --shell zsh  # Install completion without relying on shell autodetect
aegis app url --path training         # Resolve a hosted app URL without opening a browser
aegis app open --print-url            # Open the app or print the URL for remote terminals
aegis eval dimensions --json          # Inspect dimensions in automation-safe JSON
aegis eval trend --dimension retention_accuracy --json
aegis eval benchmark-validate --manifest benchmarks/claim.json --json
aegis trace list --json               # List stored traces with full ids for scripts
aegis trace import traces.jsonl --json
aegis data list --json                # Inspect cached datasets programmatically
aegis memory health --json            # Inspect memory subsystem status programmatically
aegis memory audit --json             # Inspect memory audit events programmatically
aegis eval run --config eval.yaml     # Run evaluation
aegis eval dimensions                 # List all registered dimensions
aegis eval benchmark-list             # Inspect proxy vs claim-grade suite status
aegis train start --domain legal --backend verl --run-now
aegis train list                      # Inspect hosted or local draft jobs
aegis train watch --job-id JOB_ID     # Follow live job state
aegis train metrics --job-id JOB_ID   # Inspect rewards and stages
aegis train adapter list              # Manage trained adapters
aegis logout                          # Clear the saved CLI session
```

The hosted `standard` backend is intentionally labeled as simulated/deterministic
in CLI output. Use `--backend verl` on a real GPU-backed runtime for actual
training instead of placeholder rollouts.

### Launch Runtime Posture

```bash
make aws-launch-preflight
make aws-launch-debug-ssh
make aws-launch-refresh-ssh
make aws-launch-dns-targets
make launch-manual-gpu-sync LAUNCH_MANUAL_GPU_HOST=<gpu-host>
make launch-manual-gpu-bootstrap LAUNCH_MANUAL_GPU_HOST=<gpu-host>
make launch-manual-gpu-remote-preflight LAUNCH_MANUAL_GPU_HOST=<gpu-host>
make launch-manual-gpu-remote-baseline LAUNCH_MANUAL_GPU_HOST=<gpu-host>
```

For launch, `AWS` is the public hosting and deployment plane, while the
flagship strict proof flow should be run manually from a real GPU worker
outside AWS, such as a Nebius machine. The checked-in launch configs now use
`backend: local` intentionally so they fail on laptops and non-GPU hosts rather
than silently falling back to simulated training.

## Architecture

```
src/aegis/
├── core/           # Types, config, settings (5 files)
├── cli/            # Typer CLI (1 file)
├── adapters/       # OpenAI, Anthropic, LangChain, REST (6 files)
├── api/            # FastAPI + eval/training/memory routes (9 files)
├── data/           # CUAD, LegalBench, FinanceBench loaders (5 files)
├── ingestion/      # Document parsing pipeline (6 files)
├── eval/           # Engine, 7-tier dimensions, scorers, judges (27 files)
├── environments/   # Legal & finance tool-use RL environments (4 files)
├── training/       # GRPO engine, rewards, rollouts, optimizers (16 files)
├── memory/         # 12 ops, graph, vectors, provenance (16 files)
├── plugins/        # Legal (18 dims) + Finance (20 dims) + Safety (12 dims) (6 files)
├── observatory/    # Goodhart detection, efficiency tracking (4 files)
├── security/       # RBAC (2 files)
└── store/          # SQLite, Postgres, Neo4j (5 files)
```

~100 source files. One pipeline. No bloat.

## Adapters

| Adapter | Framework |
|---------|-----------|
| `OpenAIAdapter` | OpenAI Assistants + Chat Completions |
| `AnthropicAdapter` | Anthropic Messages API |
| `LangChainAdapter` | LangChain agents |
| `RESTAdapter` | Any REST API |

## Domain Plugins

- **Legal**: 18 dimensions, CUAD dataset, contract clause extraction, citation verification
- **Finance**: 20 dimensions, FinanceBench + SEC EDGAR, numerical accuracy, formula validation

## Documentation

| Topic | Link |
|-------|------|
| Quick Start | [`docs/quickstart.md`](docs/quickstart.md) |
| Examples | [`docs/examples.md`](docs/examples.md) |
| Open-Source Release | [`docs/open-source-release.md`](docs/open-source-release.md) |
| Security Model | [`docs/security-model.md`](docs/security-model.md) |
| Maintainer Triage | [`docs/triage.md`](docs/triage.md) |
| Architecture | [`docs/architecture.md`](docs/architecture.md) |
| Community Guide | [`docs/community.md`](docs/community.md) |
| Master Implementation Plan | [`docs/master-implementation-plan.md`](docs/master-implementation-plan.md) |
| CLI Reference | [`docs/cli-reference.md`](docs/cli-reference.md) |
| API Reference | [`docs/api-reference.md`](docs/api-reference.md) |
| Dimensions | [`docs/dimensions.md`](docs/dimensions.md) |
| Scoring | [`docs/scoring.md`](docs/scoring.md) |
| Adapters | [`docs/adapters.md`](docs/adapters.md) |
| Plugins | [`docs/plugins.md`](docs/plugins.md) |
| Support | [`SUPPORT.md`](SUPPORT.md) |

## Contributing

See [CONTRIBUTING.md](CONTRIBUTING.md).

Public community norms live in [CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md), and
security disclosures should follow [SECURITY.md](SECURITY.md). Public support
and issue-routing guidance live in [SUPPORT.md](SUPPORT.md). Repository
governance and maintainer expectations live in [GOVERNANCE.md](GOVERNANCE.md)
and [MAINTAINERS.md](MAINTAINERS.md). Release procedure lives in
[docs/releasing.md](docs/releasing.md). Public trust boundaries live in
[docs/security-model.md](docs/security-model.md).
Maintainer triage flow lives in [docs/triage.md](docs/triage.md).

## Open-Source Posture

This repository is intended to be public open-core source code.

That means:

- the engine, CLI, API, reference dashboard, and self-host references live here
- production secrets, private benchmark manifests, customer data, and managed
  service credentials do not
- benchmark claims should stay aligned with the evidence tier reported by the
  runtime and manifests

For the release policy behind that boundary, see
[`docs/open-source-release.md`](docs/open-source-release.md) and
[`docs/open-core-boundary.md`](docs/open-core-boundary.md).

## License

Apache License 2.0. See [LICENSE](LICENSE).

Built by [Metronis, Inc.](https://metronis.space/)
