Metadata-Version: 2.4
Name: afterai-runner
Version: 0.1.4
Summary: AfterAI Runner — AIS signal emission and agent evaluation on customer infrastructure
Author-email: AfterAI <support@useafter.ai>
License: MIT
Keywords: afterai,ais,runner,eval,signals,evaluation
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Requires-Python: >=3.9
Description-Content-Type: text/markdown
Requires-Dist: pydantic>=2.0.0
Requires-Dist: requests>=2.28.0
Requires-Dist: afterai>=0.1.0
Requires-Dist: pyyaml>=6.0
Provides-Extra: dev
Requires-Dist: pytest>=7.0.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0.0; extra == "dev"

# AfterAI Runner

Run AI agent evaluations and emit AIS signals — on your own infrastructure.

- **Eval**: poll AfterAI for pending eval jobs, run prompts against your agent, submit responses for scoring
- **AIS emission**: send pre-aggregated metric signals to the AfterAI API from local artifact files
- **No inference proxy**: the runner runs on your infra; AfterAI never sees your agent's API keys or endpoints

## Install

```bash
pip install afterai-runner
```

Or from source:

```bash
cd afterai-runner && pip install -e .
```

## Config

| Env | Required | Default |
|-----|----------|---------|
| `AFTERAI_API_KEY` | Yes | — |
| `AFTERAI_BASE_URL` | No | `https://api.useafter.ai` |

---

## Eval

Run evaluations against your agent. AfterAI generates a synthetic dataset, you approve it in the app, then the runner executes it locally and submits results for scoring.

### Quick start

```bash
# 1. Create a starter agent config
afterai-runner eval-init

# 2. Edit afterai-agent.yaml with your agent details (see below)

# 3. Run — polls for pending eval jobs and processes them
afterai-runner eval
```

### Agent config (`afterai-agent.yaml`)

**Azure AI Foundry (v2 — recommended)**

```yaml
agent:
  type: foundry
  foundry_version: v2
  project_endpoint: https://YOUR_RESOURCE.services.ai.azure.com/api/projects/YOUR_PROJECT
  agent_name: your-agent-name
  # Auth: set AZURE_FOUNDRY_TOKEN env var, or run `az login`
```

**Azure AI Foundry (v1 — legacy)**

```yaml
agent:
  type: foundry
  foundry_version: v1
  project_endpoint: https://YOUR_PROJECT.services.ai.azure.com
  agent_id: asst_YOUR_AGENT_ID
  api_key: ${AZURE_API_KEY}
```

**Generic HTTP**

```yaml
agent:
  type: http
  url: https://your-agent.example.com/chat
  headers:
    Authorization: Bearer ${YOUR_API_KEY}
  body_template: '{"message": "{prompt}"}'
  response_path: response.text
```

> Add `afterai-agent.yaml` to `.gitignore` — keep your credentials local.

### Eval commands

```bash
# Poll continuously (default: every 10s)
afterai-runner eval

# Process pending jobs once and exit
afterai-runner eval --once

# Custom poll interval
afterai-runner eval --poll-interval 30

# Custom config path
afterai-runner eval --agent-config /path/to/my-agent.yaml
```

### Generate a dataset from your Foundry agent (optional)

If your agent is on Foundry, the runner can fetch its definition and generate an eval dataset automatically:

```bash
afterai-runner dataset-generate
```

This calls the AfterAI API to generate a synthetic dataset based on your agent's instructions and tools. You then review and approve it in the app before any eval runs.

---

## AIS Emission

Send pre-aggregated metric signals from local artifact files to the AfterAI API.

```bash
# Run ingestion from a manifest
afterai-runner run --manifest manifest.yaml

# Print payloads without sending
afterai-runner run --manifest manifest.yaml --dry-run

# ACE-from-AIS rule: read AIS signals, emit inferred ACE if threshold met
afterai-runner ace-from-ais --min-signals 2 --hours 24
afterai-runner ace-from-ais --min-signals 2 --severity high,critical --dry-run
```

### Manifest example

```yaml
version: "2026-02"
artifacts:
  - system: my-system-01
    path: /path/to/D1-2026-02-03.json
  - system: my-system-01
    path: /path/to/W1-2026-W05.json
```

### Artifact example

One file = one signal. `D1-2026-02-03.json`:

```json
{
  "signal_key": "D1",
  "cadence": "daily",
  "window": {
    "start": "2026-02-03T00:00:00Z",
    "end": "2026-02-03T23:59:59Z"
  },
  "metrics": {
    "score_mean": 0.87,
    "fail_rate": 0.03
  }
}
```

---

## License

MIT
