Metadata-Version: 2.4
Name: evalforge-runtime
Version: 0.2.6
Summary: Runtime engine for EvalForge generated applications
Project-URL: Homepage, https://github.com/JannisConen/evalforge-runtime
Project-URL: Repository, https://github.com/JannisConen/evalforge-runtime
Project-URL: Issues, https://github.com/JannisConen/evalforge-runtime/issues
Author: Jannis Conen
License-Expression: MIT
License-File: LICENSE
Keywords: ai,evalforge,evaluation,fastapi,llm,runtime
Classifier: Development Status :: 3 - Alpha
Classifier: Framework :: FastAPI
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Libraries
Requires-Python: >=3.10
Requires-Dist: aiosqlite>=0.20.0
Requires-Dist: fastapi>=0.115.0
Requires-Dist: httpx>=0.28.0
Requires-Dist: litellm>=1.83.3
Requires-Dist: pydantic-settings>=2.7.0
Requires-Dist: pydantic>=2.10.0
Requires-Dist: python-dotenv>=1.0.0
Requires-Dist: python-multipart>=0.0.18
Requires-Dist: pyyaml>=6.0
Requires-Dist: sqlalchemy>=2.0.0
Requires-Dist: uvicorn[standard]>=0.34.0
Provides-Extra: all
Requires-Dist: apscheduler<4.0.0,>=3.10.0; extra == 'all'
Requires-Dist: azure-identity>=1.15.0; extra == 'all'
Requires-Dist: azure-keyvault-secrets>=4.8.0; extra == 'all'
Requires-Dist: boto3>=1.34.0; extra == 'all'
Requires-Dist: google-auth>=2.0.0; extra == 'all'
Requires-Dist: gradio>=5.0.0; extra == 'all'
Requires-Dist: langfuse>=2.0.0; extra == 'all'
Provides-Extra: aws
Requires-Dist: boto3>=1.34.0; extra == 'aws'
Provides-Extra: azure
Requires-Dist: azure-identity>=1.15.0; extra == 'azure'
Requires-Dist: azure-keyvault-secrets>=4.8.0; extra == 'azure'
Provides-Extra: dev
Requires-Dist: pytest-asyncio>=0.24.0; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff>=0.9.0; extra == 'dev'
Provides-Extra: exchange
Requires-Dist: httpx>=0.28.0; extra == 'exchange'
Provides-Extra: gmail
Requires-Dist: google-auth>=2.0.0; extra == 'gmail'
Provides-Extra: langfuse
Requires-Dist: langfuse>=2.0.0; extra == 'langfuse'
Provides-Extra: scheduling
Requires-Dist: apscheduler<4.0.0,>=3.10.0; extra == 'scheduling'
Provides-Extra: ui
Requires-Dist: gradio>=5.0.0; extra == 'ui'
Description-Content-Type: text/markdown

# evalforge-runtime

Runtime engine for [EvalForge](https://github.com/JannisConen/evalforge-runtime) generated applications. Provides a FastAPI server that executes LLM-powered processes with cost tracking, execution logging, secret management, and file handling.

> Completely vibe-coded in 1 hour.

## Installation

```bash
pip install evalforge-runtime
```

With optional extras:

```bash
# Langfuse observability
pip install evalforge-runtime[langfuse]

# Azure Key Vault secrets
pip install evalforge-runtime[azure]

# AWS Secrets Manager
pip install evalforge-runtime[aws]

# Everything
pip install evalforge-runtime[all]
```

## Quick start

### 1. Create a config file

Create `evalforge.config.yaml` in your project root:

```yaml
project:
  id: my-project

llm:
  model: anthropic/claude-sonnet-4-6

processes:
  classify:
    process_id: classify
    trigger:
      type: webhook
    instructions: |
      Classify the incoming support ticket into one of: billing, technical, general.
```

### 2. Set your API keys

```bash
export ENDPOINT_API_KEYS=my-secret-key
export ANTHROPIC_API_KEY=sk-ant-...
```

`ENDPOINT_API_KEYS` is a comma-separated list of keys that authenticate requests to the runtime.

### 3. Start the server

```bash
evalforge-runtime start --config evalforge.config.yaml
```

The server starts on `http://localhost:8000` by default.

### 4. Call a process

```bash
curl -X POST http://localhost:8000/process/classify \
  -H "Authorization: Bearer my-secret-key" \
  -H "Content-Type: application/json" \
  -d '{"ticket_text": "I cannot log in to my account"}'
```

## API endpoints

| Method | Path | Description |
|--------|------|-------------|
| `GET` | `/health` | Health check with version and uptime |
| `POST` | `/process/<name>` | Execute a process (authenticated) |
| `GET` | `/executions` | Query execution history |
| `GET` | `/executions/:id` | Single execution detail |
| `GET` | `/executions/stats` | Aggregated cost, latency, and volume stats |

## Configuration

The runtime is configured via `evalforge.config.yaml`. Environment variables can be referenced with `${VAR_NAME}` syntax.

### Top-level sections

```yaml
project:
  id: my-project              # Required: project identifier
  evalforge_url: https://...   # Optional: EvalForge server URL
  version: 1.0.0

llm:
  model: anthropic/claude-sonnet-4-6  # Default LLM model (LiteLLM format)

database:
  url: sqlite+aiosqlite:///./data/app.db  # Execution log database

storage:
  type: local          # local or s3
  path: ./data/files   # Local file storage path

auth:
  methods:
    - type: api_key    # api_key or oauth2

secrets:
  provider: env        # env, evalforge, azure_keyvault, aws_secrets_manager, sap_credential_store

observability:
  langfuse:
    enabled: false
    host: https://cloud.langfuse.com
```

### Process configuration

Each process defines an LLM-powered task:

```yaml
processes:
  my-process:
    process_id: my-process
    trigger:
      type: webhook           # webhook, schedule, or process
      cron: "0 9 * * *"       # For schedule triggers
      after: other-process    # For process triggers (chaining)
    instructions: |
      Your LLM instructions here...
    llm_model: openai/gpt-4o  # Override default model
    connector: exchange        # Optional: built-in connector
    review:
      enabled: true
      timeout: 24h
```

### Custom process modules

For advanced logic, provide Python modules for the three-phase pipeline:

```yaml
processes:
  my-process:
    process_id: my-process
    trigger:
      type: webhook
    before_module: processes.my_process.before     # Pre-processing
    execution_module: processes.my_process.execution  # Custom LLM call
    after_module: processes.my_process.after        # Post-processing
```

Each module should define an async function matching the phase signature. See the [EvalForge docs](https://github.com/JannisConen/evalforge-runtime) for details.

## Secret providers

The runtime supports multiple secret backends:

| Provider | Config value | Description |
|----------|-------------|-------------|
| Environment | `env` | Read from environment variables (default) |
| EvalForge | `evalforge` | Fetch from EvalForge server |
| Azure Key Vault | `azure_keyvault` | Azure Key Vault (`pip install evalforge-runtime[azure]`) |
| AWS Secrets Manager | `aws_secrets_manager` | AWS Secrets Manager (`pip install evalforge-runtime[aws]`) |
| SAP Credential Store | `sap_credential_store` | SAP BTP Credential Store |

## CLI reference

```
evalforge-runtime start [OPTIONS]

Options:
  --config PATH   Path to evalforge.config.yaml (default: evalforge.config.yaml)
  --host TEXT      Bind host (default: 0.0.0.0)
  --port INT       Bind port (default: 8000)
```

## Development

```bash
# Clone and install
git clone https://github.com/JannisConen/evalforge-runtime.git
cd evalforge-runtime
pip install -e ".[dev]"

# Run tests
pytest -v

# Lint
ruff check .
```

## License

MIT
