Metadata-Version: 2.4
Name: veris-cli
Version: 2.31.0
Summary: CLI to connect local agents to the Veris backend
Project-URL: Homepage, https://github.com/veris-ai/veris-cli
Project-URL: Bug Tracker, https://github.com/veris-ai/veris-cli/issues
Author: Veris
Requires-Python: >=3.11
Requires-Dist: click>=8.1.7
Requires-Dist: httpx>=0.27.0
Requires-Dist: jinja2>=3.1.0
Requires-Dist: pathspec>=0.12.0
Requires-Dist: pydantic>=2.6
Requires-Dist: pyyaml>=6.0.1
Requires-Dist: questionary>=2.0.0
Requires-Dist: rich>=13.7.0
Requires-Dist: tenacity>=8.2.0
Provides-Extra: dev
Requires-Dist: pre-commit>=4.3.0; extra == 'dev'
Provides-Extra: test
Requires-Dist: pytest-asyncio>=0.23.0; extra == 'test'
Requires-Dist: pytest-cov>=4.1.0; extra == 'test'
Requires-Dist: pytest>=8.2; extra == 'test'
Description-Content-Type: text/markdown

# Veris CLI

[![PyPI version](https://badge.fury.io/py/veris-cli.svg)](https://badge.fury.io/py/veris-cli)
[![Tests](https://github.com/veris-ai/veris-cli/actions/workflows/test.yml/badge.svg)](https://github.com/veris-ai/veris-cli/actions/workflows/test.yml)
[![Python](https://img.shields.io/pypi/pyversions/veris-cli.svg)](https://pypi.org/project/veris-cli/)

**Connect your existing agent to Veris simulations.**

The Veris CLI lets you package your agent and run it against simulation scenarios.

## Installation

```bash
uv tool install veris-cli
```

Or from source:
```bash
uv tool install git+https://github.com/veris-ai/veris-cli.git
```

## Quick Start

### 1. Login

```bash
# Browser-based Google login (recommended)
veris login

# Or with an API key directly (for CI/scripts)
veris login YOUR_API_KEY
```

This saves your credentials to `~/.veris/config.yaml`.

### 2. Submit Your Agent (Managed Onboarding)

```bash
cd ~/my-agent
veris env submit
```

This is the **only command you need** to onboard a new agent. It packages
your repo, uploads it to Veris, and lets the platform generate
`Dockerfile.sandbox` and `veris.yaml` for you. You'll receive an email
when your environment is `ready` — typically the same business day.

No prior `veris env create` is required. If there is no env yet, `env
submit` creates one (using `--name`, `--target`, or the current directory
name).

### 3. Set Any Secrets

```bash
veris env vars set OPENAI_API_KEY=sk-your-key --secret
```

You can set secrets at any point — before or after the env reaches `ready`.

### 4. Iterate After Release

Once your env is `ready`, pull the generated config and iterate locally:

```bash
veris env config pull   # Pull generated .veris/Dockerfile.sandbox + veris.yaml
veris env push          # Build and push a new image tag
```

Re-running `veris env push` after each code change is fast. Only re-run
`veris env submit` when something structural changes (new agent entry
point, new service dependency, etc.).

> **Stack flows** — agents with non-trivial integration requirements
> (e.g. nemoclaw) use `veris env create --stack <name>`. Run `veris env
> create` without `--stack` to see the picker, which lets you choose a
> registered stack or route into `veris env submit` (the default for
> plain agents).

### 5. Generate Scenarios

```bash
veris scenarios create --num 10
veris scenarios status <SET_ID> --watch
```

### 6. Run Simulations

```bash
veris simulations create --scenario-set-id <SET_ID>
veris simulations status <SIM_RUN_ID> --watch
```

### 7. Evaluate Results

```bash
veris evaluations create --sim-run-id <SIM_RUN_ID>
veris evaluations status <SIM_RUN_ID> <EVAL_RUN_ID> --watch
```

### 8. Generate Report

```bash
veris reports create <SIM_RUN_ID>
veris reports status <REPORT_ID> --watch
veris reports get <REPORT_ID> -o results.html
```

### Full Pipeline (One Command)

```bash
# Interactive — prompts for each step
veris run

# CI — all flags, markdown summary to stdout
veris run --scenario-set-id <SET_ID> --grader-id <GRADER_ID> --report
```

## Command Reference

### Top-Level Commands

```bash
veris login [API_KEY]              # Authenticate (browser or API key)
  --profile NAME                   # Profile to log in to (sets it active)
  --backend-url URL                # Custom backend URL
  --console-url URL                # Custom console URL
  --org ORG_ID                     # Scope to organization

veris run                          # Full pipeline: simulations → evaluations → reports
  --scenario-set-id ID             # Scenario set (prompts if omitted)
  --grader-id ID                   # Grader (prompts if omitted)
  --env-id ID                      # Environment (uses config if omitted)
  --target NAME                    # Target (auto-detected if only one)
  --image-tag TAG                  # Image tag (default: latest)
  --simulation-timeout N           # Timeout per sim in seconds
  --report                         # Generate HTML report after evaluation
```

### Environment (`veris env`)

```bash
veris env create --name NAME       # Scaffold .veris/ + register environment (NAME = env/target name)
  --agent-name NAME                # Agent display name (stored in veris.yaml agent.name)
veris env push [--tag TAG]         # Build and push image to Veris
  --target NAME                    # Target to push (auto-detected if only one)
veris env list                     # List environments
veris env delete ENV_ID            # Delete environment
```

### Active Target (`veris env targets`)

```bash
veris env targets get               # Show the active target
veris env targets set NAME          # Set the active target for this project
veris env targets list              # List configured targets (from veris.yaml)
veris env targets clear             # Clear the active target
```

### Environment Config (`veris env config`)

```bash
veris env config push              # Upload veris.yaml to backend (no image build)
  --file PATH                      # Custom veris.yaml path (default: .veris/veris.yaml)
  --env-id ID                      # Override environment
  --target NAME                    # Target to upload
```

### Environment Variables (`veris env vars`)

```bash
veris env vars set K=V [K=V ...]   # Set variables
  --secret                         # Mark as secret
  --env-id ID                      # Override environment
  --target NAME                    # Target to set variables for
veris env vars list                # List variables
  --target NAME
veris env vars rm KEY              # Remove a variable
  --target NAME
```

### Scenarios (`veris scenarios`)

```bash
veris scenarios create             # Generate scenario set + grader
  --num N                          # Number of scenarios (default: 5)
  --env-id ID                      # Environment
  --image-tag TAG                  # Image tag
  --prompt TEXT                    # Generate from a natural-language prompt
  --from-langfuse CURL             # Generate from Langfuse traces (- = stdin, @file)
veris scenarios add SET_ID         # Add more scenarios to an existing set
veris scenarios status SET_ID      # Check generation progress
  --watch                          # Poll until done
veris scenarios list               # List scenario sets (with grader column)
veris scenarios get SET_ID         # Open in console browser
veris scenarios delete SET_ID      # Delete scenario set
```

#### Generate from production traces (Langfuse)

Ground a scenario set on real agent traffic. Build a traces/sessions query on
Langfuse's API page, copy the curl it gives you (URL + `Authorization: Basic`
header), and pipe it in — **keys are read from the curl and never stored
locally**. A single-trace request (`/api/public/traces/{id}`) is the most
reliable; the list endpoints can time out on Langfuse Cloud and need a tight
date filter.

```bash
# stdin (no disk) — recommended:
pbpaste | veris scenarios create \
  --env-id env_xxx \
  --from-langfuse - \
  --num 5 \
  --prompt "what to focus on (acts as the trace source's instructions)"

# or from a file: --from-langfuse @/tmp/langfuse-curl.txt
```

> Trace sources are **admin-only** server-side — use a Veris admin/dev-internal
> profile. `veris scenarios add <set_id> --from-langfuse - ...` adds another
> trace source to an existing set.

### Simulations (`veris simulations`)

```bash
veris simulations create           # Create simulation run (interactive)
  --scenario-set-id ID             # Scenario set
  --env-id ID                      # Environment
  --simulation-timeout N           # Timeout per sim
  --image-tag TAG                  # Image tag
veris simulations status SIM_RUN_ID  # Run progress + sim list
  --watch                          # Poll until done
  --log                            # Append event stream
veris simulations list             # List simulation runs
  --status STATUS                  # Filter by status
  --env-id ID                      # Filter by environment
veris simulations cancel SIM_RUN_ID  # Cancel a run
```

### Evaluations (`veris evaluations`)

```bash
veris evaluations create           # Trigger grading (interactive)
  --sim-run-id ID                  # Simulation run
  --grader-id ID                   # Grader (pre-selected by scenario set)
veris evaluations status SIM_RUN_ID EVAL_RUN_ID  # Eval progress
  --watch                          # Poll until done
veris evaluations list [SIM_RUN_ID]  # List eval runs
                                   # Without args: all evals for current env
                                   # With arg: evals for specific run
veris evaluations get SIM_RUN_ID EVAL_RUN_ID  # Open in console browser
```

### Reports (`veris reports`)

```bash
veris reports create [SIM_RUN_ID]  # Trigger report generation
  --eval-run-id ID                 # Specific eval run
veris reports status REPORT_ID     # Report progress
  --watch                          # Poll until done
veris reports list                 # List reports
veris reports get REPORT_ID        # Download report HTML
  -o PATH                          # Output path
```

### Profiles (`veris profile`)

```bash
veris profile list                 # List all profiles
veris profile get                  # Show active profile settings
veris profile use [NAME]           # Set active profile (interactive)
veris profile delete NAME          # Remove a profile
veris profile login [API_KEY]      # Alias for 'veris login' (same options)
veris profile org set ORG_ID       # Set organization for active profile
veris profile org rm               # Remove organization from active profile
```

## CI/CD Integration

```yaml
name: Veris Simulation
on:
  pull_request:
    branches: [main]

jobs:
  simulate:
    runs-on: ubuntu-latest
    environment: veris-sim-ci
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - run: pip install veris-cli
      - name: Build & push
        env:
          VERIS_API_KEY: ${{ secrets.VERIS_API_KEY }}
        run: |
          veris login "$VERIS_API_KEY"
          veris env push --tag ${{ github.sha }}
      - name: Run pipeline
        run: veris run --image-tag ${{ github.sha }} --report > summary.md
      - uses: marocchino/sticky-pull-request-comment@v2
        with:
          path: summary.md
```

## How It Works

```
veris env create              → Scaffold .veris/ + register environment
veris env push                → Build and push agent image
veris scenarios create        → Generate scenarios + grader
veris simulations create      → Run agent against scenarios
veris evaluations create      → Grade simulation results
veris reports create          → Generate failure analysis report
```

Or use `veris run` to chain all steps interactively.

## Configuration Files

### `~/.veris/config.yaml`

Global config with named profiles:
```yaml
active_profile: default
profiles:
  default:
    api_key: vrs_abc123
    backend_url: https://sandbox.api.veris.ai
    console_url: https://console.veris.ai
  staging:
    api_key: vrs_staging
    backend_url: https://sandbox.api.veris.ai
    organization_id: org_abc123
```

### `.veris/config.yaml`

Project config (created by `veris env create`). Each target's `environment_id` is stored under the profile:
```yaml
active_target: my-cool-agent-env
profiles:
  default:
    targets:
      my-cool-agent-env:
        environment_id: env_abc123
        environment_name: my-cool-agent-env
```

### `.veris/veris.yaml`

Simulation configuration. Every target is a top-level key — the target name is the backend environment name:
```yaml
version: "1.0"

my-cool-agent-env:
  services:
    - name: postgres
      config:
        SCHEMA_PATH: /agent/schemas/schema.sql
    - name: slack
      dns_aliases:
        - slack.com
  persona:
    modality:
      type: http
      url: http://localhost:8008/chat
  agent:
    name: My Cool Agent
    code_path: /agent
    entry_point: uv run app
    port: 8008
    environment:
      DATABASE_URL: postgresql://postgres:postgres@localhost:5432/veris
```

When only one target is defined, all commands auto-select it.

## Multiple Targets (Monorepo)

A single repo can manage multiple agents/environments. Each `veris env create` adds a target whose name **is** the backend environment name.

### Creating targets

```bash
# First agent
veris env create --name my-cool-agent-env --agent-name "My Cool Agent"

# Second agent — appended to the same veris.yaml
veris env create --name customer-support-env --agent-name "Customer Support"
```

After two creates, `.veris/veris.yaml` looks like:
```yaml
version: "1.0"

my-cool-agent-env:
  services: [...]
  persona: ...
  agent:
    name: My Cool Agent
    ...

customer-support-env:
  services: [...]
  persona: ...
  agent:
    name: Customer Support
    ...
```

### Pushing and running

```bash
# With one target — auto-detected, no flag needed
veris env push

# With multiple targets — specify which one
veris env push --target customer-support-env
veris run --target customer-support-env
veris env vars set API_KEY=sk-... --target customer-support-env --secret
```

### Setting an active target

Avoid typing `--target` on every command:
```bash
veris env targets set customer-support-env
veris env push                    # pushes customer-support-env
veris run                         # runs customer-support-env
veris env targets get              # shows: customer-support-env
veris env targets list             # lists all targets from veris.yaml
veris env targets clear            # clears the active target
```

### Target resolution

| Priority | Source |
|----------|--------|
| 1 | `--target` flag |
| 2 | `active_target` in `.veris/config.yaml` |
| 3 | Auto-detect: sole target in `veris.yaml` |
| 4 | Error if ambiguous |

### Per-target Dockerfile

Targets can override the default Dockerfile:
```yaml
customer-support-env:
  dockerfile: .veris/Dockerfile.support
  agent:
    name: Customer Support
    ...
```

Falls back to `.veris/Dockerfile.sandbox` when not specified.

### Cross-profile behavior

When you switch profiles (e.g. from `dev` to `staging`) and push, the CLI detects that the target exists on another profile and offers to create it on the current one.

## Profiles

The CLI supports named profiles for managing multiple backend accounts. Each profile stores its own API key, backend URL, console URL, organization, and environment IDs.

```bash
# Login to a named profile (creates it if new, sets it active)
veris login --profile acme --org org_abc123

# Switch active profile
veris profile use acme

# Manage organization
veris profile org set org_abc123
veris profile org rm
```

Profile resolution: `active_profile` in config → `"default"`.

Existing flat configs (without `profiles` key) auto-migrate on first write.

## Development

```bash
git clone https://github.com/veris-ai/veris-cli.git
cd veris-cli
uv sync
uv run pytest
uv tool install --force -e .
```

## Troubleshooting

- **"No API key found"** — Run `veris login`
- **Image build fails** — Check build logs in the output. Ensure `.veris/Dockerfile.sandbox` is valid.
- **Image push fails** — Credentials are fetched automatically. Just retry.

## Support

- GitHub Issues: https://github.com/veris-ai/veris-cli/issues
- Email: developers@veris.ai

## License

MIT
