Metadata-Version: 2.4
Name: agentops-toolkit
Version: 0.1.4
Summary: AgentOps CLI for standardized evaluation workflows
License: MIT License
        
        Copyright (c) Microsoft Corporation.
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: typer<1.0,>=0.12
Requires-Dist: pydantic<3,>=2
Requires-Dist: ruamel.yaml<1.0,>=0.18
Requires-Dist: azure-ai-projects>=2.0.1
Dynamic: license-file

<h1 align="center">AgentOps Toolkit</h1>

<p align="center">
AgentOps CLI for evaluation, observability, and operational workflows for Microsoft Foundry Agents and Models.
</p>

<p align="center">
<a href="./LICENSE"><img alt="License: MIT" src="https://img.shields.io/badge/License-MIT-green.svg"/></a>
<img alt="Status: Preview" src="https://img.shields.io/badge/Status-Preview-orange.svg"/>
<img alt="Python 3.11+" src="https://img.shields.io/badge/Python-3.11%2B-3776AB.svg"/>
<img alt="CLI" src="https://img.shields.io/badge/CLI-Typer-5A67D8.svg"/>
<img alt="Built on Microsoft Foundry" src="https://img.shields.io/badge/Built%20on-Microsoft%20Foundry-0078D4.svg"/>
</p>

## Overview

AgentOps Toolkit is a CLI built on Microsoft Foundry that standardizes evaluation and operational workflows for AI agents and models, helping teams run, monitor, and automate AgentOps processes.

The project enables:

- Consistent local and CI execution of agent evaluations
- Reusable evaluation policies through bundles
- Operational observability through tracing, monitoring, and run inspection
- Stable machine-readable outputs for automation
- Human-readable reports for PR reviews and quality gates

Operational capabilities include:

- Standardized evaluation workflows
- Run history and result inspection
- Tracing and observability
- Monitoring (dashboards and alerts)
- CI/CD automation
- Operational reporting and analysis

Core outputs:

- `results.json` (machine-readable)
- `report.md` (human-readable)

Exit code contract:

- `0` execution succeeded and all thresholds passed
- `2` execution succeeded but one or more thresholds failed
- `1` runtime or configuration error

## Quickstart

This section is structured for demos and onboarding, so you can present the project flow end-to-end in a few minutes.

<p align="center">
<img alt="Quickstart demo: agentops init and eval run" src="./media/quickstart.gif"/>
</p>

### 1) Install

```bash
python -m venv .venv
# activate your venv in the current shell
python -m pip install -U pip
python -m pip install agentops-toolkit
```

### 2) Initialize Workspace

```bash
agentops init
```

Generated structure:

```text
.agentops/
├── config.yaml
├── run.yaml
├── run-rag.yaml
├── run-agent.yaml
├── .gitignore
├── bundles/
│   ├── model_direct_baseline.yaml
│   ├── rag_retrieval_baseline.yaml
│   └── agent_tools_baseline.yaml
├── datasets/
│   ├── smoke-model-direct.yaml
│   ├── smoke-rag.yaml
│   └── smoke-agent-tools.yaml
├── data/
│   ├── smoke-model-direct.jsonl
│   ├── smoke-rag.jsonl
│   └── smoke-agent-tools.jsonl
└── results/
```

### 3) Configure Foundry Endpoint

PowerShell:

```powershell
$env:AZURE_AI_FOUNDRY_PROJECT_ENDPOINT = "https://<resource>.services.ai.azure.com/api/projects/<project>"
```

Bash/zsh:

```bash
export AZURE_AI_FOUNDRY_PROJECT_ENDPOINT="https://<resource>.services.ai.azure.com/api/projects/<project>"
```

Authentication uses `DefaultAzureCredential`:
- local: `az login`
- CI/CD: service principal env vars
- Azure-hosted: managed identity

### 4) Choose Scenario Run Config

Starter run files created by `agentops init`:
- `.agentops/run.yaml` (default model-direct)
- `.agentops/run-rag.yaml` (agent + rag baseline)
- `.agentops/run-agent.yaml` (agent + tools baseline)

Important:
- Replace placeholders (`backend.model`, `backend.agent_id`) with values that exist in your Foundry project.
- There is no universal deployment name guaranteed across all Foundry projects/regions.

### 5) Run Evaluation

```bash
agentops eval run
```

Or run a specific scenario file:

```bash
agentops eval run --config .agentops/run-rag.yaml
agentops eval run --config .agentops/run-agent.yaml
```

Default behavior:
- input config: `.agentops/run.yaml`
- output location: timestamped folder under `.agentops/results/`
- latest pointer: `.agentops/results/latest/`

### 6) Regenerate Report (Optional)

```bash
agentops report
```

Default input:
- `.agentops/results/latest/results.json`

## Evaluation Scenarios

Starter bundles created by `agentops init`:

| Bundle | Evaluators | Typical use |
|---|---|---|
| `model_direct_baseline` (default) | `SimilarityEvaluator` + `avg_latency_seconds` | Model-direct QA checks |
| `rag_retrieval_baseline` | `GroundednessEvaluator` + `avg_latency_seconds` | RAG groundedness checks |
| `agent_tools_baseline` | `TaskCompletionEvaluator` + `ToolCallAccuracyEvaluator` + `avg_latency_seconds` | Agent-with-tools baseline |

`datasets/` stores YAML dataset definitions.
`data/` stores JSONL rows referenced by dataset definitions.

## Commands

### Command Line Reference

| Command | Description | Status |
|---|---|---|
| `agentops --version` | Show installed version | ✅ |
| `agentops init [--dir DIR]` | Scaffold project workspace and starter files | ✅ |
| `agentops eval run` | Evaluate a dataset against a bundle | ✅ |
| `agentops eval compare --runs ID1,ID2` | Compare two past runs | ✅ |
| `agentops run list\|show` | List or inspect past runs | 🚧 |
| `agentops run view <id> [--entry N]` | Deep run inspection | 🚧 |
| `agentops report` | Regenerate `report.md` from `results.json` | ✅ |
| `agentops report show\|export` | View/export reports | 🚧 |
| `agentops bundle list\|show` | Browse bundle catalog | 🚧 |
| `agentops dataset validate\|describe\|import` | Dataset utilities | 🚧 |
| `agentops config cicd` | Generate GitHub Actions workflow for CI evaluation | ✅ |
| `agentops config validate\|show` | Config validation and inspection | 🚧 |
| `agentops trace init` | Tracing setup | 🚧 |
| `agentops monitor setup\|dashboard\|alert` | Monitoring operations | 🚧 |
| `agentops model list` | List Foundry model deployments | 🚧 |
| `agentops agent list` | List Foundry agents | 🚧 |

Implemented command usage:

```bash
agentops --version
agentops init [--dir <dir>]
agentops eval run [--config <path>] [--output <dir>] [--format md|html|all]
agentops eval compare --runs ID1,ID2 [--output <dir>] [--format md|html|all]
agentops report [--in <results.json>] [--out <report.md>] [--format md|html|all]
agentops config cicd [--force] [--dir <path>]
```

For planned commands, the CLI returns a friendly message indicating the command is planned but not implemented in this release.

## Project Structure

High-level code layout:

- `src/agentops/cli/` command entrypoints (Typer)
- `src/agentops/services/` orchestration workflows
- `src/agentops/backends/` execution engines (`foundry`, `subprocess`)
- `src/agentops/core/` schemas, thresholds, and report generation
- `src/agentops/templates/` starter workspace assets
- `tests/unit/` and `tests/integration/` automated tests

## Documentation

- Architecture and request flow: [docs/how-it-works.md](docs/how-it-works.md)
- Foundry agent tutorial: [docs/tutorial-basic-foundry-agent.md](docs/tutorial-basic-foundry-agent.md)
- Model-direct tutorial: [docs/tutorial-model-direct.md](docs/tutorial-model-direct.md)
- RAG tutorial: [docs/tutorial-rag.md](docs/tutorial-rag.md)
- Baseline comparison tutorial: [docs/tutorial-baseline-comparison.md](docs/tutorial-baseline-comparison.md)
- Copilot skills installation: [docs/tutorial-copilot-skills.md](docs/tutorial-copilot-skills.md)
- Built-in evaluator notes: [docs/foundry-evaluation-sdk-built-in-evaluators.md](docs/foundry-evaluation-sdk-built-in-evaluators.md)
- CI/CD setup guide: [docs/ci-github-actions.md](docs/ci-github-actions.md)

## GitHub Copilot Skills

AgentOps publishes Copilot skills that teach GitHub Copilot how to use the evaluation CLI correctly. Install them from this repository to get AI-assisted guidance for running evaluations, investigating regressions, and triage workflows.

### Available Skills

| Skill | Description |
|---|---|
| `agentops-run-evals` | Guides evaluation workflow — init, run, report, compare |
| `agentops-investigate-regression` | Regression investigation — metric deltas, threshold flips, actionable checks |
| `agentops-observability-triage` | Observability and triage — current capabilities vs planned features |

### Installation

Skills are distributed from this GitHub repository. Install them in VS Code:

1. Open **VS Code** with **GitHub Copilot Chat** enabled.
2. Use the Copilot skill install command and point to this repository:
   - Source: `Azure/agentops`
     - Skills are located under `plugins/agentops/skills/`
3. Once installed, Copilot will automatically use the skills when you ask about AgentOps evaluation, regressions, or observability.

Alternatively, you can copy the skill files manually:
```bash
# Copy skills to your user-level skills directory
cp -r plugins/agentops/skills/* ~/.agents/skills/
```

### For Repository Contributors

If you're working inside this repo, the skills under `.github/skills/` are automatically available to Copilot when the repository is your active workspace.

## Contributing

See [CONTRIBUTING.md](CONTRIBUTING.md) for architecture rules, testing expectations, and contribution workflow.
