Metadata-Version: 2.4
Name: agentops-accelerator
Version: 0.3.0
Summary: Release readiness gates and evidence for Microsoft Foundry agents
License: MIT License
        
        Copyright (c) Microsoft Corporation.
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: typer<1.0,>=0.12
Requires-Dist: pydantic<3,>=2
Requires-Dist: ruamel.yaml<1.0,>=0.18
Requires-Dist: azure-ai-projects>=2.0.1
Provides-Extra: mcp
Requires-Dist: mcp<2,>=1.0; extra == "mcp"
Provides-Extra: foundry
Requires-Dist: azure-ai-evaluation>=1.0; extra == "foundry"
Requires-Dist: azure-identity>=1.17; extra == "foundry"
Requires-Dist: azure-monitor-opentelemetry>=1.6; extra == "foundry"
Requires-Dist: pandas>=2.0; extra == "foundry"
Provides-Extra: agent
Requires-Dist: fastapi>=0.110; extra == "agent"
Requires-Dist: uvicorn[standard]>=0.30; extra == "agent"
Requires-Dist: httpx>=0.27; extra == "agent"
Requires-Dist: cryptography>=42; extra == "agent"
Requires-Dist: markdown>=3.6; extra == "agent"
Requires-Dist: azure-monitor-query>=1.3; extra == "agent"
Requires-Dist: azure-monitor-opentelemetry>=1.6; extra == "agent"
Requires-Dist: azure-identity>=1.17; extra == "agent"
Requires-Dist: azure-mgmt-cognitiveservices>=13.5; extra == "agent"
Requires-Dist: azure-mgmt-monitor>=6.0; extra == "agent"
Dynamic: license-file

<h1 align="center">AgentOps Accelerator</h1>

<p align="center">
Answer the release question for Microsoft Foundry agents: can we ship it, and where is the proof?
</p>

<p align="center">
<a href="https://pypi.org/project/agentops-accelerator/"><img alt="PyPI" src="https://img.shields.io/pypi/v/agentops-accelerator.svg?label=PyPI&color=blue"/></a>
<a href="https://marketplace.visualstudio.com/items?itemName=AgentOpsAccelerator.agentops-accelerator"><img alt="VS Code Extension" src="https://img.shields.io/badge/VS%20Code-Extension-007ACC.svg?logo=visualstudiocode"/></a>
<a href="https://github.com/Azure/agentops/actions/workflows/ci.yml"><img alt="CI" src="https://github.com/Azure/agentops/actions/workflows/ci.yml/badge.svg?branch=develop"/></a>
<a href="https://github.com/Azure/agentops/actions/workflows/release.yml"><img alt="Release" src="https://github.com/Azure/agentops/actions/workflows/release.yml/badge.svg"/></a>
<a href="https://github.com/Azure/agentops"><img alt="Status: Preview" src="https://img.shields.io/badge/Status-Preview-orange.svg"/></a>
<br/>
<a href="https://www.python.org/downloads/"><img alt="Python 3.11+" src="https://img.shields.io/badge/Python-3.11%2B-3776AB.svg"/></a>
<a href="https://typer.tiangolo.com/"><img alt="CLI: Typer" src="https://img.shields.io/badge/CLI-Typer-5A67D8.svg"/></a>
<a href="https://learn.microsoft.com/azure/ai-foundry/"><img alt="Built on Microsoft Foundry" src="https://img.shields.io/badge/Built%20on-Microsoft%20Foundry-0078D4.svg"/></a>
<a href="https://github.com/Azure/agentops/blob/main/LICENSE"><img alt="License: MIT" src="https://img.shields.io/badge/License-MIT-green.svg"/></a>
</p>

## Overview

AgentOps Accelerator helps teams turn Foundry agent work into a clear release
decision. Foundry is the agent control plane; AgentOps turns Foundry signals and
repo checks into repeatable gates, Doctor readiness, release evidence, and
trace-driven regression loops.

The project enables:

- Local and CI execution for release gates
- Foundry prompt agent, Foundry hosted endpoint, HTTP/JSON agent, and raw model targets
- Auto-selected evaluators for RAG, tools, and model quality
- Stable `results.json` for automation
- PR-friendly `report.md`
- Baseline comparison for regression detection
- Doctor checks for repo, CI/CD, telemetry, landing zones, and Foundry setup
- Release evidence packs for promotion review
- Trace promotion into regression datasets
- Cockpit navigation for AgentOps, Foundry, and Azure Monitor

## AgentOps and Microsoft Foundry

Foundry and AgentOps are designed to meet at the release boundary. Foundry is
where teams create, deploy, run, observe, and investigate agents. AgentOps is
the repo-side operating layer that turns those signals into a repeatable
ship/no-ship workflow.

| Moment | Foundry / Azure does | AgentOps adds |
|---|---|---|
| Build and version | Foundry portal, Foundry SDK/Toolkit, `microsoft-foundry` skill, azd | Pins the exact candidate in `agentops.yaml` and generates the PR/release gate around it |
| Evaluate and compare | Foundry Evaluations and official CI actions/extensions | Keeps datasets and thresholds in the repo, records evidence, and provides local/fallback runs for non-prompt targets |
| Observe and investigate | Foundry Monitor, Traces, Azure Monitor, App Insights | Surfaces deep links, telemetry readiness, Doctor findings, and Cockpit navigation |
| Decide release | Branch protection, environments, approvals | Packages `evidence.json` / `evidence.md` for promotion review |
| Improve from production | Production traces and Foundry datasets | Promotes reviewed trace learnings into regression candidates |

The rhythm is simple: build and operate the agent in Foundry, keep the release
contract in the repo, and let AgentOps connect the two into a clean review loop.

Core outputs:

- `results.json` (machine-readable)
- `report.md` (human-readable)
- `evidence.json` / `evidence.md` (from `agentops doctor --evidence-pack`)

Exit code contract:

- `0` execution succeeded and all thresholds passed
- `2` execution succeeded but one or more thresholds failed
- `1` runtime or configuration error

## Quickstart

### 1) Install

```powershell
python -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install -U pip
python -m pip install --upgrade "agentops-accelerator[foundry] @ git+https://github.com/Azure/agentops.git@main"
```

This installs the current AgentOps source from GitHub. After the next package
release, you can switch the install line back to `agentops-accelerator[foundry]`
from PyPI.

### 2) Bootstrap

```powershell
agentops init
```

This writes a single `agentops.yaml` at the project root and an
AgentOps-managed workspace under `.agentops/` for seed data, run history, and
generated evidence. It is not a second `.foundry/` project directory.

### 3) Configure your agent

Pick one of these forms for the `agent:` field - AgentOps classifies the target automatically:

```yaml
agent: "my-rag:3"                          # Foundry prompt agent (name:version)
agent: "https://...services.ai.azure.com/.../agents/<id>"  # Foundry hosted endpoint
agent: "https://api.example.com/chat"      # any HTTP/JSON agent (ACA, AKS, custom)
agent: "model:gpt-4o"                       # raw Foundry model deployment
```

AgentOps supports both Foundry Prompt Agents and Hosted Agents as evaluation
and readiness targets. Create and deploy them with Foundry tools, then reference
the published candidate in `agentops.yaml`.

For the smoke dataset, create a Foundry prompt agent such as
`agentops-smoke` and publish it with instructions that copy exact-answer
requests verbatim:

```text
If the user message starts with "Answer with exactly this sentence:",
copy only the sentence after that prefix. Do not add greetings,
markdown, citations, caveats, or explanations.
```

Evaluators come from dataset shape: `context` triggers RAG checks;
`tool_calls` / `tool_definitions` trigger tool-use checks. Minimal config:

```yaml
version: 1
agent: "agentops-smoke:2"  # Foundry saves the first published version as v2
dataset: .agentops/data/smoke.jsonl
```

### 4) Run

```powershell
az login
$env:AZURE_AI_FOUNDRY_PROJECT_ENDPOINT = "https://<resource>.services.ai.azure.com/api/projects/<project>"
$env:AZURE_OPENAI_ENDPOINT = "https://<openai-resource>.openai.azure.com"
$env:AZURE_OPENAI_DEPLOYMENT = "gpt-4o-mini"
agentops eval analyze
agentops eval run
agentops doctor --evidence-pack
```

For Foundry targets, use either `project_endpoint:` in `agentops.yaml` or
`AZURE_AI_FOUNDRY_PROJECT_ENDPOINT`. Config wins when both are set.

Outputs land in `.agentops/results/latest/`:

- `results.json` - machine-readable (versioned, stable schema)
- `report.md` - human-readable, PR-friendly

Release evidence lands in `.agentops/release/latest/`:

- `evidence.json` - machine-readable production-readiness projection
- `evidence.md` - PR/release summary

Capture the first successful run as a baseline:

```powershell
New-Item -ItemType Directory -Force .agentops\baseline | Out-Null
Copy-Item .agentops\results\latest\results.json .agentops\baseline\results.json
```

To see a visible comparison, publish a new agent version with a prompt
that paraphrases instead of copying exact-answer requests, update
`agentops.yaml` to that new `name:version`, and compare against the
baseline:

```powershell
agentops eval run --baseline .agentops/baseline/results.json
```

The report grows a `Comparison vs Baseline` section with per-metric deltas.

---

## Commands

Install optional extras as needed: `[foundry]` for eval runtime, `[agent]` for
Doctor/Cockpit, and `[mcp]` for MCP.

- `agentops --version` - show installed version.
- `agentops init` - bootstrap config and seed data.
- `agentops eval analyze` - check eval readiness.
- `agentops eval run [--baseline PATH]` - run an evaluation.
- `agentops eval promote-traces --source FILE [--apply]` - promote traces.
- `agentops report generate` - regenerate `report.md`.
- `agentops workflow analyze` - recommend CI/CD shape.
- `agentops workflow generate` - generate CI/CD workflows.
- `agentops skills install` - install Copilot or Claude skills.
- `agentops mcp serve` - start the MCP server.
- `agentops doctor [--evidence-pack]` - run readiness checks.
- `agentops cockpit` - open the local Cockpit.
- `agentops agent serve` - serve Doctor as a Copilot Extension.

## AgentOps Cockpit

`agentops cockpit` opens a localhost command center for the current workspace.
It combines eval history, Doctor findings, workflow status, and links to the
matching Foundry and Azure Monitor views.

Cockpit sections, in display order:

- **Foundry connection** - project, tenant, agent, App Insights.
- **Foundry launchpad** - links for the agent, project, and telemetry.
- **Observability readiness** - tracing, evals, red team, alerts.
- **AgentOps Doctor** - latest Doctor findings.
- **Eval gate summary** - local and CI gate history.
- **Quality gate summary** - score trends and regressions.
- **Production signal** - App Insights health snapshot.
- **CI/CD Pipelines** - GitHub Actions status.
- **Next actions** - contextual recommendations.

## Documentation

- [Prompt Agent quickstart](docs/tutorial-prompt-agent-quickstart.md) - use this when the Foundry target is `agent: name:version`.
- [Hosted Agent quickstart](docs/tutorial-hosted-agent-quickstart.md) - use this when the target is a Foundry hosted or HTTP endpoint URL.
- [End-to-end workshop](docs/tutorial-end-to-end.md) - complete Foundry + AgentOps journey: create, debug, evaluate, release, observe, red-team follow-through, and trace regression.
- [Core concepts](docs/concepts.md)
- [How it works](docs/how-it-works.md)
- [Doctor explained](docs/doctor-explained.md)
- [CI/CD with GitHub Actions](docs/ci-github-actions.md)
- [Built-in evaluator reference](docs/foundry-evaluation-sdk-built-in-evaluators.md)
- [Release process](docs/release-process.md)

## Contributing

See [CONTRIBUTING.md](CONTRIBUTING.md) for architecture rules, testing, and contribution flow.
