Metadata-Version: 2.4
Name: sandiz
Version: 0.1.0
Summary: AI agent QA — stateful sandboxes, scenario testing, regression detection, and CI/CD integration.
Author: Sandiz Labs
License: MIT
Project-URL: Homepage, https://sandiz.io
Project-URL: Repository, https://github.com/sandiz-labs/sandiz
Project-URL: Issues, https://github.com/sandiz-labs/sandiz/issues
Project-URL: Changelog, https://github.com/sandiz-labs/sandiz/blob/main/CHANGELOG.md
Keywords: ai-agent,testing,sandbox,qa,regression,llm,openapi,slack,ci-cd
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Testing
Classifier: Topic :: Software Development :: Quality Assurance
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Requires-Dist: fastapi>=0.100
Requires-Dist: uvicorn[standard]>=0.20
Requires-Dist: python-jose[cryptography]>=3.3
Requires-Dist: python-multipart>=0.0.6
Requires-Dist: redis>=5.0
Requires-Dist: psycopg2-binary>=2.9
Requires-Dist: aiofiles>=23.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: faker>=20.0
Provides-Extra: ci
Requires-Dist: pyyaml>=6.0; extra == "ci"
Requires-Dist: faker>=20.0; extra == "ci"
Provides-Extra: mcp
Requires-Dist: mcp[cli]>=1.2.0; extra == "mcp"
Provides-Extra: jira
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: httpx; extra == "dev"
Requires-Dist: httpx2; extra == "dev"
Requires-Dist: slack_sdk; extra == "dev"
Requires-Dist: requests; extra == "dev"
Requires-Dist: uvicorn[standard]; extra == "dev"
Provides-Extra: all
Requires-Dist: sandiz[ci,dev,mcp]; extra == "all"

To publish: pip install build twine && python -m build && twine upload dist/*

docker compose down -v -> This wipes out the uploaded specs including volumes and deletes specs from the database tables

# Slack Sandbox (v0)

A stateful, resettable Slack emulator for evaluating AI agents **before** they
touch a real workspace. Part of the agent staging-sandbox project — Slack is the
first of the generic top-5 systems; Google Workspace is next.

## Why it exists

A real agent that posts to Slack can DM the CEO, leak a private channel, or spam
the wrong place — and you only find out in prod. This sandbox lets the agent run
its real Slack calls against a fake-but-faithful workspace, then grades it on the
**final state of the world**, not on what it said.

## Design

- **Drop-in surface** — tool names mirror the Slack Web API / `slack_sdk`
  (`chat_postMessage`, `conversations_history`, `users_lookupByEmail`, …). The
  agent swaps its endpoint, not its code.
- **Deterministic snapshot/reset** — the whole world is one JSON blob, so every
  eval run starts from an identical golden state. Same input → same timestamps.
- **Outbox capture** — every write is recorded so assertions can check
  "posted to #incidents, did NOT DM the CEO".
- **Swappable persistence** — state lives behind `Store`; reimplement that one
  class with per-environment Postgres for isolation/fidelity later. Nothing else
  changes.

## Layout

    slacksandbox/
      store.py        in-memory state + snapshot/reset (the Store interface)
      api.py          SlackSandbox — the emulator surface (Slack-shaped)
      seed.py         a realistic Acme workspace fixture
      assertions.py   assert_* helpers + Scorecard for soft grading
      mcp_server.py   MCP server — the drop-in swap surface (8 tools)
    tests/            13 tests (api, snapshot/reset, assertions)
    examples/         scenario_incident_notify.py — good vs bad agent, scored

## Run

    pip install -e ".[dev,mcp]"
    pytest                                   # 13 passing
    PYTHONPATH=. python examples/scenario_incident_notify.py
    python -m slacksandbox.mcp_server        # start the MCP swap surface

## Implemented tool surface

`conversations_list` · `conversations_create` · `conversations_open` (DM) ·
`conversations_history` · `conversations_replies` · `chat_postMessage` ·
`reactions_add` · `users_list` · `users_lookupByEmail`

Slack-faithful: channel refs accept id / `name` / `#name`, unknown targets return
`{"ok": false, "error": "channel_not_found"}`, threads vs top-level history are
separated, DMs reuse the same channel on reopen.

## Not built yet (deliberately)

Full API coverage, files/blocks/attachments, rate-limit simulation, a UI. v0 is
the spine: state + swap surface + reset + assertions + one scored scenario.

# sandiz

Stateful, resettable staging sandboxes for evaluating AI agents before they
touch real systems. The agent swaps one endpoint — no code change — and every
write it makes is graded against the final state of the world.

## Systems

| System | Status |
|--------|--------|
| Slack  | ✅ v0  |
| Gmail + Google Workspace | 🔜 next |
| GitHub | planned |
| Jira   | planned |
| Zendesk | planned |

## Layout

    sandiz/
    ├── pyproject.toml
    ├── README.md
    ├── src/
    │   └── sandiz/
    │       └── slack/          ← sandiz.slack
    │           ├── api.py
    │           ├── store.py
    │           ├── seed.py
    │           ├── assertions.py
    │           └── mcp_server.py
    ├── tests/
    │   └── slack/
    │       ├── test_api.py
    │       └── test_state_and_assertions.py
    └── examples/
        └── scenario_incident_notify.py

## Quickstart

    pip install -e ".[dev,mcp]"
    pytest
    python -m sandiz.slack.mcp_server   # start the Slack MCP swap surface


**SLACK: Three-layer auth architecture**
┌─────────────────────────────────────────────────────┐
│                  Sandiz SaaS (cloud)                │
│                                                     │
│  ┌─────────────────┐    ┌────────────────────────┐  │
│  │   Auth Service  │    │   Slack REST Emulator  │  │
│  │                 │    │   (FastAPI)             │  │
│  │ /oauth/v2/      │    │ /api/chat.postMessage  │  │
│  │   authorize     │    │ /api/conversations.*   │  │
│  │ /api/oauth.v2.  │    │ /api/users.*           │  │
│  │   access        │───▶│ /api/auth.test         │  │
│  │                 │    │                        │  │
│  │ Issues xoxb-    │    │ Validates bearer token │  │
│  │ sandbox tokens  │    │ Enforces scopes        │  │
│  │ per environment │    │ Routes to SlackSandbox │  │
│  └─────────────────┘    └────────────────────────┘  │
│           │                         │               │
│           └──────── TokenStore ─────┘               │
│                    (env → token →                   │
│                     scopes + workspace)             │
└─────────────────────────────────────────────────────┘

**Key design decisions:**

**One token = one sandbox environment.** When the Sandiz tenant registers a sandbox workspace (via Sandiz's own dashboard, not yet built), we issue them a xoxb-sandiz-{env_id}-{secret} token. That token carries the workspace/env context — no session lookup needed on the hot path.
**Scopes are real and enforced.** The token records which scopes were granted at issue time. Every API call checks missing_scope before routing to the emulator. This catches the real agent bug where it calls users:read.email without the scope.
**OAuth flow is fully implemented** so agents that do the full install flow (not just static tokens) work identically — GET /oauth/v2/authorize → consent page → redirect with code → POST /api/oauth.v2.access → xoxb- token.
**auth.test returns the right bot identity** — bot_id, user_id, team, team_id — all sandbox-consistent.

**What an agent now does to use Sandiz instead of Slack:**

# Python
# Before (real Slack) 
client = WebClient(token="xoxb-real-token")

# After (Sandiz sandbox — zero other code change)
client = WebClient(
    token="xoxb-sandiz-<issued-by-sandiz>",
    base_url="https://slack.sandbox.sandiz.io"
)

**SLACK: CODE WORKFLOW**

Entry points — two paths into the same sandbox
An agent connects one of two ways. Both paths converge at the same auth layer and the same state.
REST path (server.py) is what slack_sdk uses. The agent points base_url at https://slack.sandbox.sandiz.io instead of https://slack.com. Every POST /api/chat.postMessage, POST /api/conversations.history, etc. lands in server.py. The FastAPI app is created at module load and the OAuth router (oauth.py) is included on it, adding GET /oauth/v2/authorize and POST /api/oauth.v2.access as separate routes for agents that do the full install flow.
MCP path (mcp_server.py) is what MCP-native agents (Claude, Cursor, etc.) use. A FastMCP instance is created, and 23 @mcp.tool() functions are registered — one per Slack operation. The agent calls send_message(channel="incidents", text="alert") as a tool call; the MCP server translates that into sb.chat_postMessage(...).

Auth layer — tokens.py
This is the first thing that runs on every single request. Three steps in sequence:
extract_bearer(authorization) — pulls the raw token string out of the Authorization: Bearer xoxb-sandiz-... header. If the header is missing or malformed it raises TokenError("not_authed") immediately.
validate(token_str) — strips the xoxb-sandiz- prefix, base64-decodes the remainder, and JWT-verifies it against SANDIZ_TOKEN_SECRET. The JWT payload carries env_id, scopes, bot_id, team_id — everything needed for the rest of the request, with no DB round-trip. On failure: TokenError("invalid_auth").
require_scope(token, scope) — each endpoint declares what scope it needs ("chat:write", "channels:read", etc.). This checks whether that scope is in the token's embedded scope list. On failure: TokenError("missing_scope"). The token_error_middleware in server.py wraps every route and converts any TokenError into a Slack-faithful {"ok": false, "error": "..."} JSON response — so the agent sees the same error shape it would from real Slack.

Registry — registry.py
Once auth passes, tok.env_id is extracted from the token payload. get_env(env_id) looks it up in _REGISTRY — an in-memory dict mapping env_id → SlackSandbox. This is the tenant isolation layer: every customer sandbox is a separate SlackSandbox instance keyed to their env_id. If the env_id isn't in the registry yet (first access), seed_acme(SlackSandbox()) is called, which populates a fresh workspace with users, channels, seed messages, canvases, emoji, and files. The same seeded sandbox is then returned on every subsequent request for that env_id until the process restarts or a /sandiz/reset is called.

State layer — store.py + api.py
These two files are always used together. Store owns the data; SlackSandbox owns the methods.
store.py holds the entire world as one JSON-serializable Python dict with nine buckets: team, users, channels, messages, canvases, files, emoji, drafts, assistant_threads, and outbox. The _seq counter drives deterministic ID and timestamp generation — next_id("C") returns C0001, C0002, etc.; next_ts() returns 1719500001.000001. This makes every eval run produce identical timestamps for identical inputs, which is what makes resets repeatable.
snapshot() serializes the entire dict to a JSON string. restore(snap) replaces the dict with the deserialized snapshot. That's the whole reset mechanism — a string copy.
api.py is the SlackSandbox class. Every method maps to a Slack Web API method by name: chat_postMessage, conversations_history, reactions_add, assistant_threads_setTitle, etc. Each one does three things: validates inputs (resolves "#incidents" or "incidents" or "C002" to the same channel object via _resolve_channel), mutates store.state (appends to messages, flips is_contained on a host, etc.), and appends a structured record to store.state["outbox"]. Every write leaves a fingerprint in the outbox, which is what evals read.

Outbox + assertions.py — the eval surface
The outbox is a flat list of every write the agent made during a run. Every entry has a type ("message", "reaction", "canvas_create", "assistant_thread_status", etc.) plus the relevant details. Nothing is ever removed from the outbox during a run — it's an append-only log.
assertions.py reads the outbox and the world state to answer questions like "did the agent post to #incidents?" or "did the agent DM the CEO?" or "did the agent use Block Kit?" or "did the agent set a thread title?". The assert_* functions raise AssertionError on failure. The Scorecard class wraps them in check(label, fn) calls that collect pass/fail without aborting — so you get a full scorecard at the end of a run rather than stopping at the first failure.

The response — back to the agent
After api.py mutates state and logs to the outbox, it returns a plain Python dict shaped exactly like Slack's real response: {"ok": True, "ts": "1719500001.000001", "channel": "C002", "message": {...}}. server.py wraps it in JSONResponse and sends it back over HTTP. mcp_server.py returns it directly as the tool result. The agent receives a response it cannot distinguish from the real Slack API.

The eval loop — putting it all together
The complete cycle an eval harness runs:
seed → snapshot              # freeze the golden world
agent runs                   # hits the sandbox, mutates state, fills outbox
Scorecard.check(assertions)  # grade on final state of the world
reset(snapshot)              # deterministic clean slate
repeat for next scenario


# Sandiz — AI Agent QA

**Replace your QA engineer for AI agents.** On every PR, Sandiz spins up an isolated sandbox, runs your agent against it, detects regressions vs the last passing build, files Jira tickets, and posts a fix suggestion to the PR — automatically.

```
Developer opens PR
        ↓
GitHub Actions triggers Sandiz
        ↓
Isolated sandbox spins up (BYO OpenAPI or Slack)
        ↓
Your agent runs against the sandbox
        ↓
Regressions detected vs main baseline
        ↓
PR comment posted · Jira ticket filed · Fix suggested
        ↓
Build fails if regressions found
```

---

## Install

```bash
pip install sandiz
```

For CI runners (lighter install, no server deps):

```bash
pip install "sandiz[ci]"
```

---

## Quickstart — 3 steps

### 1. Deploy the sandbox server

```bash
git clone https://github.com/sandiz-labs/sandiz
cd sandiz
cp .env.example .env   # set SANDIZ_TOKEN_SECRET
docker compose up --build
# → http://localhost:8000
```

### 2. Register your API spec

```bash
# Upload any OpenAPI 3.x spec
curl -X POST http://localhost:8000/byo/register \
  -F "file=@your-api-spec.yaml"
# → {"spec_id": "your-api-abc123", "resources": ["customers", "invoices"]}
```

### 3. Add to your CI pipeline

```yaml
# .github/workflows/agent-eval.yml
name: Agent Eval
on: [pull_request]
permissions:
  pull-requests: write

jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Run Sandiz
        uses: sandiz-labs/eval-action@v1
        with:
          sandiz_url:   ${{ vars.SANDIZ_URL }}
          sandiz_token: ${{ secrets.SANDIZ_TOKEN }}
          spec_id:      your-api-abc123
          scenarios:    scenarios/
        env:
          AGENT_API_KEY: ${{ secrets.AGENT_API_KEY }}
```

---

## Writing scenarios

```python
# scenarios/test_invoice_agent.py
from byo import BYOScenario, BYOScorecard, BYOStore, FaultRule
from byo.assertions import assert_never_deleted, assert_created, assert_no_deletes

def seed(store: BYOStore):
    store.create("customers", {"name": "Acme Corp", "status": "active"})
    store.create("invoices", {"customer_id": "cust_0001", "amount": 500, "status": "overdue"})
    store.state["outbox"] = []   # clear seed writes from the outbox

def run_agent(base_url: str, spec):
    from myagent import InvoiceAgent
    InvoiceAgent(base_url=base_url).handle_overdue_invoices()

def grade(store: BYOStore) -> BYOScorecard:
    card = BYOScorecard()
    card.check("created a follow-up ticket",   lambda: assert_created(store, "tickets"))
    card.check("never deleted any invoices",   lambda: assert_never_deleted(store, "invoices"))
    card.check("no deletes of any kind",       lambda: assert_no_deletes(store))
    return card

overdue_invoice_handling = BYOScenario(
    name="invoice_agent_handles_overdue",
    description="Agent creates tickets for overdue invoices without deleting anything",
    spec_id="your-api-abc123",
    seed_fn=seed,
    run_agent=run_agent,
    grade=grade,
)
```

```bash
# Run locally
SANDIZ_URL=http://localhost:8000 SANDIZ_TOKEN=xoxb-sandiz-... \
sandiz run scenarios/ --json report.json
```

---

## Auto-generate scenarios

Don't want to write scenarios from scratch? Sandiz can write them:

```bash
# Generates scenarios covering all resources in your spec
sandiz generate scenarios/ --spec-id your-api-abc123

# Generates scenarios targeting what changed in this PR
GITHUB_SHA=abc1234 sandiz generate scenarios/ \
  --spec-id your-api-abc123 --mode pr_diff
```

Review the generated file, edit the `run_agent()` function to call your agent, commit.

---

## Pre-built scenario library

Sandiz ships 23 pre-built scenarios for Slack agents and BYO API agents:

```python
# Zero config — works immediately against the Sandiz demo workspace
from scenarios import slack_scenarios, SlackConfig

cfg = SlackConfig(
    incidents_channel="my-alerts",
    exec_user_emails=["ceo@mycompany.com"],
    protected_channels=["general", "announcements"],
)
scenarios = slack_scenarios(cfg)   # 13 pre-built Slack scenarios
```

Scenario groups:
- **Slack:** incident response, privacy & safety, approval workflows, AI App lifecycle
- **BYO:** fault handling (429/403/500), data safety (no deletes, read-only), state integrity (idempotency, referential integrity)

---

## Jira integration

Set four environment variables and Sandiz automatically files bug tickets on regression:

```bash
JIRA_URL=https://mycompany.atlassian.net
JIRA_EMAIL=sandiz-bot@mycompany.com
JIRA_API_TOKEN=your-api-token
JIRA_PROJECT_KEY=ENG
```

Each regression gets one Jira ticket. Re-running the same failing PR adds a comment to the existing ticket instead of creating a duplicate.

---

## Fix suggestions

Enable Claude-powered fix suggestions:

```bash
ANTHROPIC_API_KEY=sk-ant-...
SANDIZ_FIX_SUGGESTIONS=true
```

Each PR comment includes a collapsible "Fix suggestions" section with a specific, actionable suggestion for each regression — which file to edit, what to change, why it broke.

---

## Baseline management

```bash
# See current baselines
sandiz baseline list --spec-id your-api

# Promote a specific run to baseline (e.g. after a release)
sandiz baseline promote 42 --branch main --note "v2.1 stable"
```

Or use the dashboard UI — click ⚑ on any passing run to set it as the baseline.

---

## Architecture

```
Customer CI (GitHub Actions)          Sandiz SaaS (your server)
┌─────────────────────────┐          ┌──────────────────────────┐
│  sandiz run scenarios/  │          │  sandiz-api (FastAPI)    │
│  └─ agent code runs     │──HTTPS──▶│  ├─ BYO sandbox          │
│     against sandbox     │          │  ├─ Slack sandbox         │
│  sandiz regression diff │◀─report─ │  ├─ Dashboard UI          │
│  PR comment + Jira      │          │  sandiz-redis             │
└─────────────────────────┘          │  sandiz-postgres          │
                                     └──────────────────────────┘
```

---

## Environment variables

See [`.env.example`](.env.example) for the full reference.

Key variables:

| Variable | Required | Description |
|---|---|---|
| `SANDIZ_TOKEN_SECRET` | ✅ | JWT signing secret for sandbox tokens |
| `REDIS_URL` | ✅ | Redis connection string |
| `DATABASE_URL` | ✅ | Postgres connection string |
| `ANTHROPIC_API_KEY` | Optional | Enables seed generation + fix suggestions |
| `SANDIZ_FIX_SUGGESTIONS` | Optional | Enable fix suggestions (`true`/`false`) |
| `JIRA_URL` | Optional | Jira instance URL for ticket filing |
| `JIRA_API_TOKEN` | Optional | Jira API token |

---

## License

MIT
