Metadata-Version: 2.2
Name: agently-devtools
Version: 0.1.10
Summary: Observation and developer tooling companion package for Agently
Author-email: Agently Team <developer@agently.tech>
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Requires-Dist: agently<4.2.0,>=4.1.0.2
Requires-Dist: fastapi<1.0,>=0.104
Requires-Dist: httpx<0.29.0,>=0.28.1
Requires-Dist: uvicorn<1.0,>=0.30
Provides-Extra: dev
Requires-Dist: anyio<5,>=4; extra == "dev"
Requires-Dist: build<2.0,>=1.2; extra == "dev"
Requires-Dist: pytest<9.0.0,>=8.4.0; extra == "dev"
Requires-Dist: pytest-asyncio<2.0.0,>=1.0.0; extra == "dev"

# agently-devtools

`agently-devtools` is the local observation, evaluation, and playground companion package for `agently`.

## Install

```bash
pip install -U agently agently-devtools
```

Compatible with `agently >=4.1.0.2,<4.2.0`.

Compatibility resolution uses the current Agently offline manifest first, then the Agently release registry, then a bounded legacy fallback for `4.1.0.2` through `4.1.1`.

Python `>=3.10`

## Start

```bash
agently-devtools start
```

Default local address:

- Console: `http://127.0.0.1:15596/`
- Ingest: `http://127.0.0.1:15596/observation/ingest`

Explicit listener options:

```bash
agently-devtools start --host 0.0.0.0 --port 15596 --path /observation/ingest --no-open
```

## Observation Bridge

```python
from agently import Agently
from agently_devtools import ObservationBridge

bridge = ObservationBridge(
    Agently,
    app_id="your_app_id",
    group_id="your_group_id",
)
bridge.watch(Agently)
```

Explicit endpoint styles are also supported:

```python
ObservationBridge(Agently, host="127.0.0.1", port=15596, path="/observation/ingest")
ObservationBridge(Agently, endpoint="https://devtools.example.com/observation/ingest")
```

Use selective watch when you do not want to upload every observation event:

```python
bridge = ObservationBridge(
    Agently,
    app_id="your_app_id",
)
bridge.watch(agent, flow, lookup_reference)
```

The older `bridge = ObservationBridge(...); bridge.register(Agently)` style remains compatible and emits a deprecation warning. Prefer binding the Agently object or event center in the constructor, then selecting global or scoped observation with `bridge.watch(...)`.

The bridge uses a background queue and coalesces high-frequency streaming events before upload, so passive observation does not block normal request output. By default it uses EventCenter summary delivery with `dispatch="background"`, `high_frequency_only=True`, `emit_interval=0.25`, `max_items=50`, and `max_summary_items=12`; uploads are batched with `upload_batch_size=200` and `upload_flush_interval=0.25`. You can override those values with `delivery_policy`, `upload_batch_size`, and `upload_flush_interval` on `ObservationBridge(...)`. For short-lived scripts, call `await bridge.flush()` before exit when you need all buffered events delivered. Use `bridge.get_stats()` after flush when an experiment report needs upload batch, coalescing, drop, retry, and pending-buffer diagnostics.

Use `group_id` to isolate scenario, experiment, or environment runs under the same `app_id`. Runtime Observation, recent events, run events, SSE, and WebSocket subscriptions all accept the same group scope, so unrelated experiments do not trigger each other's refreshes.

Observation event storage keeps raw RuntimeEvents for audit. HTTP recent-events, run-events, SSE, and WebSocket hot paths return compact payload views by default so large prompts, model requests, and stream payloads do not overload the browser. Compacted events include a `devtools_view.raw_event_ref`, and the web console uses that cold reference to open full raw payloads in a modal only when requested. Add `payload_mode=raw` to observation event query endpoints only for scoped raw debug output.

Local log persistence is buffered so SQLite fsync does not run for every high-frequency observation event. By default DevTools flushes the log buffer every `logs.storage.flush_interval_seconds=600`, when `logs.storage.flush_event_limit=5000` pending events are reached, on shutdown, and at rate-limited business-critical events such as completed, failed, blocked, or error observations. Tune with `logs.storage.flush_on_critical_events`, `logs.storage.critical_flush_min_interval_seconds`, and `logs.storage.flush_on_exit`. Runtime Observation queries continue to include pending buffered events before they are persisted.

TriggerFlow executions automatically publish their flow definition, so the execution graph can show the full static structure, including branches that have not been hit yet.

AgentTask action observations are consumed from `agent_task.action.started`, `agent_task.action.completed`, and `agent_task.action.failed` RuntimeEvent records. The web console groups them under the AgentTask iteration actions section when iteration metadata is present. Treat these records as display/log facts only; they are not route decisions, verifier results, quality scores, semantic relevance judgments, or task-completion acceptance. Runtime lifecycle status and business `Task Outcome` are displayed separately; a completed run lifecycle is not treated as accepted business output unless the AgentTask result evidence says so.

Model request usage is consumed from `payload.model_request_telemetry.usage_summary` when present, with compatible fallbacks to provider `usage` metadata. The web console displays usage for each model request and aggregates usage upward for the selected run branch and descendants. When provider token counts are unavailable, token fields display `NaN`; estimated input/output character lengths are shown as diagnostics, not as budget or quality gates.

## Scenario Evaluations

```python
from agently_devtools import EvaluationBridge, EvaluationCase, EvaluationRunner

bridge = EvaluationBridge(
    base_url="http://127.0.0.1:15596",
    app_id="your_app_id",
    group_id="your_eval_group",
)
runner = EvaluationRunner(bridge=bridge)

binding = bridge.bind_agent_factory(
    lambda: build_agent(),
    suite_id="support-suite",
    target_name="support-agent",
)

report = runner.run(
    binding,
    cases=[
        EvaluationCase(case_id="refund", input="Need a refund for a duplicate payment."),
        EvaluationCase(case_id="shipping", input="Where is my shipment now?"),
    ],
    rules=[
        lambda record: bool(record.output),
        lambda record: record.error is None,
    ],
    rounds=2,
)
```

Use the binding helper that matches your target:

- `bind_agent(...)` / `bind_agent_factory(...)`
- `bind_request(...)` / `bind_request_factory(...)`
- `bind_triggerflow(...)` / `bind_triggerflow_factory(...)`

`EvaluationBridge` already records rounds into DevTools and keeps observed run linkage automatically. You do not need an extra `ObservationBridge` just for Scenario Evaluations.
