Metadata-Version: 2.4
Name: detra
Version: 0.2.0
Summary: LLM/agent observability for behavior deviations, guardrails, and telemetry
Project-URL: Homepage, https://github.com/adc77/detra
Project-URL: Repository, https://github.com/adc77/detra
Project-URL: Documentation, https://github.com/adc77/detra#readme
Project-URL: Issues, https://github.com/adc77/detra/issues
Author: detra Contributors
License: MIT
License-File: LICENSE
Keywords: ai,datadog,evaluation,guardrails,llm,observability,opentelemetry,security,tracing
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: System :: Monitoring
Requires-Python: >=3.11
Requires-Dist: httpx>=0.26.0
Requires-Dist: pydantic-settings>=2.1.0
Requires-Dist: pydantic>=2.5.0
Requires-Dist: python-dotenv>=1.0.0
Requires-Dist: pyyaml>=6.0.1
Requires-Dist: structlog>=24.1.0
Provides-Extra: all
Requires-Dist: certifi>=2024.2.2; extra == 'all'
Requires-Dist: datadog-api-client>=2.20.0; extra == 'all'
Requires-Dist: datadog>=0.49.0; extra == 'all'
Requires-Dist: ddtrace>=2.10.0; extra == 'all'
Requires-Dist: dspy-ai>=2.4.0; extra == 'all'
Requires-Dist: fastapi>=0.104.0; extra == 'all'
Requires-Dist: google-genai>=0.2.0; extra == 'all'
Requires-Dist: litellm>=1.0.0; extra == 'all'
Requires-Dist: mypy>=1.8.0; extra == 'all'
Requires-Dist: opentelemetry-api>=1.22.0; extra == 'all'
Requires-Dist: opentelemetry-sdk>=1.22.0; extra == 'all'
Requires-Dist: pytest-asyncio>=0.23.0; extra == 'all'
Requires-Dist: pytest-cov>=4.1.0; extra == 'all'
Requires-Dist: pytest>=8.0.0; extra == 'all'
Requires-Dist: ruff>=0.2.0; extra == 'all'
Requires-Dist: tenacity>=8.2.0; extra == 'all'
Requires-Dist: tiktoken>=0.5.0; extra == 'all'
Requires-Dist: uvicorn[standard]>=0.24.0; extra == 'all'
Provides-Extra: datadog
Requires-Dist: certifi>=2024.2.2; extra == 'datadog'
Requires-Dist: datadog-api-client>=2.20.0; extra == 'datadog'
Requires-Dist: datadog>=0.49.0; extra == 'datadog'
Requires-Dist: ddtrace>=2.10.0; extra == 'datadog'
Provides-Extra: dev
Requires-Dist: mypy>=1.8.0; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.23.0; extra == 'dev'
Requires-Dist: pytest-cov>=4.1.0; extra == 'dev'
Requires-Dist: pytest>=8.0.0; extra == 'dev'
Requires-Dist: ruff>=0.2.0; extra == 'dev'
Provides-Extra: gemini
Requires-Dist: google-genai>=0.2.0; extra == 'gemini'
Requires-Dist: tenacity>=8.2.0; extra == 'gemini'
Requires-Dist: tiktoken>=0.5.0; extra == 'gemini'
Provides-Extra: litellm
Requires-Dist: litellm>=1.0.0; extra == 'litellm'
Provides-Extra: optimization
Requires-Dist: dspy-ai>=2.4.0; extra == 'optimization'
Provides-Extra: otel
Requires-Dist: opentelemetry-api>=1.22.0; extra == 'otel'
Requires-Dist: opentelemetry-sdk>=1.22.0; extra == 'otel'
Provides-Extra: server
Requires-Dist: fastapi>=0.104.0; extra == 'server'
Requires-Dist: uvicorn[standard]>=0.24.0; extra == 'server'
Description-Content-Type: text/markdown

# detra

**LLM/agent observability for behavior deviations, guardrails, and telemetry.**

detra traces LLM and agent steps, compares outputs against configured behavior expectations, flags deviations, and ships metrics/events to your telemetry backend. Deterministic checks run first; LLM judges are optional and should be sampled or used with compact evidence.

## Install

```bash
pip install detra                    # core only (6 deps)
pip install "detra[litellm]"         # + any-model judge
pip install "detra[otel]"            # + OpenTelemetry backend
pip install "detra[datadog]"         # + Datadog backend
pip install "detra[all]"             # everything
```

## Quick Start

```python
import detra

vg = detra.init("detra.yaml")

@vg.trace("refund_policy_agent")
async def answer_refund_question(message: str):
    result = await agent.run(message)
    return result

# Tracing + deviation checks happen automatically
result = await answer_refund_question("Can I get a refund after 90 days?")
```

## Define Behaviors in YAML

```yaml
app_name: my-llm-app

judge_config:
  provider: none        # or: litellm, gemini
  model: gpt-4o-mini

sampling:
  rate: 0.1

nodes:
  refund_policy_agent:
    expected_behaviors:
      - "Must answer with either eligible, ineligible, or needs_review"
      - "Must mention the policy window when denying a refund"
      - "Must route to needs_review for missing order dates"
    unexpected_behaviors:
      - "Approves refunds outside the policy window"
      - "Invents policy exceptions"
      - "Asks for payment details"
    adherence_threshold: 0.90
    security_checks:
      - pii_detection
      - prompt_injection
```

## Pluggable Architecture

**Backends** -- where telemetry goes:

| Backend | Install | Use case |
|---------|---------|----------|
| Console | included | Local dev, CI |
| OpenTelemetry | `detra[otel]` | Prometheus, Jaeger, OTLP |
| Datadog | `detra[datadog]` | Datadog LLM Observability |
| Custom | included | Implement the protocol |

**Judges** -- optional reviewers for sampled or ambiguous behavior checks:

| Judge | Install | Models |
|-------|---------|--------|
| LiteLLM | `detra[litellm]` | GPT-4o, Claude, Gemini, Llama, 100+ |
| Gemini | `detra[gemini]` | Google Gemini models |
| None | included | Rules-only mode |

## Evaluation Pipeline

1. **Rule checks** (fast): empty output, JSON validity, length
2. **Security scans**: PII, prompt injection, sensitive content
3. **Behavior/deviation checks**: expected/unexpected behaviors from YAML spec
4. **Optional sampled judge**: for ambiguous cases or compact evidence
5. **Flag + alert**: when score < threshold or security issues found

## Decorator Types

```python
@vg.trace("node")          # generic
@vg.workflow("pipeline")   # workflow
@vg.llm("model_call")     # LLM call
@vg.task("processing")    # task
@vg.agent("step")         # agent
```

Works on sync and async functions. Safe inside running event loops (FastAPI, etc.).

## Links

- [GitHub](https://github.com/adc77/detra)
- [Examples](https://github.com/adc77/detra/tree/main/examples)

## License

MIT
