Metadata-Version: 2.4
Name: agentdog
Version: 0.1.0
Summary: Lightweight evaluation toolkit for AI agents — test tool use, grounding, safety, and efficiency before production
Project-URL: Homepage, https://github.com/SaiTeja-Erukude/agentdog
Project-URL: Repository, https://github.com/SaiTeja-Erukude/agentdog
Project-URL: Bug Tracker, https://github.com/SaiTeja-Erukude/agentdog/issues
Project-URL: Changelog, https://github.com/SaiTeja-Erukude/agentdog/releases
Author: Sai Teja Erukude
License: MIT License
        
        Copyright (c) 2026 Sai Teja Erukude
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
License-File: LICENSE
Keywords: agent,ai,evals,evaluation,llm,prompt-injection,rag,safety,testing,tool-use
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.10
Requires-Dist: click>=8.0
Provides-Extra: dev
Requires-Dist: pytest-cov>=5.0; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Provides-Extra: llm-judge
Requires-Dist: openai>=1.0; extra == 'llm-judge'
Description-Content-Type: text/markdown

# agentdog

Lightweight evaluation toolkit for AI agents. `pytest` for agent behavior  —  test tool use, grounding, safety, and efficiency before production

```
pip install agentdog
pip install "agentdog[llm-judge]"  # for LLMJudge scorer
```

---

## Quickstart

```python
from agentdog import AgentTrace, ToolCall, TestCase, EvalRun, run
from agentdog import ContainsAnswer, UsedTools, AvoidedTools, UnderTokenLimit

trace = AgentTrace(
    input="Summarize the Q3 report.",
    output="Q3 revenue was $4.2M, up 12% YoY.",
    tool_calls=[ToolCall(name="file_search", arguments={"query": "Q3 report"})],
    retrieved_context=["Q3 revenue was $4.2M, growth 12% year over year."],
    total_tokens=620,
)

case = TestCase(
    name="q3-summary",
    tags=["rag"],
    scorers=[
        ContainsAnswer(["4.2M", "12%"]),
        UsedTools(["file_search"]),
        AvoidedTools(["send_email"]),
        UnderTokenLimit(max_tokens=1000),
    ],
)

report = run([EvalRun(case=case, trace=trace)])
report.print(verbose=True)
```

---

## CLI

Define an `evals()` function in any Python file that returns `list[EvalRun]`, then:

```bash
agentdog run my_evals.py             # run all cases
agentdog run my_evals.py -v          # verbose: show scorer details for passing cases
agentdog run my_evals.py --tag rag   # filter by tag
agentdog run my_evals.py --json-out report.json  # machine-readable output
agentdog inspect trace.json          # pretty-print a trace file
```

Exit code is `0` on full pass, `1` on any failure — CI-friendly by default.

---

## Scorers

| Category | Scorers |
|---|---|
| **Answer** | `ContainsAnswer` `ExactAnswer` `RegexAnswer` `ForbiddenContent` `AnswerNotEmpty` |
| **Tools** | `UsedTools` `AvoidedTools` `ToolCallOrder` `MaxToolCalls` `ToolArgContains` `ToolArgEquals` |
| **Grounding** | `GroundedInContext` `CitedSource` `NoContextHallucination` |
| **Safety** | `NoSensitiveDataLeaked` `NoRiskyActionTaken` `PromptInjectionResisted` |
| **Efficiency** | `UnderTokenLimit` `UnderCostLimit` `UnderLatencyLimit` `MaxRetries` |
| **LLM Judge** | `LLMJudge` — use only when deterministic checks aren't enough |

---

## Trace schema

```python
AgentTrace(
    input: str,
    output: str,
    tool_calls: list[ToolCall],        # name, arguments, output, error, latency_ms
    retrieved_context: list[str],
    total_tokens: int | None,
    total_cost_usd: float | None,
    total_latency_ms: float | None,
    num_retries: int,
    metadata: dict,
)
```

Load/save:

```python
trace = AgentTrace.from_json("trace.json")
trace.to_json("trace.json")
```

---

## Custom scorer

```python
from agentdog.scorers.base import Scorer, ScoreResult

class AnswerStartsWith(Scorer):
    def __init__(self, prefix: str):
        self.prefix = prefix

    def score(self, trace) -> ScoreResult:
        passed = trace.output.startswith(self.prefix)
        return ScoreResult(
            passed=passed,
            score=1.0 if passed else 0.0,
            reason=f"Expected output to start with {self.prefix!r}",
        )
```

---

## Example

See [`examples/sample_evals.py`](examples/sample_evals.py) for a complete working example covering RAG, safety, and prompt injection.

---

## Author

**Sai Teja Erukude**  
[GitHub](https://github.com/SaiTeja-Erukude) · [agentdog](https://github.com/SaiTeja-Erukude/agentdog)

Licensed under the [MIT License](LICENSE).
