Metadata-Version: 2.4
Name: agent-pariksha
Version: 0.1.0
Summary: Reliability tests for AI agents that take actions: inject tool failures and check what the agent really did
Author: Malaya Kumar Swain
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/Malaya-kumar-swain/agent-pariksha
Project-URL: Source, https://github.com/Malaya-kumar-swain/agent-pariksha
Project-URL: Issues, https://github.com/Malaya-kumar-swain/agent-pariksha/issues
Keywords: ai-agents,agent-evaluation,reliability,fault-injection,llm,mcp,responsible-ai,testing
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Security
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: dev
Requires-Dist: pytest>=8; extra == "dev"
Dynamic: license-file

# Agent-Pariksha

**Find out what your AI agent does when its tools fail.**

*Pariksha (परीक्षा) means test. An agent that takes actions must be tested on the bad days, not only the good ones.*

[![ci](https://github.com/Malaya-kumar-swain/agent-pariksha/actions/workflows/ci.yml/badge.svg)](https://github.com/Malaya-kumar-swain/agent-pariksha/actions)
![python](https://img.shields.io/badge/python-3.10%2B-blue) ![license](https://img.shields.io/badge/license-Apache--2.0-green) ![deps](https://img.shields.io/badge/core%20dependencies-0-brightgreen)

---

## The problem

Demos test agents when every tool works. In production, a payment call times out after the money has already moved, a service returns garbage, or a retrieved record contains a hidden instruction. A chatbot that fails gives a bad answer. An agent that fails can pay a bill twice.

Agent-Pariksha injects these failures and checks what the agent actually did, not only what it said.

## What it tests

| Fault | What happens | What can go wrong |
|---|---|---|
| `transient` | A tool fails once, then recovers | The agent gives up too early |
| `lost_response` | The action succeeds, but the agent sees a timeout | **The agent retries and pays twice** |
| `timeout` | A tool keeps failing | The agent loops, or claims success |
| `malformed` | A tool returns garbage | The agent acts on bad data |
| `injection` | Tool output contains a hidden instruction | The agent obeys it and moves money |

| Check | Question |
|---|---|
| `no_duplicate_action` | Did a side effect happen more than once? |
| `no_forbidden_action` | Did it call a tool it must not call, such as refunding above its limit? |
| `correct_order` | Did it verify before acting? |
| `bounded_calls` | Did it stop instead of retrying forever? |
| `honest_about_failure` | When the task was impossible, did it say so? |
| `task_completed` | When the task was possible, did it finish it? |

The suite runs 16 tests across 4 scenarios in a made-up bank: balance lookup, bill payment, large refund and order status. Nothing real is called and no real money or data is involved.

## Demo (no model or API key needed)

```bash
pip install agent-pariksha
python -m pariksha demo
```

From source: `git clone https://github.com/Malaya-kumar-swain/agent-pariksha && cd agent-pariksha && pip install -e .`

| Scripted demo agent | Reliability score | Example failure |
|---|---|---|
| `naive-agent` | 19% (3 of 16) | Pays twice after a lost response, refunds above its limit, obeys injected text |
| `guarded-agent` | 100% (16 of 16) | None |

> Both demo agents are scripted to illustrate the method. They are not measurements of any real AI system. Run the suite on your own agent to get real numbers.

## Test your own agent

**Your own agent.** Anything with a `run(task, box)` method works. Each scenario has different tools, so give your agent the list from `box.specs()`, send its tool calls through `box.call`, and return its final answer:

```python
from pariksha import run

class MyAgent:
    name = "my-agent"

    def run(self, task, box):
        return my_agent(task, tools=box.specs(), call_tool=box.call)  # my_agent is your own code

print(run(MyAgent()).to_markdown())
```

**A model with tool calling**, through any OpenAI-compatible endpoint:

```bash
python -m pariksha run --base-url https://api.example.com/v1 --model some-model --api-key-env MY_API_KEY --delay 3 --out report --fail-below 0.9
```

`--delay` paces requests for rate limits, and rate-limited requests are retried. A run that still errors counts as a failure and is flagged in the report, so a crash can never look like a pass. `--fail-below` exits with an error, so a drop in reliability can block a release.

## Honest limits

- **Early version:** 16 tests is a starting point, not a certification.
- **Real models:** the model connection has so far been tested only against a local fake server, not a live model.
- **Honesty check:** `honest_about_failure` looks for failure words in English, so it can be fooled by unusual wording and does not cover other languages yet.
- **Related tools:** broader agent evaluation frameworks exist. This project stays small and focuses on one question: is the agent safe when its tools fail?

## Project layout

```
pariksha/
  core.py       tools, fault injection, record of calls and side effects
  scenarios.py  test scenarios and checks
  agents.py     demo agents and the OpenAI-compatible agent
  runner.py     run the suite and build reports
  __main__.py   command line: demo, list, run
tests/          12 tests
```

## Licence

Apache 2.0. Built by [Malaya Kumar Swain](https://github.com/Malaya-kumar-swain). Views and code are my own.
