Metadata-Version: 2.4
Name: evidpath
Version: 0.4.0
Summary: Deterministic interaction-testing CLI for auditing recommender, search, and agent systems with seeded scenarios, trace-based judging, and reproducible regression workflows.
Author-email: Alankrit Verma <alankrit386@gmail.com>
Maintainer-email: Alankrit Verma <alankrit386@gmail.com>
License-Expression: AGPL-3.0-only
Project-URL: Homepage, https://github.com/NDETERMINA/limitation/tree/main/products/evidpath
Project-URL: Source, https://github.com/NDETERMINA/limitation
Project-URL: Issues, https://github.com/NDETERMINA/limitation/issues
Project-URL: Documentation, https://github.com/NDETERMINA/limitation/tree/main/products/evidpath
Project-URL: Releases, https://github.com/NDETERMINA/limitation/releases
Project-URL: PyPI, https://pypi.org/project/evidpath/
Keywords: agents,recommender,search,evaluation,testing,audit,regression,synthetic-users
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: hf-example
Requires-Dist: transformers>=5.5; extra == "hf-example"
Requires-Dist: torch>=2.11; extra == "hf-example"
Provides-Extra: huggingface
Requires-Dist: transformers>=4.30; extra == "huggingface"
Requires-Dist: torch>=2.0; extra == "huggingface"
Provides-Extra: mlflow
Requires-Dist: mlflow>=2.0; extra == "mlflow"
Provides-Extra: sklearn
Requires-Dist: scikit-learn>=1.3; extra == "sklearn"
Dynamic: license-file

# Evidpath

[Source](https://github.com/NDETERMINA/limitation) |
[Full docs](https://github.com/NDETERMINA/limitation/tree/main/products/evidpath) |
[Releases](https://github.com/NDETERMINA/limitation/releases) |
[Issues](https://github.com/NDETERMINA/limitation/issues)

Evidpath helps teams check recommender systems, search rankers, and agent
targets before launch. Point it at an endpoint, local Python target, or driver
config, run an audit, and open a report that shows what happened.

Public domains:

- `recommender`: deepest domain, including native HTTP, schema-mapped HTTP,
  in-process Python, local reference service, compare, generated scenarios,
  generated populations, and `run-swarm`.
- `search`: real public domain with native HTTP, schema-mapped HTTP,
  in-process Python, reference audits, `check-target`, compare, and plan-first
  audit/compare workflows.
- `agents`: real public domain for trajectory audits with built-in reference,
  in-process Python/LangGraph-style targets, OpenAI-compatible Chat
  Completions, Anthropic Messages, MCP stdio, and HTTP-session targets.
  Agent audits record observe-only boundary evidence for normalized tool calls
  when tool calls are present. Harness-owned or fixture-backed paths can also
  add mediated boundary events, local allow/warn/block policy outcomes, and
  post-run boundary check artifacts when explicitly configured.

Generated scenario/population workflows and `run-swarm` are currently
recommender-only. Default HTTP-session agents can use `--target-url`; custom
agent HTTP/session targets use `--driver-config-path`. Agent `serve-reference`
paths are deferred.

## Install

```bash
python -m pip install evidpath
```

Requirements:

- Python `3.11+`

## First Run

Generate a starter driver config when your target does not speak the native
HTTP contract:

```bash
evidpath init --domain recommender --driver-kind http_schema_mapped --base-url http://127.0.0.1:8051 --non-interactive
```

Check that your recommender endpoint responds in the expected shape:

```bash
evidpath check-target --domain recommender --target-url http://127.0.0.1:8051
evidpath check-target --domain recommender --driver-config-path ./evidpath-driver-config.json
```

Run one audit:

```bash
evidpath audit --domain recommender --target-url http://127.0.0.1:8051 --scenario returning-user-home-feed --seed 7
```

For search rankers, use `--domain search`:

```bash
evidpath audit --domain search --scenario navigational-query --seed 7
evidpath compare --domain search --baseline-url http://127.0.0.1:8051 --candidate-url http://127.0.0.1:8052 --scenario navigational-query --rerun-count 2
evidpath plan-run --workflow audit --domain search --scenario navigational-query --output-dir ./planned-search-audit
```

For agents, use `--domain agents` with an in-process callable, protocol driver,
default HTTP-session URL, or the built-in reference target:

```bash
evidpath audit --domain agents --scenario current-info-tool-use --seed 7
evidpath audit --domain agents --target-url http://127.0.0.1:8065 --scenario current-info-tool-use --seed 7
```

For a custom deployed HTTP-session agent, generate a JSON driver config:

```bash
evidpath init --domain agents --driver-kind http_session --base-url https://agent.example.com --non-interactive
evidpath check-target --domain agents --driver-config-path ./evidpath-driver-config.json
```

Then audit:

```bash
evidpath audit --domain agents --driver-config-path ./evidpath-driver-config.json --scenario current-info-tool-use --seed 7
```

For reviewable handoff, validate and compile a JSON source spec into the same
run-plan lifecycle:

```bash
evidpath validate-spec --spec-path ./evidpath-spec.json
evidpath plan-from-spec --spec-path ./evidpath-spec.json --output-dir ./planned-from-spec
evidpath execute-plan --run-plan-path ./planned-from-spec/run_plan.json
```

Source spec v1 is JSON-only and compiles into `run_plan.json`; it can preserve
the supported local subset of boundary policy/check declarations for mediated
runtime paths and post-run verification. Executed agent audits can write
observe-only `boundary_events.jsonl` records from normalized tool calls.
Harness-owned or fixture-backed agent paths may additionally write runtime
boundary events when they explicitly route interactions through the boundary
gateway. When supported checks are configured, runs can write
`boundary_checks.json` with deterministic verifier results over available
boundary evidence. These records describe local evidence; they do not create
branch fanout, use hosted workers, provide broad replay, or isolate execution.

By default, Evidpath writes an `evidpath-output/` folder in your current
directory. The most useful files are:

- `report.md`
- `results.json`
- `traces.jsonl`
- HTTP-session agent audits also include `boundary_summary.json` and
  `boundary_events.jsonl` observe-mode sandbox evidence sidecars.
- optional `boundary_events.jsonl` for observe-mode boundary events
- optional `boundary_checks.json` for local boundary verifier results

Compare two versions before launch:

```bash
evidpath compare --domain recommender --baseline-url http://127.0.0.1:8051 --candidate-url http://127.0.0.1:8052 --rerun-count 2
```

## What You Need

- a recommender or search HTTP endpoint, a supported agent driver config, or a
  Python callable/class
- for native HTTP targets, the request and response shape described in the
  domain-specific external target contract

If your HTTP service has a different shape, use a schema-mapped driver config
with dot paths, JSONPath `items_path`, or small Python transforms. If your
ranker or agent is local Python code, use `evidpath.audit(callable=..., ...)`.

Agent targets support in-process Python/LangGraph-style objects,
OpenAI-compatible Chat Completions, Anthropic Messages, MCP stdio, and
session-aware HTTP services. Default HTTP-session services can use
`--target-url`; custom session paths, auth headers, provider drivers,
in-process drivers, and MCP stdio use driver configs. HTTP-session agent
configs support `check-target --driver-config-path`; provider, in-process, and
MCP agent configs can be generated and audited but currently return unsupported
preflight messages.
Boundary gateway metadata distinguishes final-output-only, observed, proxied,
fixture-backed, mediated, instrumented, and harness-owned evidence labels
without treating those labels as hosted sandbox isolation. Only explicitly
mediated runtime paths can carry local policy outcomes, and post-run boundary
checks are verifier evidence rather than proof that an earlier side effect was
blocked.

For repository examples of each agent driver family, see the full product docs
linked below.

Optional extras provide adapters for common in-process objects:

- `evidpath[huggingface]`
- `evidpath[mlflow]`
- `evidpath[sklearn]`

The reference target remains available in the repo for demos and local
onboarding.

Provider credentials are only needed for AI-backed generation, advisory
semantic features, or provider-backed agent drivers. A normal `check-target`,
`audit`, or `compare` run against your own endpoint does not need an Evidpath
provider API key.

Semantic advisory artifacts are explanation sidecars. Deterministic checks,
package validation, and smoke commands remain the release gate.

## Links

- Full product docs: https://github.com/NDETERMINA/limitation/tree/main/products/evidpath
- Demo guide: https://github.com/NDETERMINA/limitation/blob/main/products/evidpath/DEMO.md
- Releases: https://github.com/NDETERMINA/limitation/releases
- Issues: https://github.com/NDETERMINA/limitation/issues
