Metadata-Version: 2.4
Name: evidpath
Version: 0.5.0
Summary: Deterministic interaction-testing CLI for auditing recommender, search, and agent systems with seeded scenarios, trace-based judging, and reproducible regression workflows.
Author-email: Alankrit Verma <alankrit386@gmail.com>
Maintainer-email: Alankrit Verma <alankrit386@gmail.com>
License-Expression: AGPL-3.0-only
Project-URL: Homepage, https://github.com/NDETERMINA/limitation/tree/main/products/evidpath
Project-URL: Source, https://github.com/NDETERMINA/limitation
Project-URL: Issues, https://github.com/NDETERMINA/limitation/issues
Project-URL: Documentation, https://github.com/NDETERMINA/limitation/tree/main/products/evidpath
Project-URL: Releases, https://github.com/NDETERMINA/limitation/releases
Project-URL: PyPI, https://pypi.org/project/evidpath/
Keywords: agents,recommender,search,evaluation,testing,audit,regression,synthetic-users
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: hf-example
Requires-Dist: transformers>=5.5; extra == "hf-example"
Requires-Dist: torch>=2.11; extra == "hf-example"
Provides-Extra: huggingface
Requires-Dist: transformers>=4.30; extra == "huggingface"
Requires-Dist: torch>=2.0; extra == "huggingface"
Provides-Extra: mlflow
Requires-Dist: mlflow>=2.0; extra == "mlflow"
Provides-Extra: sklearn
Requires-Dist: scikit-learn>=1.3; extra == "sklearn"
Dynamic: license-file

# Evidpath

[Source](https://github.com/NDETERMINA/limitation) |
[Full docs](https://github.com/NDETERMINA/limitation/tree/main/products/evidpath) |
[Releases](https://github.com/NDETERMINA/limitation/releases) |
[Issues](https://github.com/NDETERMINA/limitation/issues)

Evidpath helps teams check recommender systems, search rankers, and agent
targets before launch. Point it at an endpoint, local Python target, or driver
config, run an audit, and open a report that shows what happened.

Public domains:

- `recommender`: deepest domain, including native HTTP, schema-mapped HTTP,
  in-process Python, local reference service, compare, generated scenarios,
  generated populations, and `run-swarm`.
- `search`: real public domain with native HTTP, schema-mapped HTTP,
  in-process Python, reference audits, `check-target`, compare, generated
  scenarios, generated populations, and plan-first audit/compare workflows.
  Search does not support `run-swarm` yet.
- `agents`: real public domain for trajectory audits with built-in reference,
  in-process Python/LangGraph-style targets, OpenAI-compatible Chat
  Completions, Anthropic Messages, MCP stdio, and HTTP-session targets.
  Agent audits record observe-only boundary evidence for normalized tool calls
  when tool calls are present. Harness-owned or fixture-backed paths can also
  add mediated boundary events, local allow/warn/block policy outcomes, and
  post-run boundary check artifacts when explicitly configured. Supported local
  boundary evidence can additionally produce bounded branch manifests for
  alternate outcomes such as timeouts and permission denials. Agent source
  specs can also reference local task/environment packs that bundle one task,
  local fixture declarations, boundary expectations, setup/reset state, and
  safe exports without claiming hosted isolation. Agents also support the first
  bounded local `run-swarm` path over task/environment pack cases, with an
  `agent_swarm_manifest.json`, per-case run roots, workspace diffs, command
  evidence, verifier output, target-brain refs, deterministic advisory helper
  sidecars, task-by-brain summaries, and local corpus review.

Generated scenario/population workflows are supported for recommender and
search. Agent generated coverage is supported as generated local
task/environment packs with generated coverage sidecars that run against a
user-provided agent. Agent `run-swarm` is supported for the bounded local
task/environment-pack matrix path with executable brain variants from source
specs. Metadata-only or unsupported variants are preserved with reason codes.
Agent generation-selection source specs can now write an
`agent_generation_selection_bundle.json` before execution, recording generated
task/environment/boundary/verifier candidates, selected candidate ids,
lifecycle counts, provider/cache status, seeds, hashes, and manual-review
promotion mode. Run reports, `runs show`, static viewers, and corpus review
surface this generation quality separately from deterministic target behavior.
Generated customer brain candidate specs may be recorded as deferred evidence,
but they are not executable target brains.

Generated customer agent brains, generic agent scenario/population generation,
judge-swarm release gating, hosted approval, managed sandbox isolation,
managed secrets/egress, and search `run-swarm` are deferred.
Default HTTP-session agents can use `--target-url`; custom agent HTTP/session
targets use `--driver-config-path`. Agent `serve-reference` paths are
deferred.

## Install

```bash
python -m pip install evidpath
```

Requirements:

- Python `3.11+`

## First Run

Generate a starter driver config when your target does not speak the native
HTTP contract:

```bash
evidpath init --domain recommender --driver-kind http_schema_mapped --base-url http://127.0.0.1:8051 --non-interactive
```

Check that your recommender endpoint responds in the expected shape:

```bash
evidpath check-target --domain recommender --target-url http://127.0.0.1:8051
evidpath check-target --domain recommender --driver-config-path ./evidpath-driver-config.json
```

Run one audit:

```bash
evidpath audit --domain recommender --target-url http://127.0.0.1:8051 --scenario returning-user-home-feed --seed 7
```

For search rankers, use `--domain search`:

```bash
evidpath audit --domain search --scenario navigational-query --seed 7
evidpath compare --domain search --baseline-url http://127.0.0.1:8051 --candidate-url http://127.0.0.1:8052 --scenario navigational-query --rerun-count 2
evidpath plan-run --workflow audit --domain search --scenario navigational-query --output-dir ./planned-search-audit
evidpath generate-scenarios --domain search --mode fixture --brief "fresh typo and ambiguous query behavior" --output-dir ./search-generated
evidpath generate-population --domain search --mode fixture --brief "freshness-sensitive search users" --population-size 4 --population-candidate-count 8 --output-dir ./search-generated
evidpath audit --domain search --scenario-pack-path ./search-generated/scenario-packs/fresh-typo-and-ambiguous-query-behavior-fixture.json --population-pack-path ./search-generated/population-packs/freshness-sensitive-search-users-fixture.json --output-dir ./search-generated-audit
evidpath compare --domain search --baseline-url http://127.0.0.1:8051 --candidate-url http://127.0.0.1:8052 --scenario-pack-path ./search-generated/scenario-packs/fresh-typo-and-ambiguous-query-behavior-fixture.json --population-pack-path ./search-generated/population-packs/freshness-sensitive-search-users-fixture.json --rerun-count 1 --output-dir ./search-generated-compare
```

Generated search and recommender packs include local
`generated_coverage_plan.json` and `generated_coverage_manifest.json` sidecars
that record provenance, validator status, selected/rejected coverage, local
promotion mode, and hashes. These sidecars explain generated coverage; the
deterministic judge and compare policy still decide outcomes.

For agents, generate a reviewable local task/environment pack:

```bash
evidpath generate-agent-pack --brief "support agent current-info and refusal coverage" --output-dir ./agent-generated
evidpath audit --domain agents --scenario-pack-path ./agent-generated/agent-packs/support-agent-current-info-and-refusal-coverage-fixture.json --output-dir ./agent-generated-audit
```

This writes an `evidpath.agent_task_environment_pack/v1` pack plus generated
coverage sidecars. It can describe supported local tools, fixtures, boundary
checks, deterministic fault profiles, and safe exports. It does not generate an
agent brain, target implementation, hosted sandbox, managed secret/egress
backend, SQL/browser/email/queue/MCP state, or release approval workflow.

Run a bounded local agent swarm over a task/environment pack and driver config:

```bash
evidpath run-swarm \
  --domain agents \
  --brief "local coding proof" \
  --agent-pack-path products/evidpath/examples/agent_coding_swarm/coding-agent-pack.json \
  --driver-config-path products/evidpath/examples/agent_coding_swarm/driver_config.json \
  --agent-max-cases 1 \
  --semantic-mode off \
  --output-dir ./agent-coding-swarm
```

This writes `agent_swarm_manifest.json` and per-case run roots with local
workspace diffs, bounded command evidence, verifier output, and standard
reports/results/traces where available. It also records target-brain identity,
brain visibility, unsupported target seed status where applicable, provider
cache status, and advisory reporter/critic artifacts. Advisory artifacts are
visible review aids; they do not turn deterministic failed or needs-attention
evidence into a pass. The first agent swarm path is local and fixture-backed; it does
not claim hosted isolation, generated customer agent brains, managed
egress/secrets, browser/computer-use automation, deploys, purchases, deletes,
or externally visible side effects.

To exercise generation-selection before the swarm executes:

```bash
evidpath plan-from-spec --spec-path products/evidpath/examples/source_specs/agent_run_swarm_generation_selection_fixture.json --output-dir ./agent-generation-selection
evidpath execute-plan --run-plan-path ./agent-generation-selection/run_plan.json
evidpath runs show ./agent-generation-selection --root ./agent-generation-selection
```

For agents, use `--domain agents` with an in-process callable, protocol driver,
default HTTP-session URL, or the built-in reference target:

```bash
evidpath audit --domain agents --scenario current-info-tool-use --seed 7
evidpath audit --domain agents --target-url http://127.0.0.1:8065 --scenario current-info-tool-use --seed 7
```

For a custom deployed HTTP-session agent, generate a JSON driver config:

```bash
evidpath init --domain agents --driver-kind http_session --base-url https://agent.example.com --non-interactive
evidpath check-target --domain agents --driver-config-path ./evidpath-driver-config.json
```

Then audit:

```bash
evidpath audit --domain agents --driver-config-path ./evidpath-driver-config.json --scenario current-info-tool-use --seed 7
```

For reviewable handoff, validate and compile a JSON source spec into the same
run-plan lifecycle:

```bash
evidpath validate-spec --spec-path ./evidpath-spec.json
evidpath plan-from-spec --spec-path ./evidpath-spec.json --output-dir ./planned-from-spec
evidpath execute-plan --run-plan-path ./planned-from-spec/run_plan.json
```

Saved local evidence can be indexed and inspected without executing another
run:

```bash
evidpath runs index --root ./evidpath-output
evidpath runs list --root ./evidpath-output
evidpath runs show run-agent-1 --root ./evidpath-output
evidpath runs promote-baseline run-agent-1 --root ./evidpath-output
evidpath runs open run-agent-1 --root ./evidpath-output
```

This writes local `run_index.json`, `baseline_registry.json`, and bundle-local
`index.html` viewer artifacts. It is local-first run management, not hosted
dashboards, team approval, scheduling, automatic reruns, managed storage, or
managed sandbox isolation.

For a cross-artifact local review pass, build a corpus over saved runs, trace
imports, generated coverage sidecars, packs, boundary evidence, and baselines:

```bash
evidpath corpus index --root ./evidpath-output --with-health --with-review-summary
evidpath corpus list --index ./evidpath-output/corpus_index.json --domain agents
evidpath corpus promote-candidates --index ./evidpath-output/corpus_index.json
evidpath corpus open --index ./evidpath-output/corpus_index.json
```

This writes `corpus_index.json`, `corpus_health.json`,
`review_manifest.json`, `review_summary.json`, and
`review_workbench/index.html` as local files. Corpus commands do not execute
audits, compares, trace imports, generated coverage, branch tests, saved
plans, hosted calls, or pack mutation. They are local artifact review, not
hosted/team review, managed sandboxing, scheduled replay, judge-swarm gating,
automatic release approval, or generated agent brains.

Source spec v1 is JSON-only and compiles into `run_plan.json`; it can preserve
the supported local subset of boundary policy/check/branch declarations for
mediated runtime paths, post-run verification, and bounded branch matrices.
For agents, source specs can also reference a local task/environment pack:

```bash
PYTHONPATH=products/evidpath evidpath validate-spec --spec-path products/evidpath/examples/source_specs/agent_audit_task_environment_pack.json
PYTHONPATH=products/evidpath evidpath plan-from-spec --spec-path products/evidpath/examples/source_specs/agent_audit_task_environment_pack.json --output-dir /tmp/evidpath-agent-pack-demo
PYTHONPATH=products/evidpath evidpath execute-plan --run-plan-path /tmp/evidpath-agent-pack-demo/run_plan.json
```

Agent `run-swarm` source specs can also execute multiple local brain variants
in one swarm and preserve advisory-helper intent without hidden provider calls:

```bash
PYTHONPATH=products/evidpath evidpath validate-spec --spec-path products/evidpath/examples/source_specs/agent_run_swarm_brain_advisory.json
PYTHONPATH=products/evidpath evidpath plan-from-spec --spec-path products/evidpath/examples/source_specs/agent_run_swarm_brain_advisory.json --output-dir /tmp/evidpath-agent-brain-swarm
PYTHONPATH=products/evidpath evidpath execute-plan --run-plan-path /tmp/evidpath-agent-brain-swarm/run_plan.json
```

For a two-brain synthetic example that produces a clear compliant versus
faulty target delta, use
`products/evidpath/examples/source_specs/agent_run_swarm_multi_brain_synthetic.json`.
Provider helper mode is explicit and cache-aware: set
`agent_brains.helper_swarms.mode` to `provider` and choose `replay`, `live`,
or `refresh`. Missing, mismatched, malformed, or unsafe helper provider output
is recorded as advisory-only helper metadata and does not change deterministic
case status.

Pack v1 is local and declarative: fixture files, local HTTP fixture routes,
deterministic fault profiles, boundary configs, and safe export artifacts. It
does not provide generated agent brains, arbitrary commands, Docker/VM or
hosted workers, managed egress, managed secrets, replay stores, hosted
production trace connectors, browser state, SQL fixtures, email sinks, queues,
or MCP server state. Native agent run artifacts can be imported into a
review-required draft pack:

```bash
PYTHONPATH=products/evidpath evidpath import-traces --domain agents --run-dir /tmp/evidpath-agent-pack-demo --output-dir /tmp/evidpath-agent-trace-import
```

This writes `trace_import.json`, `trace_import_report.md`, and
`draft-agent-pack.json`. The draft must be reviewed before baseline or
release-gating use.
Executed agent audits can write
observe-only `boundary_events.jsonl` records from normalized tool calls.
Harness-owned or fixture-backed agent paths may additionally write runtime
boundary events when they explicitly route interactions through the boundary
gateway. When supported checks are configured, runs can write
`boundary_checks.json` with deterministic verifier results over available
boundary evidence. When local branch testing is configured, runs can write
`boundary_branch_manifest.json` with bounded simulated outcomes such as
timeout, auth denied, malformed result, rate limit, stale result, adversarial
sensor content, partial write, permission denied, ambiguous status, or success
override. These records describe local evidence; they do not use hosted
workers, provide broad replay, isolate execution, or prove recovery for hidden
final-output-only boundaries.

By default, Evidpath writes an `evidpath-output/` folder in your current
directory. The most useful files are:

- `report.md`
- `results.json`
- `traces.jsonl`
- HTTP-session agent audits also include `boundary_summary.json` and
  `boundary_events.jsonl` observe-mode sandbox evidence sidecars.
- optional `boundary_events.jsonl` for observe-mode boundary events
- optional `boundary_checks.json` for local boundary verifier results
- optional `boundary_branch_manifest.json` for explicitly configured local
  boundary branch testing

Compare two versions before launch:

```bash
evidpath compare --domain recommender --baseline-url http://127.0.0.1:8051 --candidate-url http://127.0.0.1:8052 --rerun-count 2
```

## What You Need

- a recommender or search HTTP endpoint, a supported agent driver config, or a
  Python callable/class
- for native HTTP targets, the request and response shape described in the
  domain-specific external target contract

If your HTTP service has a different shape, use a schema-mapped driver config
with dot paths, JSONPath `items_path`, or small Python transforms. If your
ranker or agent is local Python code, use `evidpath.audit(callable=..., ...)`.

Agent targets support in-process Python/LangGraph-style objects,
OpenAI-compatible Chat Completions, Anthropic Messages, MCP stdio, and
session-aware HTTP services. Default HTTP-session services can use
`--target-url`; custom session paths, auth headers, provider drivers,
in-process drivers, and MCP stdio use driver configs. HTTP-session agent
configs support `check-target --driver-config-path`; provider, in-process, and
MCP agent configs can be generated and audited but currently return unsupported
preflight messages.
Boundary gateway metadata distinguishes final-output-only, observed, proxied,
fixture-backed, mediated, instrumented, and harness-owned evidence labels
without treating those labels as hosted sandbox isolation. Only explicitly
mediated runtime paths can carry local policy outcomes, and post-run boundary
checks are verifier evidence rather than proof that an earlier side effect was
blocked.

For repository examples of each agent driver family, see the full product docs
linked below.

Optional extras provide adapters for common in-process objects:

- `evidpath[huggingface]`
- `evidpath[mlflow]`
- `evidpath[sklearn]`

The reference target remains available in the repo for demos and local
onboarding.

Provider credentials are only needed for AI-backed generation, advisory
semantic features, or provider-backed agent drivers. A normal `check-target`,
`audit`, or `compare` run against your own endpoint does not need an Evidpath
provider API key.

Provider-backed generation supports local provider-cache modes:

- `live`: call the provider and write safe replay evidence.
- `replay`: read a matching cache record without a live provider call.
- `refresh`: call the provider again and record refreshed evidence.

Fixture mode stays deterministic and cache-free. Cache sidecars store prompt
digests, seed-vector status, provider/cache status, and safe response payloads
only after redaction checks. They are local evidence artifacts, not hosted
replay, managed sandboxing, or managed secrets.

Semantic advisory artifacts are explanation sidecars. Deterministic checks,
package validation, and smoke commands remain the release gate.

## Links

- Full product docs: https://github.com/NDETERMINA/limitation/tree/main/products/evidpath
- Demo guide: https://github.com/NDETERMINA/limitation/blob/main/products/evidpath/DEMO.md
- Releases: https://github.com/NDETERMINA/limitation/releases
- Issues: https://github.com/NDETERMINA/limitation/issues
