Metadata-Version: 2.4
Name: path-index
Version: 0.0.1
Summary: PATH-Index: Pathwise Assessment of Trajectory Health - a diagnostic framework for the execution integrity of AI agent trajectories.
Project-URL: Homepage, https://github.com/vsrivas7/path-index
Project-URL: Repository, https://github.com/vsrivas7/path-index
Author: V. Srivastava
License: Apache-2.0
Keywords: ai-agents,benchmarking,diagnostics,evaluation,llm,trajectory
Classifier: Development Status :: 2 - Pre-Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.9
Description-Content-Type: text/markdown

# PATH-Index

**Pathwise Assessment of Trajectory Health** — a diagnostic framework for measuring the execution integrity of AI agent trajectories.

> 📄 Companion paper: *PATH-Index: Decomposing Where and Why AI Agents Fail* (in preparation; preprint link coming soon)

PATH-Index is the trajectory-level sibling of [VerifyIndex](https://github.com/vsrivas7/verifyindex). The two frameworks form one measurement program:

- **VerifyIndex** evaluates the *evidential integrity of AI outputs* (claim-level).
- **PATH-Index** evaluates the *execution integrity of AI agent trajectories* (step-level, pathwise).

"Pathwise" is meant in the stochastic-process sense: quality is assessed along each individual trajectory, not merely in expectation over tasks.

## Why another agent evaluation framework?

Most agent benchmarks report aggregate task success. That tells you *whether* an agent failed, but not *where* in the trajectory the failure originated, *how far* it propagated, or *whether* the agent recovered. PATH-Index is designed for failure localization: separating the causal **origin** of a failure from its downstream **manifestation**, and reporting a diagnostic profile rather than a single leaderboard number.

## The Trajectory Funnel

Each step of a trajectory passes through five nested gates, evaluated in order:

| Gate | Question it answers |
|---|---|
| **G — Goal Alignment** | Is the step's sub-goal well-formed and consistent with the user's task, as evidenced by observable actions? |
| **S — State Grounding** | Does the agent have (and use) the environment state and context the step requires? |
| **T — Tool Selection** | Given goal and grounding, is the chosen tool (or "respond directly") appropriate? |
| **A — Argument Fidelity** | Are the tool arguments correct, well-typed, and consistent with the grounded state? |
| **O — Outcome** | Did the call produce its specified effect (execution validity), as opposed to erroring, no-op-ing, or silently corrupting state? |

Because the gates are nested, step yield factorizes as a product of sequential conditionals:

```
H_T = P(G) · P(S | G) · P(T | S) · P(A | T) · P(O | A)
```

**Trajectory Health is reported as a profile across these stages plus cross-step diagnostics — not as a single scalar.** A composite score is optional and derived, never the headline.

## Beyond the funnel: cross-step diagnostics

The reported profile also includes quantities that live across steps rather than inside the yield identity:

- **Failure persistence** (how long a failure run lasts) — reported separately from
- **Causal propagation** (counterfactually defined downstream damage, requiring paired trajectories)
- **Recovery** — binary recovery events and graded recovery quality, reported as distinct objects
- **Oversight** — recall, precision, and false-defer rate of escalation/deferral behavior
- **Cost profile** — a vector over steps, tool calls, tokens, latency, and monetary cost
- **Reliability** — PassAll@k and PassAny@k across repeated runs

Diagnostic evaluation is scored against six metrics, including origin identification (Top-1 and Top-2 recall), manifestation F1, affected-stage recall, propagation-depth error, and recovery-time error.

## Status

🚧 **Pre-release.** The founding paper is in preparation. This repository reserves the name and will host:

- [ ] Paper preprint and citation info
- [ ] Reference implementation (annotation rubrics, scoring pipeline)
- [ ] Calibration / pilot-threshold simulation code
- [ ] Example annotated trajectories

Watch or star the repo to be notified when the preprint and reference implementation land.

## Citation

Citation information will be added when the preprint is available. In the meantime, please reference this repository:

```bibtex
@misc{pathindex2026,
  title  = {PATH-Index: Decomposing Where and Why AI Agents Fail},
  author = {Srivastava, V.},
  year   = {2026},
  note   = {Framework repository: https://github.com/vsrivas7/path-index}
}
```

## Related

- [VerifyIndex](https://github.com/vsrivas7/verifyindex) — sequential measurement of claim-level Verification Yield for AI outputs.

## License

TBD (code will be released under a permissive open-source license; annotation rubrics and paper text under CC BY 4.0).
