Metadata-Version: 2.4
Name: hb-eval-sdk
Version: 2.9.1
Summary: HB-Eval SDK: operational reliability evaluation for agentic AI — fault-injection battery, runtime monitoring, MCP server, and LangChain/LangGraph/CrewAI adapters
Author-email: Abuelgasim Mohamed Ibrahim Adam <abuelgasim.hbeval@outlook.com>
License: MIT
Project-URL: Homepage, https://hbeval.com
Project-URL: Repository, https://github.com/hb-evalSystem/hb-eval-sdk
Project-URL: Documentation, https://github.com/hb-evalSystem/HB-System/blob/main/docs
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Intended Audience :: Developers
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.9
Description-Content-Type: text/markdown
Requires-Dist: requests>=2.28.0
Requires-Dist: cryptography>=41.0.0
Requires-Dist: pydantic>=2.0.0
Provides-Extra: otel
Requires-Dist: opentelemetry-api>=1.30.0; extra == "otel"
Requires-Dist: opentelemetry-sdk>=1.30.0; extra == "otel"
Provides-Extra: langchain
Requires-Dist: langchain-core>=0.1.0; extra == "langchain"
Provides-Extra: langgraph
Requires-Dist: langgraph>=0.2.0; extra == "langgraph"
Requires-Dist: langchain-core>=0.1.0; extra == "langgraph"
Provides-Extra: crewai
Requires-Dist: crewai>=0.28.0; extra == "crewai"
Provides-Extra: mcp
Requires-Dist: mcp>=1.2.0; extra == "mcp"
Provides-Extra: all-integrations
Requires-Dist: langchain-core>=0.1.0; extra == "all-integrations"
Requires-Dist: langgraph>=0.2.0; extra == "all-integrations"
Requires-Dist: crewai>=0.28.0; extra == "all-integrations"
Requires-Dist: mcp>=1.2.0; extra == "all-integrations"
Provides-Extra: dev
Requires-Dist: pytest>=7.0.0; extra == "dev"
Requires-Dist: flake8>=6.0.0; extra == "dev"


# HB-Eval SDK

**Operational reliability measurement for agentic AI.**

Your agent finished the task. Did it behave?

Benchmarks measure whether an agent completed its work. Almost nothing measures
how it behaved while things were going wrong — and that is where production
incidents come from. HB-Eval measures it, watches it live, and can stop a run
before the damage lands.

[hbeval.com](https://hbeval.com) · [Documentation](https://hbeval.com/docs) ·
[Architecture](https://hbeval.com/architecture) ·
[Science](https://hbeval.com/science)

---

## Install

```bash
pip install hb-eval-sdk==2.9.0
```

Python 3.10+. Three dependencies: `requests`, `cryptography`, `pydantic`.

---

## The five metrics

| | | Answers |
|---|---|---|
| **PEI** | Planning Efficiency | Is the plan holding, or is the agent redoing it? |
| **FRR** | Failure Resilience | Of the steps that met a fault, how many still completed? |
| **IRS** | Deliberate Handling | Was the fault handled deliberately, or met by reflex? |
| **TI** | Traceability | Can each decision be followed afterwards? |
| **CSI** | Consistency Stability | Does the same task produce the same behaviour across runs? |

**Undefined is never reported as zero.** Resilience cannot be scored before a
fault happens; consistency cannot be measured within a single session. Those
come back as `None` and stay `None` through the wire protocol, the database and
every chart. Substituting zero would claim a measured failure on a dimension
nothing examined — and a CI gate or an auditor reading that zero would act on
it.

---

## Quick start

### Measure a live agent

```python
from hb_eval_sdk import HBEvalClient

client = HBEvalClient(api_key=..., aes_key=..., signing_secret=...)

with client.monitor(
    agent_id="support-agent",
    halt_policy={"metric": "frr", "below": 0.5, "for_steps": 3},
) as session:
    for step in my_agent.run(task):
        session.record_step(
            action=step.name,
            success=step.ok,
            had_fault=step.faulted,
            recovered_intentionally=step.recovery_was_reasoned,
            traceable=step.reasoning_recorded,
        )
        if session.should_halt:
            break          # cooperative: your loop decides how to stop
```

Metrics are computed locally, in your process, at roughly 0.002 ms per step —
flat from a hundred steps to fifty thousand. Safe Halt is decided locally too,
so your policy still fires when the network is down. A guard that needs a round
trip is a guard that fails exactly when infrastructure is already struggling.

### Run the fault battery

```python
report = client.evaluate_with_battery(
    {"system": "...", "question": "..."},
    my_agent,
    n_scenarios=30,
    seed=42,               # same seed, same battery
)
```

Six fault types across six domains: `tool_failure`, `context_corruption`,
`stochastic`, `adversarial`, `cascade`, `combined`.

### Zero-setup instrumentation

```python
import hb_eval_sdk.auto     # that is the whole setup
```

Derives reliability signals from OpenTelemetry spans you already emit. Nothing
else to add.

---

## Framework adapters

```python
from hb_eval_sdk import (
    adapt_langchain_agent,
    adapt_langgraph_agent,
    adapt_crewai_agent,
)

runner = adapt_langchain_agent(agent_executor)
runner = adapt_langgraph_agent(compiled_graph)
runner = adapt_crewai_agent(crew_agent)
```

One call. Your agent keeps its own control flow — HB-Eval measures it rather
than replacing it.

---

## Behavioural evidence

Scoring from an agent's text alone lets its wording decide its score. A
counterfactual test showed the size of that: the same run — three identical
retries, no re-plan — scored FRR 1.00 when described as *"I recognized the
failure and deliberately replanned"*, and 0.00 when described honestly as
*"the operation failed"*.

The agent that failed and said so scored worse than the one that failed and did
not. Return a dict from your runner and the scorer weighs conduct over claims:

```python
def my_agent(system_prompt, question):
    trace = run_my_agent(question)
    return {
        "response": trace.answer,
        "success": trace.completed,
        "had_fault": trace.hit_a_fault,
        "retries": trace.identical_retry_count,
        "replanned": trace.changed_approach,
        "recovered_intentionally": trace.recovery_was_reasoned,
        "traceable": trace.reasoning_recorded,
    }
```

Text may support a score the trace corroborates; it can never manufacture one
the trace contradicts. Unsupported claims are named in the result rather than
quietly discounted.

Every result carries an evidence level — **E0** text only, **E1** partial
trace, **E2** complete — because two scores of 0.87 are not equally trustworthy
when one was checked against conduct and the other inferred from a paragraph.

---

## Deliberate handling, not only recovery

IRS counts three forms of handling a fault on purpose:

- **Recovery** — the agent reads the failure and changes approach.
- **Resistance** — asked to skip verification, the agent refuses.
- **Abstention** — every source is degraded, so it declines rather than
  inventing an answer.

The last two leave no recovery in the trace and no span saying they happened.
Where they matter, mark them:

```python
session.record_step(
    action="refuse unsafe instruction",
    success=True,
    had_fault=True,
    handled_deliberately=True,     # the one judgement no trace can make
)
```

An earlier definition counted recovery alone, which scored refusing an unsafe
instruction identically to complying with it. See
[the metric evolution record](https://hbeval.com/science) — v1 and v2 results
are not directly comparable, and every report carries its scoring version.

---

## CI gate

```yaml
- uses: hb-evalSystem/hb-eval-sdk@v2.9.0
  with:
    agent: 'myapp.agent:run'
    baseline: '.hbeval/baseline.json'
    enforce: false      # start here
```

Start in warn mode. Agents built on language models vary between runs, and a
gate that blocks on ordinary variance gets switched off within days — after
which it protects nothing. Compare against your own baseline rather than an
absolute floor, and a failing run never updates it.

---

## Agent Passport

A signed record of observed behaviour: the five metrics over 30, 60 and 90
days, every halt decision with the policy that caused it, every alert and
whether it was delivered, and what produced the numbers.

Verification runs in the reader's own browser against a published Ed25519 key.
Nobody has to ask us whether a passport is genuine.

It is **not a certification**. HB-Eval is not an accreditation body, performs
no third-party audit, and makes no warranty about future behaviour.

---

## What this does not do

- **It does not judge correctness.** A confidently wrong answer delivered
  without errors scores well on every metric here.
- **It cannot undo anything.** Safe Halt stops the next step; it does not
  reverse the previous one.
- **It is cooperative.** A halt raises a flag your loop must check.
- **The metrics are not independently validated.** Calibration so far is
  preliminary and internal. See
  [the science page](https://hbeval.com/science) for what has and has not been
  demonstrated.

A project arguing that reliability claims should be measured rather than
asserted is a poor place to start making unmeasured ones.

---

## Links

- **Platform** — [hbeval.com](https://hbeval.com)
- **Documentation** — [hbeval.com/docs](https://hbeval.com/docs)
- **Live demo** — [hbeval.com/demo](https://hbeval.com/demo)
- **Repository** — [github.com/hb-evalSystem/HB-System](https://github.com/hb-evalSystem/HB-System)
- **TypeScript SDK** — [hb-eval-sdk-js](https://www.npmjs.com/package/hb-eval-sdk-js)

MIT licensed. Abuelgasim Mohamed Ibrahim Adam.
