Metadata-Version: 2.4
Name: deepeval-openttt
Version: 0.1.0
Summary: TTTPS Proof-of-Time provenance metric for DeepEval: verify that an LLM output carries a genuine, content-bound audit-trail timestamp from the public KPP Provenance API
License: MIT
Project-URL: Homepage, https://github.com/Helm-Protocol/OpenTTT
Project-URL: Repository, https://github.com/Helm-Protocol/OpenTTT
Project-URL: Issues, https://github.com/Helm-Protocol/OpenTTT/issues
Keywords: deepeval,proof-of-time,provenance,temporal-attestation,audit-trail,llm-evaluation,metric
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.9
Description-Content-Type: text/markdown
Requires-Dist: httpx>=0.24
Requires-Dist: deepeval<5,>=4.0

# deepeval-openttt

TTTPS Proof-of-Time provenance metric for [DeepEval](https://github.com/confident-ai/deepeval).
It answers one question about a test case: does this output carry a genuine
Proof-of-Time receipt that actually covers it, according to the public
self-serve [KPP Provenance API](https://kpp.kenosian.com)?

Deterministic and rule based. There is no LLM judge, no model call, and no
API key needed for scoring existing receipts.

This attaches and checks a cryptographic audit-trail timestamp and integrity
hash. It does **not** certify legal or regulatory compliance (EU AI Act, FDA,
and so on). Treat it as an audit-trail timestamp, not a compliance claim.

## Install

```bash
pip install deepeval-openttt
```

## Usage

Score a receipt your pipeline already produced, for example one attached by
`litellm-openttt`, `llamaindex-openttt` or `smolagents-openttt`:

```python
from deepeval.test_case import LLMTestCase
from deepeval_openttt import TTTPSProvenanceMetric

test_case = LLMTestCase(
    input="how tall is the Eiffel Tower?",
    actual_output="The Eiffel Tower is 330 metres tall.",
    metadata={"tttps_receipt": receipt},   # raw KPP receipt dict
)

metric = TTTPSProvenanceMetric()
metric.measure(test_case)
print(metric.score, metric.reason)
# 1.0  Structurally valid receipt (id=...). cross-checked against the KPP Provenance API (...)
```

Or let the metric seal the output itself (needs a key, mint one for free with
`POST https://kpp.kenosian.com/v1/keys`):

```python
metric = TTTPSProvenanceMetric(anchor_if_missing=True)   # reads KPP_API_KEY
```

Both `evaluate([...], [metric])` and `assert_test` work, since this is a
plain `BaseMetric` with `measure()` and `a_measure()`.

## What it checks, and what it cannot

Two layers. The structural layer asks "is a well formed receipt attached".
The cross-check layer, on by default, asks "does the API still have this
record, and does it cover this exact output":

1. content binding: `sha256(actual_output)` equals the anchored digest
2. existence: the API returns a verified record for this receipt id
3. field agreement: the receipt's digest and timestamp equal the API's

Measured 2026-08-05 over 25 rows of the published corpus
[Pittro/verifiable-ai-provenance-bench](https://huggingface.co/datasets/Pittro/verifiable-ai-provenance-bench)
plus 5 negatives built by mutating real receipts:

| case | `cross_check=False` | default (`cross_check=True`) |
|---|---|---|
| genuine receipt lifted onto different output | 1.0 (missed) | 0.0 (caught) |
| receipt blob edited, one hex nibble flipped | 1.0 (missed) | 0.0 (caught) |
| timestamp rewritten one year into the past | 1.0 (missed) | 0.0 (caught) |
| receipt id that was never issued | 1.0 (missed) | 0.0 (caught) |
| no receipt at all | 0.0 (caught) | 0.0 (caught) |
| 25 published genuine receipts | 1.00 | 1.00 |

Negatives caught: 1 of 5 structure only, 5 of 5 with the default.
Separation between the genuine mean and the negative mean: 0.20 structure
only, 1.00 with the default.

Reproducing those rows needs only your own key: anchor a text, then (a)
attach the receipt to a different text, (b) flip one hex character of
`content_hash`, (c) rewrite `time` by a year, (d) invent a 24 hex character
receipt id, (e) attach nothing.

What it does not tell you:

- nothing about whether the output is correct, safe or useful. It is a
  provenance check, not a quality check
- a receipt proves the content existed no later than the anchored time. It
  cannot prove the content did not exist earlier
- if your pipeline anchors a different byte string than `actual_output` (a
  canonical JSON, a trimmed answer), content binding fails. Anchor the exact
  string you evaluate, or set `bind_content=False` and accept that a lifted
  receipt then scores 1.0
- with `cross_check=False` it detects nothing beyond a missing or malformed
  receipt
- it does not require any particular time source. The API reports
  `time_source` (`roughtime_chain` or `local`) on the record kept in
  `metric.tttps_server_record`, so gate on that yourself if chain backed time
  matters to you

## Score semantics

Per case the score is binary, 1.0 or 0.0: a receipt either covers this output
or it does not. The continuous number a DeepEval report shows for a suite is
the mean over N cases, in other words a pass rate, not a per case confidence.

## Options

| argument | default | meaning |
|---|---|---|
| `cross_check` | `True` | verify the receipt against the API |
| `bind_content` | `True` | require `sha256(actual_output)` to equal the anchored digest |
| `anchor_if_missing` | `False` | seal the output when no receipt is attached |
| `metadata_key` | `"tttps_receipt"` | where to read the receipt on the test case |
| `timestamp_tolerance_s` | `1.0` | allowed gap between claimed and recorded time |
| `threshold` | `1.0` | DeepEval pass threshold |

Environment: `KPP_BASE` (default `https://kpp.kenosian.com`), `KPP_API_KEY`,
`KPP_TIMEOUT_S` (seconds, default `1.0`).

Fail-open: a slow or unreachable API degrades the score and records why in
`metric.reason`. It never raises out of the eval run. With `cross_check=True`
the metric makes one `POST /v1/verify` round trip per test case.

## License

MIT
