Metadata-Version: 2.5
Name: sooth
Version: 0.2.3
Summary: Verify AI-generated text against source material. Claim-by-claim trust reports powered by Jev (TypeSafe System One).
Project-URL: Homepage, https://github.com/naufalhilmiaji/sooth
Project-URL: Documentation, https://github.com/naufalhilmiaji/sooth/tree/main/docs
Author-email: Naufal Hilmiaji <nhilmiaji@gmail.com>
License: MIT
License-File: LICENSE
Keywords: ai,jev,llm,trust,typesafe,verification
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Software Development :: Quality Assurance
Requires-Python: >=3.11
Requires-Dist: typesafe-sdk
Provides-Extra: dev
Requires-Dist: pytest; extra == 'dev'
Requires-Dist: ruff; extra == 'dev'
Description-Content-Type: text/markdown

# Sooth

**LLMs generate. Sooth verifies.**

Claim-by-claim fact-checking for AI output, designed for CI. Sooth checks every sentence in a draft against your source material and returns `PASS` / `FAIL` / `REVIEW` with calibrated probabilities and the exact source line behind each verdict.

[![PyPI](https://img.shields.io/pypi/v/sooth.svg)](https://pypi.org/project/sooth/)
[![Python](https://img.shields.io/pypi/pyversions/sooth.svg)](https://pypi.org/project/sooth/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://github.com/naufalhilmiaji/sooth/blob/main/LICENSE)
[![CI](https://github.com/naufalhilmiaji/sooth/actions/workflows/ci.yml/badge.svg)](https://github.com/naufalhilmiaji/sooth/actions/workflows/ci.yml)

## See it catch a hallucination

```bash
pip install sooth
sooth demo        # no API key needed
```

Real output — seven claims in an AI-written news summary, three numbers quietly wrong:

```
# Sooth

**PASS 2 · FAIL 3 · REVIEW 1 · UNCHECKABLE 1** — threshold 0.70

| # | Claim | Verdict | P | Why (P distribution) | Evidence |
|---|-------|---------|---|----------------------|----------|
| 1 | Aturan baru BEI menurunkan batas bawah harga saham menjadi Rp 1 per saham mulai 28 September 2026. | ✅ PASS | ████████ 1.00 | supports 1.00 / not_found 0.00 / contradicts 0.00 · details 0.98 | `examples/news-1.md:3` "Saham-saham Grup Bakrie kembali menjadi sorotan menjelang…" |
| 2 | Menurut analis Bumiputera Sekuritas, aturan ini berlaku untuk seluruh emiten di BEI. | ✅ PASS | ████████ 1.00 | supports 1.00 / not_found 0.00 / contradicts 0.00 · details 0.82 | `examples/news-1.md:7` "Ia menyebut aturan ini berlaku untuk seluruh saham di BEI…" |
| 3 | BNBR baru menuntaskan rights issue bernilai besar di harga Rp 33. | ❌ FAIL | ████████ 1.00 | supports 0.00 / not_found 0.00 / contradicts 1.00 · details 0.01 · missing #s: 33 | `examples/news-1.md:25` "Saham ini juga baru menyelesaikan rights issue dalam juml…" |
| 4 | Saham VIVA disuspensi di harga Rp 50. | ❌ FAIL | ████████ 1.00 | supports 0.00 / not_found 0.00 / contradicts 1.00 · details 0.02 |  |
| 5 | BUMI hanya perlu turun sekitar 10% untuk menyentuh level Rp 50. | ❌ FAIL | ████████ 1.00 | supports 0.00 / not_found 0.00 / contradicts 1.00 · details 0.01 · missing #s: 10 | `examples/news-1.md:23` "Adapun PT Bumi Resources Tbk (BUMI) di sekitar Rp192 haru…" |
| 6 | BEI juga menetapkan batas atas harga saham Rp 5.000 per saham di pasar reguler. | ⚠️ REVIEW | ████████ 1.00 | supports 0.00 / not_found 1.00 / contradicts 0.00 · details 0.02 · missing #s: 5.000 |  |
| 7 | Para investor ritel sangat senang dengan aturan baru ini. | ➖ UNCHECKABLE | ██░░░░░░ 0.25 | P(checkable)=0.25 |  |

## Needs review

- line 13: BEI juga menetapkan batas atas harga saham Rp 5.000 per saham di pasar reguler.
- line 15: Para investor ritel sangat senang dengan aturan baru ini.
```

Every verdict carries its probability distribution and the source span it was judged against. When Sooth is unsure, it says `REVIEW` instead of guessing.

## Why

AI writes fast, nobody checks. Claims ship wrong — usually a number, a hedge, or a name that drifted. Asking another LLM "is this right?" just produces more prose.

Sooth takes a different bet: **the model makes typed judgments, code makes the decision.** Every rule that turns a probability into a verdict is a readable line of Python, not a prompt.

## Install

```bash
pip install sooth
export TYPESAFE_API_KEY=...       # get one at console.typesafe.ai
```

## Usage

```bash
sooth --source docs/policy.md --source tickets/t123.md --text draft-reply.md

# CI-friendly exit codes
sooth --source policy.md --text draft.md --format plain
#   0 = clean  ·  1 = any FAIL  ·  2 = any REVIEW  ·  3 = usage/config error

# Options
#   --confidence T   REVIEW below this confidence (default 0.7)
#   --format md|plain
#   -o FILE          write report to file
#   --log FILE       append full judgment trace (one JSONL line per run)
```

Use it in CI as a quality gate for generated content:

```yaml
- name: Verify AI output
  uses: naufalhilmiaji/sooth@v0.2.3
  with:
    source: docs/policy.md
    text: generated-reply.md
  env:
    TYPESAFE_API_KEY: ${{ secrets.TYPESAFE_API_KEY }}
```

`source` takes newline-separated paths when you have several documents. The step fails the build on any `FAIL` — and on any `REVIEW`, so add `continue-on-error: true` if you only want to block on hard contradictions. The full report is appended to the job summary.

Prefer plain shell? `sooth --source policy.md --text draft.md` gives the same exit codes.

## How it works

1. Draft is split into claims (one sentence each).
2. Each claim gets four questions to Jev, fanned out in parallel batches: *is this checkable?*, *does the source support it?* (`supports` / `contradicts` / `not_found`), *do all details match exactly?*, and *which source span is the evidence?*
3. Verdicts are mapped in code: uncheckable → `UNCHECKABLE`; low confidence → `REVIEW`; then `PASS` / `FAIL`. Safeguards demote `PASS` to `REVIEW` when details drift or claim numbers are absent from the source (checked in plain code).
4. The report shows the full probability distribution per claim — not just a label.

The decision logic lives in [`src/sooth/verify.py`](https://github.com/naufalhilmiaji/sooth/blob/main/src/sooth/verify.py) in a dozen readable lines. Change thresholds and rules there, not in prompts.

Judgments come from [Jev](https://docs.typesafe.ai) (TypeSafe System One), which returns typed distributions instead of generated prose.

## Why not just use an LLM judge?

Not claiming Sooth is universally more accurate. Claiming it gives you **architectural guarantees** a prompt cannot:

| | Sooth | Generic LLM judge |
|---|-------|-------------------|
| Claim-level verification | ✅ | Sometimes |
| Source span behind each verdict | ✅ | Sometimes |
| `PASS` / `FAIL` / `REVIEW` | ✅ fixed vocabulary | Generated labels |
| Probability distribution | ✅ | Usually none |
| Verdict rules in readable code | ✅ | Prompt-dependent |
| CI exit codes | ✅ | ❌ |
| Audit log (`--log` JSONL) | ✅ | DIY |

## Who it's for

Developers building systems that generate text from trusted material:

- **RAG pipelines** — retrieved the right document, still wrote the wrong number
- **AI agents** — before an agent's reply reaches a human
- **Support / ops copilots** — policy-backed answers, checked against the policy
- **Report & summary generation** — figures that must match the source
- **Compliance workflows** — evidence trail for every claim, not vibes

## Known limits (alpha)

- Safeguards catch most drift and smuggling: claim numbers absent from the source demote `PASS` to `REVIEW`, and a detail-match gate flags altered hedges ("about 74%" → "over 74%"). Not perfect — read the `Why` column before trusting a verdict.
- Derived numbers (totals, values computed outside the source) look "missing" and land in `REVIEW`.
- Source + questions must fit ~64k tokens — split long documents yourself.

## Development

```bash
pip install -e ".[dev]"
python3 tests/test_core.py     # pure checks, no network
pytest                         # same suite
bash tests/smoke.sh            # live smoke (needs TYPESAFE_API_KEY)
PYTHONPATH=src python3 tests/calibrate.py   # live calibration, 30 labeled claims
```

Docs: [PRD](https://github.com/naufalhilmiaji/sooth/blob/main/docs/PRD.md) · [Design](https://github.com/naufalhilmiaji/sooth/blob/main/docs/DESIGN.md) · [Testing](https://github.com/naufalhilmiaji/sooth/blob/main/docs/TESTING.md)

## Roadmap

- v0.2.3 shipped: GitHub Action — CI quality gate in one `uses:` line
- v0.2.2 shipped: `sooth demo` (offline, no API key)
- v0.2.1 shipped: source-span evidence, published on PyPI
- later: hosted web app — paste UI, history, team review queues

## License

MIT © Naufal Hilmiaji. Powered by [TypeSafe](https://typesafe.ai) / Jev.
