Metadata-Version: 2.5
Name: stated-confidence
Version: 0.1.0
Summary: Make LLM agents state the confidence level and the evidential basis behind each claim.
Project-URL: Homepage, https://github.com/johnwlockwood/stated-confidence
Author-email: John Lockwood <johnwlockwood@gmail.com>
License-Expression: MIT
License-File: LICENSE
Keywords: agents,confidence,grounding,hallucination,llm,pydantic,pydantic-ai
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Typing :: Typed
Requires-Python: >=3.12
Requires-Dist: pydantic>=2.5
Provides-Extra: pydantic-ai
Requires-Dist: pydantic-ai-slim>=1.0; extra == 'pydantic-ai'
Description-Content-Type: text/markdown

# stated-confidence

Make LLM agents say how sure they are, and what that certainty rests on.

Ask a model to attach a confidence **level** and an evidential **basis** to
each claim it produces and it fabricates less. The level alone is common
practice; the basis is what does the work. Choosing between "documented
fact", "expert opinion", and "educated guess" forces a provenance check the
model would otherwise skip, and a claim it cannot back gets labelled as a
guess or quietly dropped instead of dressed up as a figure.

This package is that schema, the vocabulary behind it, prompt guidance for
filling it, and the consistency checks that send hedging-without-meaning-it
back to the model. It was extracted from a production knowledge-graph
generator where it was observed to sharply reduce invented facts and
figures. The effect is observed, not yet measured; an evaluation harness is
planned.

## Install

```bash
pip install stated-confidence            # pydantic only
pip install "stated-confidence[pydantic-ai]"
```

## The shape

```python
from stated_confidence import Confidence, DataAnalysisBasis


class Finding(BaseModel):
    text: str
    confidence: Confidence[DataAnalysisBasis]
```

A `Confidence` has:

| field       | meaning                                                                   |
| ----------- | ------------------------------------------------------------------------- |
| `level`     | `HIGH`, `MEDIUM`, `LOW`, or `SPECULATIVE`. Never a number.                |
| `basis`     | The kind of evidence, from a basis set chosen for the domain.             |
| `reasoning` | One or two sentences on why that level and basis apply.                   |
| `evidence`  | What concretely backs the claim: sources, ids, column names, tool results. |
| `type`      | Optional: `FACTUAL`, `CONSENSUS`, `THEORETICAL`, `SPECULATIVE`.           |
| `label`     | Optional localized label for display.                                     |

`SPECULATIVE` is not the bottom of the scale. It is the level for an idea
the model is offering rather than a claim it is making: an extension a
source suggests, a hypothesis, a what-if. A generator that has somewhere
honest to put an idea does not dress it up as a fact; take that room away
and the ideas come back as fabricated "facts" or disappear altogether.

The level scale is fixed. The **basis set** is the extension point. Three
ship:

- `GeneralKnowledgeBasis` for claims about the world: documented fact,
  scientific consensus, historical record, industry standard, expert
  opinion, theoretical framework, fictional universe, speculation, educated
  guess.
- `DataAnalysisBasis` for agents reasoning over data they retrieved: query
  result, computed, schema inference, general knowledge, assumption.
- `DecisionBasis` for recommendations: policy, measured outcome, precedent,
  trade-off analysis, expert judgment, assumption.

Define your own by subclassing `BasisEnum` with a `__terms__` table and a
`__fallback__` value.

## Prompting

```python
from stated_confidence import confidence_instructions

system_prompt += confidence_instructions(DataAnalysisBasis, localized=True)
```

## Pydantic AI

```python
from pydantic_ai import Agent
from stated_confidence.pydantic_ai import confidence_output_validator

agent = Agent("openai:gpt-5", output_type=Report)
agent.output_validator(confidence_output_validator())
```

Every `Confidence` anywhere in the output is checked. A `HIGH` level on a
basis of `ASSUMPTION`, a `SPECULATIVE` level on a basis of `DOCUMENTED_FACT`,
or a non-`HIGH` level with empty reasoning comes back to the model as a
retry listing what to fix. Pass extra `(ctx, output) -> list[str]` callables for rules that
need run context, such as requiring that evidence names a column the
agent's query actually returned.

## Lenient parsing, strict prompting

Models drift. `"High"` parses. `0.9` parses (as `HIGH`). An invented basis
value parses as the set's fallback and is logged at `WARNING` on the
`stated_confidence` logger so the set can grow from what models actually
say.

## Display

`describe(value)` returns a label and description for any level, type, or
basis value; `tone(level)` maps to `positive`, `neutral`, `caution`, or
`warning` with no CSS framework attached. `taxonomy.json` at the package
root is the same vocabulary as data, for other languages to consume.
