Metadata-Version: 2.4
Name: attestq
Version: 0.4.0
Summary: Answer security questionnaires and compliance attestations from your evidence — a model-agnostic RAG kernel.
Author: Vinay Vobbilichetty
License: MIT
Project-URL: Homepage, https://github.com/vinayvobbili/attestq
Project-URL: Repository, https://github.com/vinayvobbili/attestq
Project-URL: Issues, https://github.com/vinayvobbili/attestq/issues
Keywords: security,compliance,rag,questionnaire,vendor-risk,third-party-risk,due-diligence,audit,attestation,llm
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Information Technology
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Security
Classifier: Topic :: Office/Business
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: chroma
Requires-Dist: chromadb>=0.4; extra == "chroma"
Provides-Extra: openai
Requires-Dist: openai>=1.0; extra == "openai"
Provides-Extra: ollama
Requires-Dist: requests>=2.28; extra == "ollama"
Provides-Extra: rerank
Requires-Dist: sentence-transformers>=2.2; extra == "rerank"
Provides-Extra: loaders
Requires-Dist: pypdf>=4.0; extra == "loaders"
Requires-Dist: python-docx>=1.1; extra == "loaders"
Requires-Dist: openpyxl>=3.1; extra == "loaders"
Provides-Extra: export
Requires-Dist: python-docx>=1.1; extra == "export"
Provides-Extra: yaml
Requires-Dist: pyyaml>=6.0; extra == "yaml"
Provides-Extra: all
Requires-Dist: chromadb>=0.4; extra == "all"
Requires-Dist: openai>=1.0; extra == "all"
Requires-Dist: requests>=2.28; extra == "all"
Requires-Dist: sentence-transformers>=2.2; extra == "all"
Requires-Dist: pypdf>=4.0; extra == "all"
Requires-Dist: python-docx>=1.1; extra == "all"
Requires-Dist: openpyxl>=3.1; extra == "all"
Requires-Dist: pyyaml>=6.0; extra == "all"
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: pytest-cov; extra == "dev"
Requires-Dist: build; extra == "dev"
Requires-Dist: twine; extra == "dev"
Requires-Dist: ruff; extra == "dev"
Dynamic: license-file

# attestq

**Answer security questionnaires and compliance attestations from your own evidence.**

`attestq` is a small, model-agnostic RAG kernel for a problem every security and
GRC team has: you have a *questionnaire* (a vendor security review, a SIG/CAIQ
response, an audit-evidence request, a due-diligence form, the security section
of an RFP) and you have a *pile of evidence* (SOC 2 reports, policies, standards,
prior questionnaires). `attestq` retrieves the relevant evidence for each
question and drafts a grounded, **cited** answer — and, just as importantly, tells
you plainly when the evidence isn't there.

It is **not** a classic ML / training problem. It's retrieval + an LLM you bring
yourself. attestq owns the orchestration; you own the model, the embedder, and
the store.

## Why it exists

The same pattern keeps getting rebuilt one-off inside every program: chunk the
docs, embed them, retrieve per question, prompt an LLM, paste into a form. Done
naively it hallucinates (answers with no supporting evidence) and it silently
drops the one focused document that actually had the answer. attestq bakes in the
two hard-won fixes:

- **Confidence gate.** When the best retrieved evidence scores below a threshold,
  the question is answered *"insufficient evidence"* **without calling the LLM**.
  Absence of evidence is a first-class, valid result — not an invitation to guess.
- **Wide rerank window.** The kernel keeps a generous number of chunks after
  reranking so a single relevant document isn't dropped on a small corpus.

## Install

```bash
pip install attestq                 # dependency-free core
pip install "attestq[chroma]"       # + persistent Chroma vector store
pip install "attestq[openai]"       # + OpenAI-compatible chat adapter
pip install "attestq[ollama]"       # + local Ollama embeddings
pip install "attestq[rerank]"       # + cross-encoder reranker
pip install "attestq[loaders]"      # + pdf / docx / xlsx loaders
pip install "attestq[all]"          # everything
```

The core has **zero** third-party dependencies. Adapters are opt-in extras.

## Quick start

You inject any chat model and any embedder as plain callables:

```python
from attestq import Engine, Question

# Bring your own model + embedder (one-liners around any provider).
def my_chat(prompt: str) -> str:
    ...   # call OpenAI, Anthropic, a local model, your corporate gateway, ...

def my_embed(texts):
    ...   # return one vector per text

engine = Engine(chat=my_chat, embed=my_embed)   # in-memory store by default

# Ingest a vendor's evidence into its own namespace.
engine.ingest(
    [
        ("All customer data at rest is encrypted with AES-256...", {"source": "DataProtection.pdf"}),
        ("MFA is enforced for all privileged access...", {"source": "AccessControl.docx"}),
    ],
    namespace="helios",
)

# Answer one control.
ans = engine.evaluate(
    Question(
        id="ENC-1",
        prompt="Is customer data encrypted at rest?",
        choices=["Met", "Not Met", "Not Applicable"],
    ),
    namespace="helios",
)

print(ans.determination)          # "Met"
print(ans.confidence)             # 0.0 - 1.0 retrieval confidence
print(ans.insufficient_evidence)  # False
for c in ans.citations:
    print(c.source, "->", c.snippet)
```

Run a whole questionnaire:

```python
from attestq import Questionnaire

qn = Questionnaire(
    id="vendor-sec-review",
    title="Vendor Security Review",
    questions=[
        Question(id="ENC-1", prompt="Is data encrypted at rest?", choices=["Met", "Not Met", "Not Applicable"]),
        Question(id="IAM-1", prompt="Is MFA enforced for privileged access?", choices=["Met", "Not Met", "Not Applicable"]),
    ],
)

answers = engine.evaluate_all(qn, namespace="helios", on_answer=lambda a: print(a.question_id, a.determination))
```

## Command line

Installing attestq gives you an `attestq` command. Run the bundled sample, or
point it at your own questionnaire and evidence:

```bash
# Run the bundled fictional sample assessment end-to-end
attestq demo -o report.md

# Evaluate your questionnaire against a folder of evidence
attestq run -q questionnaire.yaml -e ./vendor-evidence -n acme -o report.docx

# Pipe JSON to another tool
attestq run -q q.json -e ./evidence --format json | jq .summary
```

Providers are resolved from flags or environment, so the same command works
against OpenAI-compatible endpoints or a local Ollama:

```bash
export OPENAI_API_KEY=sk-...                 # uses OpenAI by default
attestq demo

attestq demo --provider ollama               # local, no key, nothing leaves the host
attestq run -q q.yaml -e ./ev --provider openai --base-url https://my-gateway/v1
```

## Try it instantly (no setup)

The built-in `HashEmbedder` needs no model and no service, so you can watch the
retrieval pipeline and the confidence gate work the moment you install:

```bash
pip install attestq
python examples/quickstart.py
```

See [`examples/`](examples/) for a full provider-wired demo
(`helios_demo.py`) and a minimal web UI (`examples/web/`) that runs the sample
assessment in your browser.

## How it works

```
evidence docs ──▶ chunk ──▶ embed ──▶ vector store (namespaced per corpus)

question ──▶ embed ──▶ retrieve(k) ──▶ rerank(top_k) ──▶ confidence gate
                                                              │
                              below threshold ───────────────┤──▶ "insufficient evidence" (no LLM call)
                                                              │
                              above threshold ──▶ prompt LLM ─┴──▶ parse ──▶ Answer(determination, summary, citations, confidence)
```

## Verifying the draft

Drafting is half the job; the other half is not trusting the draft. Turn on
`verify=True` and every answer carries two reports:

```python
engine = Engine(chat=my_llm, embed=my_embedder, verify=True)
answer = engine.evaluate(question, namespace="vendor-x")

if answer.needs_review:
    print(answer.grounding.unverified)   # ['2019-03-01'] — asserted, not in evidence
    print(answer.quality.detail())       # "only 20% of the answer is carried by ..."
```

**`answer.grounding`** extracts every concrete value the draft asserted — dates,
versions, percentages, standards, durations — and confirms each one actually
occurs in the evidence. This catches the highest-liability failure: *"certified
to ISO 27001 since 2019-03-01"* when neither appears in a single retrieved
document.

**`answer.quality`** catches the quieter failure — an answer that restates the
question and cites evidence it never drew on:

> **Q:** Are encryption keys securely managed through defined key-management processes?
> **A:** Yes. Encryption keys are securely managed through defined key-management processes.

That asserts nothing the question didn't already say. Per sentence, it measures
*support* (closeness to the retrieved evidence) against *echo* (closeness to the
question), each judged against the question's own similarity to that evidence —
so there is no magic cosine constant to recalibrate when you swap embedders.

Both checks are **deterministic**: no second LLM grading the first, which would
only share the drafter's blind spots. Both **flag and never rewrite** — a
flagged answer may still be the right answer, so the disposition stays with the
reviewer. Verification is off by default; it costs one extra embedding call per
answer.

One judgment measurement genuinely can't make is whether a question wanted
documentary evidence at all — *"What is your registered legal entity name?"* has
a correct answer no policy document will ever back. Inject a small model to
decide, and it may only ever **clear** a flag, never raise one:

```python
from attestq import make_claim_classifier

engine = Engine(chat=big_model, embed=embedder, verify=True,
                claim_classifier=make_claim_classifier(small_fast_model))
```

Every failure path in that classifier — endpoint down, unparseable reply —
leaves the flag standing.

## Closing the loop

Every reviewer who edits a draft before shipping it is handing you a quality
label for free — and most systems throw it away, because the final answer
overwrites the draft in place. Keep both and you can answer the question every
RAG stack should be able to answer and usually can't: *does the confidence
number mean anything?*

```python
from attestq import outcome_from_answer, build_scorecard

outcomes = [
    outcome_from_answer(answer, question, final=what_the_reviewer_shipped)
    for answer, question, what_the_reviewer_shipped in reviewed_items
]

card = build_scorecard(outcomes)
card["acceptance_rate"]   # 0.62 — shipped as drafted
card["calibrated"]        # True / False / None ("not enough data to say")
card["source_trust"]      # worst-performing documents first
```

Three things come out of it:

- **Acceptance rate** — how often a draft ships as written, with edits graded
  from cosmetic to wholesale rewrite. Items still awaiting review count as
  neither, so the number doesn't move with queue depth.
- **Calibration by confidence band** — the real test of your gate. A healthy
  profile is monotonic: acceptance climbing with confidence. Flat means the
  score is decorative and every threshold built on it is arbitrary; inverted
  means it's actively misleading. `is_calibrated` returns `None` rather than a
  verdict when the sample is too thin to support one.
- **Per-source trust** — which documents back answers that survive review and
  which back rewrites. A document that repeatedly produces rewritten answers is
  stale, ambiguous, or wrong for the questions it keeps matching. This is the
  raw material for source-authority weighting at retrieval time.

`outcome_from_answer` is duck-typed, so records from a system that never touched
attestq fold into the same scorecard as long as they expose the same handful of
attributes.

Everything is swappable:

| Piece | Default | Swap for |
|-------|---------|----------|
| Chat model | *you inject it* | any LLM / gateway |
| Embedder | *you inject it* | Ollama, sentence-transformers, OpenAI |
| Vector store | `InMemoryVectorStore` | `attestq[chroma]` |
| Reranker | none | `attestq[rerank]` cross-encoder |
| Prompt / parser | evidence-only default | your own `prompt_builder` / `response_parser` |

## Design principles

- **Bring your own model.** No provider is hard-wired. A lambda is enough.
- **Grounded or silent.** Answers cite their evidence; thin evidence yields an
  explicit "insufficient" result, never a confident guess.
- **The model proposes, deterministic code disposes.** Verification is plain
  string and vector math, so the verifier cannot hallucinate the way an
  LLM-grading-an-LLM judge can. It flags for a human; it never rewrites.
- **Per-corpus isolation.** One store, many namespaces — keep each vendor's
  evidence separate without standing up a new index each time.
- **Light core.** The kernel imports nothing third-party; heavy deps stay in
  extras you opt into.

## Status

Usable today: the core kernel, the verification and feedback layers,
in-memory + Chroma
stores, OpenAI/Ollama adapters, a cross-encoder reranker, document loaders,
JSON/Markdown/Word export, a CLI, and a web demo. Contributions and issues
welcome.

## License

MIT
