Metadata-Version: 2.4
Name: nkama-fact-benchmark
Version: 0.1.0
Summary: Evidence-gated benchmark for testing whether AI assistants can prove what they claim.
Author: KK Nkama
Keywords: ai,benchmark,evidence,verification,prompt-engineering,agents
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Testing
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.9
Description-Content-Type: text/markdown

# Nkama Fact Benchmark

Evidence-gated tools for testing whether an AI assistant can prove what it claims.

The package gives you three public command-line tools:

```bash
nkama-fact-benchmark
nkama-prompt-filter
nkama-evidence-layer
nkama-truth-filter
```

It is designed for use through `uvx`:

```bash
uvx nkama-fact-benchmark --profile public
uvx nkama-fact-benchmark prompt "Build a browser game with tests."
uvx nkama-prompt-filter "Build a browser game with tests." --output prompt_check
```

Before publishing or sharing a built package, audit the release files:

```bash
nkama-fact-benchmark security-audit dist/*.whl dist/*.tar.gz
```

## What It Does

Nkama Fact Benchmark does not promise that an AI is always right. It makes AI work more testable by asking for evidence, running local checks, and marking unavailable proof as blocked instead of pretending it passed.

Typical flow:

```text
raw prompt
  -> evidence-wrapped prompt
  -> AI answer or generated files
  -> evidence manifest / validator
  -> pass, fail, or blocked report
```

## Prompt Filter

Use the prompt filter before sending a task to an AI:

```bash
nkama-prompt-filter "Build a browser game with tests." --output prompt_check
```

This writes:

```text
prompt_check/
  original_prompt.md
  evidence_prompt.md
  prompt_analysis.json
  README.md
```

Paste `evidence_prompt.md` into your AI assistant.

## Python Library

```python
from nkama_fact_benchmark.prompt_filter import analyze_prompt, wrap_prompt, write_prompt_package

prompt = "Build a browser game with tests."
analysis = analyze_prompt(prompt)
evidence_prompt = wrap_prompt(prompt)
write_prompt_package(prompt=prompt, output_dir="prompt_check")
```

## Evidence Layer

If an AI generates files, ask it to include an `evidence_manifest.json`, then verify it:

```bash
nkama-evidence-layer path/to/evidence_manifest.json
nkama-evidence-layer path/to/evidence_manifest.json --allow-commands
```

Command checks are disabled unless you explicitly pass `--allow-commands`.

## Truth Filter

Use the truth filter to compare multiple AI submissions against the same task:

```bash
nkama-truth-filter init "Browser Game Comparison" --template evidence
nkama-truth-filter run browser-game-comparison
```

## Public Safety Defaults

The public profile is designed to be portable:

- no private documents are read by default
- no external model calls are made by default
- no shell commands run unless explicitly allowed
- blocked evidence is not counted as success
- reports are written as JSON and Markdown
- release artifacts can be audited for private paths, internal package names, unexpected commands, and dependencies

Private/local profiles can be used for a specific developer's own machine, but those checks are opt-in.

## Status

This package is alpha software. It is useful for evidence-gated AI workflows, prompt testing, and local verification experiments. It is not a guarantee of truth, correctness, safety, legal validity, or production readiness.
