Metadata-Version: 2.4
Name: nkama-fact-benchmark
Version: 0.1.12
Summary: Evidence-gated benchmark for testing whether AI assistants can prove what they claim.
Author: KK Nkama
License-Expression: Apache-2.0
Keywords: ai,benchmark,evidence,verification,prompt-engineering,agents
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Testing
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Dynamic: license-file

# Nkama Fact Benchmark

Evidence-gated tools for testing whether an AI assistant can prove what it claims.

The package gives you four public command-line tools:

```bash
nkama-fact-benchmark
nkama-prompt-filter
nkama-evidence-layer
nkama-truth-filter
```

It is designed for use through `uvx`:

```bash
uvx nkama-fact-benchmark
uvx nkama-fact-benchmark intro
uvx nkama-fact-benchmark activate
uvx nkama-fact-benchmark browser-benchmark
uvx nkama-fact-benchmark inspect path/to/nkama_run
uvx nkama-fact-benchmark selftest
uvx nkama-fact-benchmark agent
uvx nkama-fact-benchmark agent-run "Create a Cambridge A2/B1 past perfect lesson plan." --provider claude --allow-external-model
uvx nkama-fact-benchmark agent-run "Build a tiny tested Python project." --provider claude --allow-external-model --allow-claude-tools --allow-command "python3 -m unittest *"
uvx nkama-fact-benchmark start
uvx nkama-fact-benchmark prompt "Build a browser game with tests."
uvx nkama-fact-benchmark run "Build a browser game with tests." --output nkama_run_browser_game
uvx --from nkama-fact-benchmark nkama-prompt-filter "Build a browser game with tests." --output prompt_check
```

Before publishing or sharing a built package, audit the release files:

```bash
nkama-fact-benchmark security-audit dist/*.whl dist/*.tar.gz
```

## What It Does

Nkama Fact Benchmark does not promise that an AI is always right. It makes AI work more testable by asking for evidence, running local checks, and marking unavailable proof as blocked instead of pretending it passed.

Typical flow:

```text
raw prompt
  -> evidence-wrapped prompt
  -> AI answer or generated files
  -> evidence manifest / validator
  -> pass, fail, or blocked report
```

## Public Introduction

Run the package with no subcommand when you want the tool to introduce itself:

```bash
uvx nkama-fact-benchmark
```

This prints a stable public identity for the tool: what it does, what it does not promise, the core workflow, and the safety rule that blocked evidence is not success.

The first run does not build anything by itself. It introduces the protocol and waits for a task or command, which helps preserve context for the actual work.

Use `activate` when an AI chat or agent session should treat Nkama as the working protocol:

```bash
uvx nkama-fact-benchmark activate
```

The activation text tells the assistant to ask for the user's task, verify claims with tools where possible, mark unavailable evidence as blocked, and keep the protocol active across the session.

If the AI has sandbox file storage, the activation/agent protocol asks it to keep a small `NKAMA_SESSION_STATE.md` file with the active task, files, checks, and open limitations. This is a reminder inside that sandbox, not a guarantee of permanent memory after the environment resets.

Use `browser-benchmark` when you want to test whether an AI browser/chat sandbox reports terminal evidence honestly:

```bash
uvx nkama-fact-benchmark browser-benchmark
```

It prints a copy-paste test with two real commands and one intentional fake command trap. A good AI should say what it actually ran, quote or summarize real terminal output, reject the fake command as invalid, and avoid inventing datasets, API keys, judges, browser engines, hidden services, or remote endpoints.

Use `inspect` when an AI has already generated a run folder and you want Nkama to explain what it actually is:

```bash
uvx nkama-fact-benchmark inspect path/to/nkama_run
```

`inspect` classifies the folder as values such as `design_only`, `working_document`, `working_code_unverified`, `verified_build`, `fake_evidence`, `incomplete`, `failed_evidence`, or `blocked`. This is useful when an AI creates a folder full of Markdown and JSON and you need to know whether it is only a design, a working artifact, or a verified build.

Run the package self-test when you want machine-readable proof that the public package checks are working:

```bash
uvx nkama-fact-benchmark selftest
```

## Agent Protocol

Use `agent` when an AI coding agent has terminal access and should treat Nkama Fact Benchmark as its working protocol:

```bash
uvx nkama-fact-benchmark agent
```

With no task, it prints the protocol an AI agent should follow. With a task, it prepares the evidence workspace and writes `AGENT_PROTOCOL.md`:

```bash
uvx nkama-fact-benchmark agent "Create a Cambridge A2/B1 past perfect lesson plan." --output nkama_agent_lesson
```

The AI agent should read `AGENT_PROTOCOL.md` and `evidence_prompt.md`, build in `ai_output/`, update `ai_output/evidence_manifest.json`, run `nkama-evidence-layer`, and report pass/fail/blocked honestly.

## Agent Run

Use `agent-run` when you want Nkama Fact Benchmark to call an external model through a local provider CLI and capture the answer:

```bash
uvx nkama-fact-benchmark agent-run "Create a Cambridge A2/B1 past perfect lesson plan." --provider claude --allow-external-model --output nkama_agent_lesson
```

The first public provider is `claude`, using the local Claude CLI. External model calls are blocked unless you pass `--allow-external-model`. By default Claude receives no tools; this public mode is text-only.

For controlled agent work, you can explicitly enable scoped Claude tools:

```bash
uvx nkama-fact-benchmark agent-run \
  "Build a tiny tested Python project." \
  --provider claude \
  --allow-external-model \
  --allow-claude-tools \
  --allowed-dir ./my_project \
  --allow-command "python3 -m unittest *" \
  --max-budget-usd 0.50 \
  --timeout-seconds 120 \
  --output nkama_agent_build
```

Tool mode writes a permission contract into `AGENT_PROTOCOL.md`:

```text
This mode may grant Claude/Codex tools.
Allowed directories: ...
Allowed commands: ...
Allowed external model: ...
Allowed browser/MCP tools: ...
Budget cap: ...
```

If the task needs a directory, command, browser/MCP tool, credential, private file, or external service that was not allowed, the provider must ask for that exact permission and the run should remain blocked until the user grants it. The public package does not enable unlimited permissions by default.

Use `--timeout-seconds` to cap wall-clock runtime. Use `--max-budget-usd` to cap Claude CLI API spend. Nkama treats timeout, budget exhaustion, missing auth, denied permission, and unavailable evidence as blocked/failed states, not success.

The package captures the provider's text answer, then Nkama composes the final `ai_output/ANSWER.md`. This matters: the provider reports only model-level answer/evidence/limitations, while the Nkama runner owns the real file and verification sections. The runner writes `MODEL_RUN_REPORT.json`, verifies `ai_output/evidence_manifest.json`, and records the evidence summary in the final answer.

If the provider is not logged in, asks for unavailable tools, omits the required provider sections, or otherwise fails the contract, the run is marked `blocked` or `fail` instead of being treated as verified.

If you do not pass `--allow-external-model`, the workspace is still created, but the model run is marked blocked instead of pretending it happened.

## Start For Normal Users

Use `start` when you want the tool to ask for your prompt:

```bash
uvx nkama-fact-benchmark start
```

It asks what you want the AI to build, answer, or verify. Then it creates a run folder containing the AI-ready evidence prompt, starter output folder, evidence manifest, and verification instructions.

You can also pass the prompt directly:

```bash
uvx nkama-fact-benchmark start "Build a browser game with tests." --output nkama_run_browser_game
```

## Run Folder

Use the run command when you want a complete folder for one AI task:

```bash
uvx nkama-fact-benchmark run "Build a browser game with tests." --output nkama_run_browser_game
```

This writes:

```text
nkama_run_browser_game/
  original_prompt.md
  evidence_prompt.md
  prompt_analysis.json
  run_contract.json
  README.md
  ai_output/
    ANSWER.md
    evidence_manifest.json
```

Paste `evidence_prompt.md` into the AI assistant, put the generated files in `ai_output/`, update `ai_output/evidence_manifest.json`, then verify:

```bash
uvx --from nkama-fact-benchmark nkama-evidence-layer nkama_run_browser_game/ai_output/evidence_manifest.json
```

Then inspect the whole folder:

```bash
uvx nkama-fact-benchmark inspect nkama_run_browser_game
```

## Prompt Filter

Use the prompt filter before sending a task to an AI:

```bash
uvx --from nkama-fact-benchmark nkama-prompt-filter "Build a browser game with tests." --output prompt_check
```

This writes:

```text
prompt_check/
  original_prompt.md
  evidence_prompt.md
  prompt_analysis.json
  README.md
```

Paste `evidence_prompt.md` into your AI assistant.

## Python Library

```python
from nkama_fact_benchmark.prompt_filter import analyze_prompt, wrap_prompt, write_prompt_package

prompt = "Build a browser game with tests."
analysis = analyze_prompt(prompt)
evidence_prompt = wrap_prompt(prompt)
write_prompt_package(prompt=prompt, output_dir="prompt_check")
```

## Evidence Layer

If an AI generates files, ask it to include an `evidence_manifest.json`, then verify it:

```bash
uvx --from nkama-fact-benchmark nkama-evidence-layer path/to/evidence_manifest.json
uvx --from nkama-fact-benchmark nkama-evidence-layer path/to/evidence_manifest.json --allow-commands
```

Command checks are disabled unless you explicitly pass `--allow-commands`.

## Truth Filter

Use the truth filter to compare multiple AI submissions against the same task:

```bash
uvx --from nkama-fact-benchmark nkama-truth-filter init "Browser Game Comparison"
uvx --from nkama-fact-benchmark nkama-truth-filter run browser-game-comparison
```

## Public Safety Defaults

The public profile is designed to be portable:

- no private documents are read by default
- no external model calls are made by default
- no shell commands run unless explicitly allowed
- blocked evidence is not counted as success
- reports are written as JSON and Markdown
- release artifacts can be audited for private paths, internal package names, unexpected commands, and dependencies

Private/local profiles can be used for a specific developer's own machine, but those checks are opt-in.

## Status

This package is alpha software. It is useful for evidence-gated AI workflows, prompt testing, and local verification experiments. It is not a guarantee of truth, correctness, safety, legal validity, or production readiness.

License: Apache-2.0.
