Metadata-Version: 2.4
Name: dummybot
Version: 0.1.0
Summary: A scripted LLM test double that verifies what your pipeline feeds the model, not just what it does with the response.
Author: Usman
License-Expression: MIT
Project-URL: Homepage, https://github.com/Sicatho/dummybot
Project-URL: Issues, https://github.com/Sicatho/dummybot/issues
Keywords: llm,testing,pytest,mock,test-double,prompt,ollama,openai
Classifier: Development Status :: 4 - Beta
Classifier: Framework :: Pytest
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Topic :: Software Development :: Testing
Classifier: Topic :: Software Development :: Testing :: Mocking
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: dev
Requires-Dist: pytest>=7; extra == "dev"
Dynamic: license-file

# dummybot

**A scripted LLM test double that verifies what your pipeline *feeds* the model — not just what it does with the response.**

Most LLM mocking looks like this:

```python
def fake_model(prompt):
    return "42"
```

That proves your code can handle `"42"`. It proves nothing about the prompt: did the retrieved context actually make it in? Did the system instructions survive your template refactor? Did an internal debug blob leak into the model's input? In LLM pipelines, most real bugs are **feed bugs** — the model was called with the wrong thing — and a return-only fake is blind to all of them.

Dummybot treats the model seam as a contract boundary:

- **Trigger-keyed dispatch** — each scripted line answers only prompts containing its trigger substring, in script order, with per-line use limits.
- **Prompt input assertions** — required substrings, forbidden substrings, and any-of groups checked against the *actual* prompt your pipeline sent.
- **Typed faults, never silent drift** — an unmatched call, a failed assertion, or a partially-consumed script raises a structured error naming exactly what went wrong (with a prompt digest and the available triggers).
- **Completion contract** — assert that every expected model call actually happened; the pytest fixture does this automatically at teardown.
- **Scripted failures** — lines can simulate provider errors (timeouts, refusals) so you can test your error handling deterministically.
- **Zero dependencies** — pure standard library; pytest is only needed for the optional fixture.

## Install

```bash
pip install dummybot
```

## Quickstart

```python
from dummybot import Dummybot, PromptAssertion, Script, ScriptLine

script = Script(
    name="rag-happy-path",
    lines=(
        ScriptLine(
            line_id="answer",
            trigger="USER QUESTION:",           # matches as a substring of the prompt
            response="Paris",
            assertions=(
                PromptAssertion.requires("RETRIEVED CONTEXT:", "Eiffel"),
                PromptAssertion.forbids("INTERNAL_DEBUG"),
            ),
        ),
    ),
)

bot = Dummybot(script)

# Patch it in wherever your pipeline calls its model provider:
answer = my_rag_pipeline(question="Where is the Eiffel Tower?", model_call=bot)

assert answer == "Paris"
bot.assert_all_expected_calls_consumed()   # every expected model call happened
```

If the pipeline drops the retrieved context on the floor, you don't get a
mysterious wrong answer — you get:

```
DummybotFault: [input_assertion_failed] input_assertion_failed
  assertion_label: 'required_substrings'
  line_id: 'answer'
  missing_required_substrings: ('RETRIEVED CONTEXT:', 'Eiffel')
  prompt_digest: 'sha256:…'
```

## The pytest fixture

Installing the package registers a `dummybot` fixture (a factory). Bots built
through it are automatically completion-checked at test teardown:

```python
def test_pipeline_calls_planner_then_answerer(dummybot):
    bot = dummybot({
        "name": "two-step",
        "lines": [
            {"line_id": "plan", "trigger": "PLAN THE TASK", "response": '{"steps": ["look up"]}'},
            {"line_id": "answer", "trigger": "FINAL ANSWER", "response": "done",
             "assertions": [{"required_substrings": ["steps"]}]},
        ],
    })
    run_pipeline(model_call=bot)
    # teardown fails the test if either line was never used
```

Opt out per-bot with `dummybot(script, autocheck=False)`.

## Scripts as JSON

Scripts are plain data — keep them next to your tests and load them:

```python
from dummybot import load_script
bot = Dummybot(load_script("tests/scripts/rag_happy_path.json"))
```

```json
{
  "schema": "dummybot_script.v1",
  "name": "rag-happy-path",
  "lines": [
    {
      "line_id": "answer",
      "trigger": "USER QUESTION:",
      "response": "Paris",
      "max_uses": 1,
      "assertions": [
        {"required_substrings": ["RETRIEVED CONTEXT:"], "forbidden_substrings": ["INTERNAL_DEBUG"]}
      ]
    }
  ]
}
```

## Simulating model failures

```python
script = Script(name="outage", lines=(
    ScriptLine(line_id="down", trigger="USER QUESTION:", fault_type="model_unavailable"),
))

bot = Dummybot(script)                      # raises ScriptedModelError on match
bot = Dummybot(script,                      # or raise the exception type your seam catches
               fault_factory=lambda line, prompt: TimeoutError("scripted outage"))
```

## Shaping the return value for your seam

By default a matched line returns plain text (`raw_response` if set, else
`response` — use `raw_response` to include `<think>` blocks your seam is
supposed to strip). If your model-call seam expects a richer object, pass a
`response_factory`:

```python
def ollama_shaped(line, prompt, kwargs):
    return {
        "response": line.response,
        "model": kwargs.get("model", "qwen3:4b"),
        "prompt_eval_count": 0,
        "eval_count": 0,
    }

bot = Dummybot(script, response_factory=ollama_shaped)
```

Helpers for common response shapes:

```python
from dummybot import json_response, thinking_json_response

json_response({"answer": 42})                       # deterministic sorted-key JSON
thinking_json_response("let me think", {"answer": 42})  # <think>…</think> + JSON
```

## Semantics worth knowing

- **Matching order**: first line in script order whose trigger is in the prompt
  and that has uses remaining. Exhausted lines fall through to later lines —
  so two lines with the same trigger model a two-step conversation.
- **`max_uses`** is both a cap and an expectation: `max_uses=2` means "matches
  at most twice, and the completion contract expects exactly two uses."
  `max_uses=0` means unlimited and optional.
- **Failed assertions don't consume the line** — fix the feed and the same
  line is still there.
- **Scripted fault lines do consume the line** — the call happened; it's
  recorded in `bot.calls` and satisfies the completion contract.
- **`bot.calls`** records every matched call: line id, full prompt, prompt
  digest, and the kwargs your pipeline passed (model name, temperature, …).

## Origin

Dummybot was extracted from the test infrastructure of **Tamago**, a
local-first LLM orchestration runtime, where it replaces the model at the
Ollama caller seam so that ~2,800 deterministic tests can pin down not just
what the pipeline does with model output, but what every stage feeds the
model. The feed-assertion idea earned its keep there — most of the bugs it
caught were prompts that silently lost or leaked context.

## License

MIT
