Metadata-Version: 2.4
Name: robotframework-chat
Version: 1.18.0
Summary: Robot Framework-based test harness for systematically testing LLMs
Project-URL: Homepage, https://github.com/tkarcheski/robotframework-chat
Project-URL: Repository, https://github.com/tkarcheski/robotframework-chat
Project-URL: Issues, https://github.com/tkarcheski/robotframework-chat/issues
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: benchmark,llm,ollama,openai,robotframework,testing
Classifier: Development Status :: 3 - Alpha
Classifier: Framework :: Robot Framework
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.11
Requires-Dist: certifi==2025.11.12
Requires-Dist: charset-normalizer==3.4.4
Requires-Dist: docker==7.1.0
Requires-Dist: idna==3.11
Requires-Dist: iniconfig==2.3.0
Requires-Dist: packaging==25.0
Requires-Dist: pluggy==1.6.0
Requires-Dist: pygments==2.19.2
Requires-Dist: pytest==9.0.2
Requires-Dist: python-dotenv>=1.2.2
Requires-Dist: pyyaml>=6.0.3
Requires-Dist: redis>=5.0
Requires-Dist: requests==2.32.5
Requires-Dist: robotframework-lint==1.1
Requires-Dist: robotframework-requests>=0.9.7
Requires-Dist: robotframework==7.4.1
Requires-Dist: urllib3==2.6.2
Provides-Extra: dev
Requires-Dist: build; extra == 'dev'
Requires-Dist: fakeredis>=2.20; extra == 'dev'
Requires-Dist: mypy==1.10.0; extra == 'dev'
Requires-Dist: pip-audit; extra == 'dev'
Requires-Dist: pre-commit; extra == 'dev'
Requires-Dist: pytest-cov; extra == 'dev'
Requires-Dist: python-dotenv; extra == 'dev'
Requires-Dist: ruff==0.14.10; extra == 'dev'
Requires-Dist: twine; extra == 'dev'
Requires-Dist: types-docker; extra == 'dev'
Requires-Dist: types-pyyaml; extra == 'dev'
Requires-Dist: types-requests; extra == 'dev'
Provides-Extra: playwright
Requires-Dist: markdownify>=0.14.1; extra == 'playwright'
Requires-Dist: robotframework-browser>=18.0.0; extra == 'playwright'
Provides-Extra: superset
Requires-Dist: psycopg2-binary>=2.9; extra == 'superset'
Requires-Dist: sqlalchemy>=2.0; extra == 'superset'
Provides-Extra: swebench
Requires-Dist: datasets>=2.19.0; extra == 'swebench'
Requires-Dist: swebench>=2.1.0; extra == 'swebench'
Description-Content-Type: text/markdown

# robotframework-chat

[![Try Claude Code — free guest pass](https://img.shields.io/badge/Claude%20Code-free%20guest%20pass-D97757?logo=claude&logoColor=white)](https://claude.ai/referral/4Q8ajgGjHQ)
[![Project wiki on DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/tkarcheski/robotframework-chat)

> This repo is built with [Claude Code](https://claude.com/claude-code). Grab a guest pass above to try it yourself.

A Robot Framework-based test harness for systematically testing Large Language Models (LLMs) using LLMs as both the system under test and as automated graders. Test results are archived to SQL and visualized in Apache Superset dashboards.

---

## Quick Start

### Prerequisites

- **Python 3.11+** and **astral-uv** for dependency management
- **Docker** for containerized code execution, LLM testing, and the Superset stack
- **Ollama** (optional) for local LLM testing

### Installation (Linux / macOS)

```bash
make install                # Install all dependencies (runs uv sync with extras)
pre-commit install          # Install pre-commit hooks
ollama pull qwen3:32b       # Pull default LLM model (optional)
```

### Installation (Windows)

The `tasks.py` script provides a cross-platform alternative to the Makefile.
It requires only Python and `uv` — no `make`, `bash`, or Unix tools needed.

```powershell
uv run python tasks.py install      # Install all dependencies
uv run pre-commit install           # Install pre-commit hooks
ollama pull qwen3:32b               # Pull default LLM model (optional)
uv run python tasks.py help         # List all available targets
```

> **Note:** Docker-based tests require [Docker Desktop for Windows](https://docs.docker.com/desktop/install/windows-install/) with the WSL 2 backend enabled.

### Running Tests

```bash
# Linux / macOS
make robot-dryrun           # Validate all suites parse (no LLM calls — fast sanity check)
make robot                  # Run all Robot Framework test suites
make robot-math             # Run math tests
make robot-docker           # Run Docker tests
make robot-safety           # Run safety tests

# All platforms (including Windows)
uv run python tasks.py robot        # Run all suites
uv run python tasks.py robot-math   # Run math tests
uv run python tasks.py robot-dryrun # Validate tests (dry run)
uv run python tasks.py check        # Lint + typecheck + coverage
```

### Superset Dashboard

```bash
# Linux / macOS
cp .env.example .env        # Configure environment
make docker-up              # Start PostgreSQL + Redis + Superset
make bootstrap              # First-time Superset initialization

# Windows — tasks.py copies .env automatically if missing
uv run python tasks.py docker-up
```

Open <http://localhost:8088> to view the dashboard.

---

## Ollama Configuration

### Pulling Models

The default model is `qwen3:32b` (set via `DEFAULT_MODEL` in `.env`).
Pull additional models depending on how many you want to test against:

**Starter (3 models):**

```bash
ollama pull qwen3:32b
ollama pull llama3.2:latest
ollama pull gemma2:latest
```

**Standard (4–5 models):**

```bash
ollama pull qwen3:32b
ollama pull llama3.2:latest
ollama pull gemma2:latest
ollama pull mistral:latest
ollama pull qwen3.5:27b
```

**Full fleet** — pull all models from `config/test_suites.yaml`:

```bash
make cron-sync-models        # Pulls any master models missing locally
```

### Loading Multiple Models Simultaneously

By default Ollama keeps up to 3 models loaded in memory (3 × number of GPUs,
or 3 for CPU inference). To load more models concurrently, configure these
Ollama server environment variables:

| Variable | Default | Description |
|---|---|---|
| `OLLAMA_MAX_LOADED_MODELS` | 3 × GPUs (or 3) | Max models resident in memory at once |
| `OLLAMA_NUM_PARALLEL` | `1` | Parallel requests per loaded model |
| `OLLAMA_MAX_QUEUE` | `512` | Max queued requests before rejecting |

> **Memory note:** each loaded model consumes VRAM/RAM proportional to its
> size. A 7B Q4 model uses ~4 GB; a 27B model uses ~16 GB. Setting
> `OLLAMA_NUM_PARALLEL` > 1 multiplies context memory per model.

**Linux (systemd):**

```bash
sudo systemctl edit ollama.service
```

Add under `[Service]`:

```ini
[Service]
Environment="OLLAMA_MAX_LOADED_MODELS=5"
Environment="OLLAMA_NUM_PARALLEL=2"
```

Then restart:

```bash
sudo systemctl restart ollama
```

**macOS:**

```bash
launchctl setenv OLLAMA_MAX_LOADED_MODELS 5
launchctl setenv OLLAMA_NUM_PARALLEL 2
```

Restart the Ollama application after setting these.

**Windows:**

Set `OLLAMA_MAX_LOADED_MODELS` and `OLLAMA_NUM_PARALLEL` as system environment
variables, then restart Ollama.

### VRAM Sizing Guide

| Models Loaded | Recommended VRAM | Example Hardware |
|---|---|---|
| 3 (default) | 24 GB | RTX 4090, M2 Pro |
| 4 | 32 GB | 2× RTX 4080, M2 Max |
| 5+ | 48+ GB | 2× RTX 4090, M3 Ultra |

Actual requirements depend on model sizes and quantization levels.

### Auto-Discovery and Multi-Model Testing

The test harness auto-discovers available models at startup and skips tests
for models that are not installed — you will never get failures from missing
models.

```bash
make discover-local-models   # List models available on all configured nodes
make run-local-models        # Curated hosts (host-config.toml), parallel + loaded-model priority
make run-all-external        # Legacy wide-net discovery (env vars / subnet scan)

# Windows
uv run python scripts/run_local_models.py --discover-models --mode external
uv run python scripts/run_local_models.py
```

Use `ITERATIONS` for continuous testing (both targets):

```bash
make run-local-models ITERATIONS=-1   # Run forever
make run-local-models ITERATIONS=0    # Stop on first error
```

### Multi-Host Setup with host-config.toml

`make run-local-models` reads a curated host inventory from
`host-config.toml` (git-ignored) at the repo root:

```bash
cp host-config.toml.example host-config.toml   # then edit endpoints
```

A global `(model, suite)` job queue is scheduled across the configured
hosts in parallel. Each host prefers jobs whose model is already loaded
in VRAM (per Ollama's `/api/ps`), avoiding cold model loads. Per-host
`max_parallel`, `priority`, and `skip_models` plus a `global_max_parallel`
cap are configured in the TOML — see `host-config.toml.example`.

### Wide-Net Discovery (run-all-external)

`make run-all-external` preserves the previous env-var driven behavior:
it probes `OLLAMA_NODES_LIST` hosts (or scans `OLLAMA_SUBNET`, or falls
back to `OLLAMA_ENDPOINT`) and runs everything it finds, sequentially by
default (`execution.parallel` in `config/local_models.yaml`):

```bash
OLLAMA_NODES_LIST=localhost,gpu-server-1,gpu-server-2
```

Check node status with:

```bash
make discover-local-nodes
```

### Project Environment Variables

| Variable | Default | Description |
|---|---|---|
| `LLM_PROVIDER` | `ollama` | Provider backend (`ollama` or `openai`) |
| `OLLAMA_ENDPOINT` | `http://localhost:11434` | Ollama API endpoint |
| `DEFAULT_MODEL` | `qwen3:32b` | Model used for standard test runs |
| `OLLAMA_TIMEOUT` | `5400` | Request timeout in seconds (90 min) |
| `OLLAMA_NODES_LIST` | `localhost` | Comma-separated Ollama hostnames |

---

## Generating Model Cards

Model cards are objective SWOT analysis summaries of LLM test performance. They combine empirical metrics (pass rates, latency, throughput) with LLM-generated qualitative analysis.

### Setup

Install the Superset extra (required for database querying):

```bash
uv sync --extra superset
```

### Generate Cards for All Models

```bash
# Using Make
make model-cards

# Or directly
uv run python -m rfc.make_model_cards
```

Cards are written to `model_cards/<model_slug>.md` and ready to commit and publish.

### Generate Card for a Single Model

```bash
uv run python -m rfc.make_model_cards --model qwen2.5:72b
```

### Customize Output Directory

```bash
uv run python -m rfc.make_model_cards --output docs/models/
```

### Configuration

Environment variables (or CLI flags):

| Variable | CLI Flag | Default | Description |
|---|---|---|---|
| `DATABASE_URL` | `--database-url` | `sqlite:///data/test_history.db` | Test results database |
| `OLLAMA_ENDPOINT` | `--ollama-endpoint` | `http://localhost:11434` | Ollama API endpoint |
| `MODEL_CARD_LLM` | `--llm-model` | `qwen2.5:72b` | LLM for SWOT analysis |

Example with custom settings:

```bash
uv run python -m rfc.make_model_cards \
  --output model_cards/ \
  --ollama-endpoint http://gpu-server:11434 \
  --llm-model llama3.2:latest
```

### Card Format

Each card includes:

- **Metadata:** Provider, parameters, quantization, context window
- **Benchmarks:** Pass rate, latency (p50/p95/p99), throughput per suite
- **Overall Results:** Aggregated metrics + 7d vs 30d prior trend
- **SWOT Analysis:** LLM-generated Strengths, Weaknesses, Opportunities, Threats

Example card: [model_cards/qwen2.5_72b.md](https://github.com/tkarcheski/robotframework-chat/blob/main/model_cards/qwen2.5_72b.md) (if available)

---

## Example Test

```robot
*** Test Cases ***
LLM Can Do Basic Math
    ${answer}=    Ask LLM    What is 2 + 2?
    ${score}    ${reason}=    Grade Answer    What is 2 + 2?    4    ${answer}
    Should Be Equal As Integers    ${score}    1
```

---

## Core Philosophy

- **LLMs are software** — test them like software
- **Determinism before intelligence** — structured, machine-verifiable evaluation first
- **Constrained grading** — scores, categories, pass/fail; no prose from the evaluation layer
- **Modular by design** — composable pieces; new providers and graders plug in without rewriting core
- **Robot Framework as the orchestration layer** — readable, keyword-driven tests
- **Every test run is archived** — listeners always active, results flow to SQL
- **CI-native, regression-focused** — if it can't run unattended, it's not done

See [ai/agents.md](https://github.com/tkarcheski/robotframework-chat/blob/main/ai/agents.md#core-philosophy) for the full philosophy.

---

## Documentation

| Document | Description |
|----------|-------------|
| [docs/TEST_DATABASE.md](https://github.com/tkarcheski/robotframework-chat/blob/main/docs/TEST_DATABASE.md) | Database schema and usage |
| [docs/GRAFANA_SUPERSET_SETUP.md](https://github.com/tkarcheski/robotframework-chat/blob/main/docs/GRAFANA_SUPERSET_SETUP.md) | Superset visualization stack setup (Grafana deferred to v2+) |
| [docs/SUPERSET_EXPORT_GUIDE.md](https://github.com/tkarcheski/robotframework-chat/blob/main/docs/SUPERSET_EXPORT_GUIDE.md) | Superset dashboard export, import, and backup |
| [Ollama Configuration](#ollama-configuration) | Multi-model loading, VRAM sizing, and multi-node setup |
| [.claude/agents/](https://github.com/tkarcheski/robotframework-chat/tree/main/.claude/agents) | Role prompts for the four-role agent system — engineering, test-design, project-management, design |
| [CHANGELOG.md](https://github.com/tkarcheski/robotframework-chat/blob/main/CHANGELOG.md) | Release history since v1.4.3, thematic per minor line |

---

## Role System

This repo runs a **four-role agent system** for continuous development and quality assurance:

| Role | Responsibility |
|------|----------------|
| **engineering** | Picks up `status:ready` issues, implements on a branch, opens pull requests |
| **test-design** | Reviews open PRs, writes test plans, executes them, posts PASS/FAIL verdicts |
| **project-management** | Triages issues, sets priorities, monitors CI health, grooms the backlog |
| **design** | Architecture RFCs, system-wide improvements, open-ended design exploration |

The roles communicate through GitHub labels (`status:*`, `P0–P3`, `from:*`, `type:*`) and run concurrently in isolated git worktrees. Role prompts live in [`.claude/agents/`](https://github.com/tkarcheski/robotframework-chat/tree/main/.claude/agents); the full role contract and git topology docs are maintained in the private development monorepo.

---

## Contributing

### How this repo is published

This repository is the **public mirror** of a private development monorepo.
Development lands in the monorepo, whose publisher assembles the public
surface and opens a normal, reviewable pull request against the
`claude-code-staging` branch (`publish.sh --pr`); a publish allowlist
controls exactly which paths ship publicly. The owner reviews and merges
publish PRs — nothing is merged with failing checks — and `main` is
release-only. Issues and pull requests opened here are welcome: they are
triaged on this repo and forward-ported into the monorepo when accepted.

### Development workflow

1. Read [CLAUDE.md](https://github.com/tkarcheski/robotframework-chat/blob/main/CLAUDE.md) for the development workflow and TDD discipline
2. Follow the code style guidelines in [ai/agents.md](https://github.com/tkarcheski/robotframework-chat/blob/main/ai/agents.md)
3. Add tests for new features (see [ai/testing.md](https://github.com/tkarcheski/robotframework-chat/blob/main/ai/testing.md) for grading tiers)
4. Run `pre-commit run --all-files` before committing

See [CHANGELOG.md](https://github.com/tkarcheski/robotframework-chat/blob/main/CHANGELOG.md) for release history.
