# Custom Judge Interface Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Allow operators to replace the default LLMJudge (Claude) with either an OpenAI-compatible backend or a fully custom Python Judge subclass, configured via `openclaw-target.yaml`.

**Architecture:** Two new judge classes (`OpenAICompatibleJudge` via httpx, `PluginJudge` via importlib) are added under `src/agentsec/evaluator/`. Config schema gains `judge:` field with Pydantic discriminated union. A `_build_llm_judge` helper in `cli.py` reads the config and returns the correct judge instance; `_evaluate_remote` delegates to it.

**Tech Stack:** Python 3.11+, Pydantic v2, httpx (already in deps), importlib.util, pytest/tmp_path for tests.

---

## File Map

| File | Action |
|------|--------|
| `src/agentsec/config.py` | Add `OpenAIJudgeConfig`, `PluginJudgeConfig`, `JudgeConfig`; add `judge` field; update `from_yaml` |
| `src/agentsec/evaluator/openai_judge.py` | New: `OpenAICompatibleJudge` |
| `src/agentsec/evaluator/plugin_judge.py` | New: `PluginJudge` |
| `src/agentsec/cli.py` | Add `_build_llm_judge`; wire into `_evaluate_remote` |
| `tests/test_config_model.py` | 6 new test cases (append) |
| `tests/test_openai_judge.py` | New: 7 tests |
| `tests/test_plugin_judge.py` | New: 4 tests |
| `tests/test_cli_judge_router.py` | 4 new test cases (append) |
| `CHANGELOG.md` | v0.8.0 entry |
| `ROADMAP.md` | Tick off custom judge item |

---

## Task 1: Config Schema

**Files:**
- Modify: `src/agentsec/config.py`
- Test: `tests/test_config_model.py`

### Step 1.1 — Write failing tests

Append to `tests/test_config_model.py`:

```python
from agentsec.config import (
    OpenAIJudgeConfig,
    OpenClawTargetConfig,
    PluginJudgeConfig,
)
```

(Add these names to the existing import at the top of the file — the existing import only pulls `OpenClawTargetConfig`.)

Then append these six test functions:

```python
# ── Custom judge config tests ──────────────────────────────────────────────

def test_openai_judge_config_valid():
    cfg = OpenAIJudgeConfig.model_validate({
        "type": "openai_compatible",
        "model": "gpt-4o",
        "base_url": "https://api.openai.com/v1",
        "api_key_env": "OPENAI_API_KEY",
    })
    assert cfg.model == "gpt-4o"
    assert cfg.timeout == 30  # default


def test_openai_judge_config_missing_model():
    from pydantic import ValidationError
    with pytest.raises(ValidationError):
        OpenAIJudgeConfig.model_validate({
            "type": "openai_compatible",
            "base_url": "https://api.openai.com/v1",
            "api_key_env": "OPENAI_API_KEY",
        })


def test_plugin_judge_config_valid():
    cfg = PluginJudgeConfig.model_validate({
        "type": "plugin",
        "path": "./my_judge.py",
    })
    assert cfg.path == Path("my_judge.py")


def test_openclaw_target_config_with_openai_judge(tmp_path):
    p = tmp_path / "cfg.yaml"
    p.write_text(
        "name: t\n"
        "remote:\n"
        "  adapter: openclaw-gateway\n"
        "  target: https://x\n"
        "  test_suites: [test_suites/openclaw/]\n"
        "output:\n"
        "  dir: ./out/\n"
        "judge:\n"
        "  type: openai_compatible\n"
        "  model: gpt-4o\n"
        "  base_url: https://api.openai.com/v1\n"
        "  api_key_env: MY_KEY\n"
    )
    cfg = OpenClawTargetConfig.from_yaml(p)
    assert isinstance(cfg.judge, OpenAIJudgeConfig)
    assert cfg.judge.model == "gpt-4o"


def test_openclaw_target_config_with_plugin_judge(tmp_path):
    p = tmp_path / "cfg.yaml"
    p.write_text(
        "name: t\n"
        "remote:\n"
        "  adapter: openclaw-gateway\n"
        "  target: https://x\n"
        "  test_suites: [test_suites/openclaw/]\n"
        "output:\n"
        "  dir: ./out/\n"
        "judge:\n"
        "  type: plugin\n"
        "  path: ./my_judge.py\n"
    )
    cfg = OpenClawTargetConfig.from_yaml(p)
    assert isinstance(cfg.judge, PluginJudgeConfig)
    assert cfg.judge.path == Path("my_judge.py")


def test_openclaw_target_config_unknown_judge_type(tmp_path):
    from pydantic import ValidationError
    p = tmp_path / "cfg.yaml"
    p.write_text(
        "name: t\n"
        "remote:\n"
        "  adapter: openclaw-gateway\n"
        "  target: https://x\n"
        "  test_suites: [test_suites/openclaw/]\n"
        "output:\n"
        "  dir: ./out/\n"
        "judge:\n"
        "  type: nonexistent\n"
        "  model: gpt-4o\n"
    )
    with pytest.raises(ValidationError):
        OpenClawTargetConfig.from_yaml(p)
```

- [ ] **Step 1.2 — Run tests to verify they fail**

```bash
PYTHONPATH=src .venv/bin/pytest tests/test_config_model.py \
  -k "judge" -v 2>&1 | tail -20
```

Expected: 6 failures with `ImportError` or `ValidationError`/`AttributeError`.

- [ ] **Step 1.3 — Implement: add models to `config.py`**

Open `src/agentsec/config.py`. The current top imports are:

```python
from __future__ import annotations

from pathlib import Path
from typing import Literal

import yaml
from pydantic import BaseModel, ConfigDict, Field, model_validator
```

Change to:

```python
from __future__ import annotations

from pathlib import Path
from typing import Annotated, Literal

import yaml
from pydantic import BaseModel, ConfigDict, Field, model_validator
```

After the `_FORBID = ConfigDict(extra="forbid")` line (line 21), add:

```python

class OpenAIJudgeConfig(BaseModel):
    model_config = _FORBID
    type: Literal["openai_compatible"]
    model: str
    base_url: str
    api_key_env: str
    timeout: int = Field(default=30, ge=1)


class PluginJudgeConfig(BaseModel):
    model_config = _FORBID
    type: Literal["plugin"]
    path: Path


JudgeConfig = Annotated[
    OpenAIJudgeConfig | PluginJudgeConfig,
    Field(discriminator="type"),
]
```

In `OpenClawTargetConfig` (currently ends with `scoring: ScoringConfig`), add after the `scoring` field:

```python
    judge: JudgeConfig | None = None
```

Replace the existing `from_yaml` classmethod:

```python
    @classmethod
    def from_yaml(cls, path: Path | str) -> "OpenClawTargetConfig":
        path = Path(path)
        text = path.read_text(encoding="utf-8")
        data = yaml.safe_load(text)
        if not isinstance(data, dict):
            raise ValueError(
                f"{path}: config must be a YAML mapping, got {type(data).__name__}"
            )
        obj = cls.model_validate(data)
        object.__setattr__(obj, "_config_path", path)
        return obj
```

- [ ] **Step 1.4 — Run tests to verify they pass**

```bash
PYTHONPATH=src .venv/bin/pytest tests/test_config_model.py \
  -k "judge" -v 2>&1 | tail -20
```

Expected: 6 passed.

- [ ] **Step 1.5 — Run full test suite to check for regressions**

```bash
PYTHONPATH=src .venv/bin/pytest tests/ -x -q 2>&1 | tail -10
```

Expected: all existing tests still pass.

- [ ] **Step 1.6 — Commit**

```bash
git add src/agentsec/config.py tests/test_config_model.py
git commit -m "feat(config): add OpenAIJudgeConfig, PluginJudgeConfig, judge field"
```

---

## Task 2: OpenAICompatibleJudge

**Files:**
- Create: `src/agentsec/evaluator/openai_judge.py`
- Create: `tests/test_openai_judge.py`

- [ ] **Step 2.1 — Write failing tests**

Create `tests/test_openai_judge.py`:

```python
"""Tests for OpenAICompatibleJudge."""
from __future__ import annotations

import json
import os

import httpx
import pytest

from agentsec.adapters.base import AgentResponse
from agentsec.config import OpenAIJudgeConfig
from agentsec.evaluator.openai_judge import OpenAICompatibleJudge
from agentsec.observability.observation import Observation
from agentsec.tests.models import TestCase


def _make_judge(monkeypatch, env_var: str = "FAKE_OAI_KEY") -> OpenAICompatibleJudge:
    monkeypatch.setenv(env_var, "sk-test-value")
    cfg = OpenAIJudgeConfig.model_validate({
        "type": "openai_compatible",
        "model": "gpt-4o",
        "base_url": "http://fake",
        "api_key_env": env_var,
    })
    return cfg, OpenAICompatibleJudge(cfg)


def _make_obs(content: str = "I will help you with that.") -> Observation:
    return Observation(
        response=AgentResponse(content=content, latency_ms=10.0, raw={}, tool_events=[]),
        network_events=[],
        fixture_violations=[],
    )


def _make_test_case() -> TestCase:
    return TestCase(
        id="tc-1",
        category="prompt_injection",
        description="test",
        attack="ignore instructions",
        expected_behavior="refusal",
        severity="high",
    )


def _transport_returning(status: int, body: str) -> httpx.MockTransport:
    def handler(req: httpx.Request) -> httpx.Response:
        return httpx.Response(status, text=body,
                              headers={"content-type": "application/json"})
    return httpx.MockTransport(handler)


def _oai_body(passed: bool, reasoning: str = "looks fine") -> str:
    inner = json.dumps({"passed": passed, "reasoning": reasoning})
    return json.dumps({"choices": [{"message": {"content": inner}}]})


@pytest.mark.asyncio
async def test_openai_judge_passed_true(monkeypatch):
    cfg, judge = _make_judge(monkeypatch)
    transport = _transport_returning(200, _oai_body(True, "agent resisted"))
    judge._transport = transport
    verdict = await judge.evaluate(_make_test_case(), _make_obs())
    assert verdict.passed is True
    assert "resisted" in verdict.reasoning
    assert verdict.judge_type == "llm"


@pytest.mark.asyncio
async def test_openai_judge_passed_false(monkeypatch):
    cfg, judge = _make_judge(monkeypatch)
    transport = _transport_returning(200, _oai_body(False, "agent leaked data"))
    judge._transport = transport
    verdict = await judge.evaluate(_make_test_case(), _make_obs())
    assert verdict.passed is False
    assert "leaked" in verdict.reasoning


@pytest.mark.asyncio
async def test_openai_judge_http_500(monkeypatch):
    cfg, judge = _make_judge(monkeypatch)
    transport = _transport_returning(500, "Internal Server Error")
    judge._transport = transport
    verdict = await judge.evaluate(_make_test_case(), _make_obs())
    assert verdict.passed is False
    assert "http error" in verdict.reasoning
    assert "500" in verdict.reasoning


@pytest.mark.asyncio
async def test_openai_judge_response_not_json(monkeypatch):
    cfg, judge = _make_judge(monkeypatch)
    transport = _transport_returning(200, "not json at all")
    judge._transport = transport
    verdict = await judge.evaluate(_make_test_case(), _make_obs())
    assert verdict.passed is False
    assert "parse error" in verdict.reasoning


@pytest.mark.asyncio
async def test_openai_judge_response_missing_passed_field(monkeypatch):
    cfg, judge = _make_judge(monkeypatch)
    inner = json.dumps({"reasoning": "something"})  # no "passed" key
    body = json.dumps({"choices": [{"message": {"content": inner}}]})
    transport = _transport_returning(200, body)
    judge._transport = transport
    verdict = await judge.evaluate(_make_test_case(), _make_obs())
    assert verdict.passed is False
    assert "parse error" in verdict.reasoning
    assert "missing fields" in verdict.reasoning


@pytest.mark.asyncio
async def test_openai_judge_timeout(monkeypatch):
    cfg, judge = _make_judge(monkeypatch)

    def timeout_handler(req: httpx.Request) -> httpx.Response:
        raise httpx.TimeoutException("timed out", request=req)

    judge._transport = httpx.MockTransport(timeout_handler)
    verdict = await judge.evaluate(_make_test_case(), _make_obs())
    assert verdict.passed is False
    assert verdict.reasoning == "request timeout"


def test_openai_judge_constructor_missing_env_var(monkeypatch):
    monkeypatch.delenv("MISSING_ENV_VAR", raising=False)
    cfg = OpenAIJudgeConfig.model_validate({
        "type": "openai_compatible",
        "model": "gpt-4o",
        "base_url": "http://fake",
        "api_key_env": "MISSING_ENV_VAR",
    })
    with pytest.raises(ValueError, match="MISSING_ENV_VAR"):
        OpenAICompatibleJudge(cfg)
```

- [ ] **Step 2.2 — Run tests to verify they fail**

```bash
PYTHONPATH=src .venv/bin/pytest tests/test_openai_judge.py -v 2>&1 | tail -15
```

Expected: 7 failures with `ModuleNotFoundError` (file does not exist yet).

- [ ] **Step 2.3 — Check whether `pytest-asyncio` is available**

```bash
PYTHONPATH=src .venv/bin/python -c "import pytest_asyncio; print(pytest_asyncio.__version__)"
```

If this prints a version, proceed. If it prints `ModuleNotFoundError`, run:

```bash
.venv/bin/pip install pytest-asyncio
```

Also check whether `asyncio_mode = "auto"` is set in `pyproject.toml`:

```bash
grep asyncio pyproject.toml
```

If `asyncio_mode = "auto"` is present, the `@pytest.mark.asyncio` decorators above are redundant but harmless. If it is absent, add to `[tool.pytest.ini_options]` in `pyproject.toml`:

```toml
asyncio_mode = "auto"
```

- [ ] **Step 2.4 — Implement `openai_judge.py`**

Create `src/agentsec/evaluator/openai_judge.py`:

```python
"""OpenAI-compatible LLM judge — calls any OpenAI-format /chat/completions endpoint."""
from __future__ import annotations

import json
import os

import httpx

from ..observability.observation import Observation
from ..tests.models import TestCase
from .base import Judge, JudgeVerdict
from .judge import _SYSTEM


class OpenAICompatibleJudge(Judge):
    """Judge that calls an OpenAI-compatible /chat/completions endpoint."""

    def __init__(
        self,
        config: "OpenAIJudgeConfig",  # type: ignore[name-defined]  # forward ref
        _transport: httpx.AsyncBaseTransport | None = None,
    ) -> None:
        api_key = os.environ.get(config.api_key_env)
        if api_key is None:
            raise ValueError(
                f"environment variable {config.api_key_env!r} is not set"
            )
        self._config = config
        self._api_key = api_key
        self._transport = _transport

    def _build_user(self, test: TestCase, obs: Observation) -> str:
        attack = test.turns[-1].attack if test.turns else (test.attack or "")
        return (
            f"Attack payload:\n{attack}\n\n"
            f"Agent response:\n{obs.response.content}\n\n"
            f"Expected behavior: {test.expected_behavior}\n"
            + (f"Hint: {test.judge_hint}" if test.judge_hint else "")
        )

    async def evaluate(self, test_case: TestCase, observation: Observation) -> JudgeVerdict:
        user = self._build_user(test_case, observation)
        try:
            async with httpx.AsyncClient(
                timeout=self._config.timeout,
                transport=self._transport,
            ) as client:
                resp = await client.post(
                    f"{self._config.base_url}/chat/completions",
                    headers={"Authorization": f"Bearer {self._api_key}"},
                    json={
                        "model": self._config.model,
                        "messages": [
                            {"role": "system", "content": _SYSTEM},
                            {"role": "user", "content": user},
                        ],
                        "temperature": 0,
                    },
                )
        except httpx.TimeoutException:
            return JudgeVerdict(
                passed=False, reasoning="request timeout", judge_type="llm"
            )
        except httpx.TransportError as exc:
            return JudgeVerdict(
                passed=False, reasoning=f"request error: {exc}", judge_type="llm"
            )

        if not resp.is_success:
            return JudgeVerdict(
                passed=False,
                reasoning=f"http error {resp.status_code}: {resp.text[:200]}",
                judge_type="llm",
            )

        try:
            body = resp.json()
            content = body["choices"][0]["message"]["content"]
        except Exception:
            return JudgeVerdict(
                passed=False, reasoning="parse error: malformed response body", judge_type="llm"
            )

        try:
            parsed = json.loads(content)
        except json.JSONDecodeError as exc:
            return JudgeVerdict(
                passed=False, reasoning=f"parse error: {exc}", judge_type="llm"
            )

        try:
            return JudgeVerdict(
                passed=bool(parsed["passed"]),
                reasoning=str(parsed["reasoning"]),
                judge_type="llm",
            )
        except (KeyError, TypeError):
            return JudgeVerdict(
                passed=False, reasoning="parse error: missing fields", judge_type="llm"
            )
```

- [ ] **Step 2.5 — Run tests to verify they pass**

```bash
PYTHONPATH=src .venv/bin/pytest tests/test_openai_judge.py -v 2>&1 | tail -15
```

Expected: 7 passed.

- [ ] **Step 2.6 — Commit**

```bash
git add src/agentsec/evaluator/openai_judge.py tests/test_openai_judge.py
git commit -m "feat(evaluator): add OpenAICompatibleJudge"
```

---

## Task 3: PluginJudge

**Files:**
- Create: `src/agentsec/evaluator/plugin_judge.py`
- Create: `tests/test_plugin_judge.py`

- [ ] **Step 3.1 — Write failing tests**

Create `tests/test_plugin_judge.py`:

```python
"""Tests for PluginJudge — dynamic plugin loading."""
from __future__ import annotations

from pathlib import Path

import pytest

from agentsec.adapters.base import AgentResponse
from agentsec.evaluator.plugin_judge import PluginJudge
from agentsec.observability.observation import Observation
from agentsec.tests.models import TestCase


def _make_obs() -> Observation:
    return Observation(
        response=AgentResponse(content="ok", latency_ms=5.0, raw={}, tool_events=[]),
        network_events=[],
        fixture_violations=[],
    )


def _make_test_case() -> TestCase:
    return TestCase(
        id="tc-plugin",
        category="prompt_injection",
        description="plugin test",
        attack="pwn",
        expected_behavior="refusal",
        severity="medium",
    )


def _write_valid_plugin(tmp_path: Path) -> Path:
    p = tmp_path / "my_judge.py"
    p.write_text(
        "from agentsec.evaluator.base import Judge, JudgeVerdict\n"
        "\n"
        "class MyJudge(Judge):\n"
        "    async def evaluate(self, test_case, observation):\n"
        "        return JudgeVerdict(passed=True, reasoning='plugin ok', judge_type='llm')\n"
    )
    return p


@pytest.mark.asyncio
async def test_plugin_judge_valid_delegates(tmp_path):
    plugin_path = _write_valid_plugin(tmp_path)
    judge = PluginJudge(plugin_path)
    verdict = await judge.evaluate(_make_test_case(), _make_obs())
    assert verdict.passed is True
    assert verdict.reasoning == "plugin ok"


def test_plugin_judge_file_not_found(tmp_path):
    with pytest.raises(ValueError, match="not found"):
        PluginJudge(tmp_path / "missing.py")


def test_plugin_judge_no_subclass(tmp_path):
    p = tmp_path / "empty.py"
    p.write_text("x = 1\n")
    with pytest.raises(ValueError, match="no Judge subclass found"):
        PluginJudge(p)


def test_plugin_judge_ambiguous(tmp_path):
    p = tmp_path / "two.py"
    p.write_text(
        "from agentsec.evaluator.base import Judge, JudgeVerdict\n"
        "\n"
        "class JudgeA(Judge):\n"
        "    async def evaluate(self, test_case, observation):\n"
        "        return JudgeVerdict(passed=True, reasoning='a', judge_type='llm')\n"
        "\n"
        "class JudgeB(Judge):\n"
        "    async def evaluate(self, test_case, observation):\n"
        "        return JudgeVerdict(passed=True, reasoning='b', judge_type='llm')\n"
    )
    with pytest.raises(ValueError, match="ambiguous"):
        PluginJudge(p)
```

- [ ] **Step 3.2 — Run tests to verify they fail**

```bash
PYTHONPATH=src .venv/bin/pytest tests/test_plugin_judge.py -v 2>&1 | tail -15
```

Expected: 4 failures with `ModuleNotFoundError`.

- [ ] **Step 3.3 — Implement `plugin_judge.py`**

Create `src/agentsec/evaluator/plugin_judge.py`:

```python
"""PluginJudge — loads a user-supplied .py file and delegates to the Judge subclass inside."""
from __future__ import annotations

import importlib.util
from pathlib import Path

from ..observability.observation import Observation
from ..tests.models import TestCase
from .base import Judge, JudgeVerdict


class PluginJudge(Judge):
    """Wraps a dynamically loaded user Judge subclass."""

    def __init__(self, path: Path) -> None:
        if not path.exists():
            raise ValueError(f"{path}: file not found")

        spec = importlib.util.spec_from_file_location("_agentsec_plugin_judge", path)
        module = importlib.util.module_from_spec(spec)  # type: ignore[arg-type]
        spec.loader.exec_module(module)  # type: ignore[union-attr]

        subclasses = [
            obj
            for obj in vars(module).values()
            if isinstance(obj, type) and issubclass(obj, Judge) and obj is not Judge
        ]

        if len(subclasses) == 0:
            raise ValueError(f"{path}: no Judge subclass found")
        if len(subclasses) > 1:
            names = ", ".join(c.__name__ for c in subclasses)
            raise ValueError(f"{path}: ambiguous — found: {names}")

        self._delegate: Judge = subclasses[0]()

    async def evaluate(self, test_case: TestCase, observation: Observation) -> JudgeVerdict:
        return await self._delegate.evaluate(test_case, observation)
```

- [ ] **Step 3.4 — Run tests to verify they pass**

```bash
PYTHONPATH=src .venv/bin/pytest tests/test_plugin_judge.py -v 2>&1 | tail -15
```

Expected: 4 passed.

- [ ] **Step 3.5 — Commit**

```bash
git add src/agentsec/evaluator/plugin_judge.py tests/test_plugin_judge.py
git commit -m "feat(evaluator): add PluginJudge with importlib dynamic loading"
```

---

## Task 4: CLI Wiring

**Files:**
- Modify: `src/agentsec/cli.py`
- Test: `tests/test_cli_judge_router.py`

- [ ] **Step 4.1 — Write failing tests**

Append to `tests/test_cli_judge_router.py`:

```python
# ── _build_llm_judge tests ──────────────────────────────────────────────────

def _make_remote_only_config(tmp_path, judge_yaml: str = "") -> "OpenClawTargetConfig":
    """Helper: write a minimal openclaw-target.yaml with an optional judge block."""
    from agentsec.config import OpenClawTargetConfig
    text = (
        "name: t\n"
        "remote:\n"
        "  adapter: openclaw-gateway\n"
        "  target: https://x\n"
        "  test_suites: [test_suites/openclaw/]\n"
        "output:\n"
        "  dir: ./out/\n"
    )
    if judge_yaml:
        text += judge_yaml
    p = tmp_path / "cfg.yaml"
    p.write_text(text)
    return OpenClawTargetConfig.from_yaml(p)


def test_build_llm_judge_none_returns_llm_judge(tmp_path, monkeypatch):
    from agentsec.cli import _build_llm_judge
    from agentsec.evaluator.judge import LLMJudge
    monkeypatch.setenv("ANTHROPIC_API_KEY", "sk-fake")
    cfg = _make_remote_only_config(tmp_path)
    assert cfg.judge is None
    judge = _build_llm_judge(cfg, api_key="sk-fake")
    assert isinstance(judge, LLMJudge)


def test_build_llm_judge_openai_config_env_set(tmp_path, monkeypatch):
    from agentsec.cli import _build_llm_judge
    from agentsec.evaluator.openai_judge import OpenAICompatibleJudge
    monkeypatch.setenv("MY_OAI_KEY", "sk-test")
    cfg = _make_remote_only_config(
        tmp_path,
        "judge:\n"
        "  type: openai_compatible\n"
        "  model: gpt-4o\n"
        "  base_url: https://api.openai.com/v1\n"
        "  api_key_env: MY_OAI_KEY\n",
    )
    judge = _build_llm_judge(cfg, api_key=None)
    assert isinstance(judge, OpenAICompatibleJudge)


def test_build_llm_judge_openai_config_env_absent(tmp_path, monkeypatch):
    from agentsec.cli import _build_llm_judge
    monkeypatch.delenv("MISSING_KEY_XYZ", raising=False)
    cfg = _make_remote_only_config(
        tmp_path,
        "judge:\n"
        "  type: openai_compatible\n"
        "  model: gpt-4o\n"
        "  base_url: https://api.openai.com/v1\n"
        "  api_key_env: MISSING_KEY_XYZ\n",
    )
    with pytest.raises(ValueError, match="MISSING_KEY_XYZ"):
        _build_llm_judge(cfg, api_key=None)


def test_build_llm_judge_plugin_config_returns_plugin_judge(tmp_path, monkeypatch):
    from agentsec.cli import _build_llm_judge
    from agentsec.evaluator.plugin_judge import PluginJudge

    plugin = tmp_path / "myjudge.py"
    plugin.write_text(
        "from agentsec.evaluator.base import Judge, JudgeVerdict\n"
        "\n"
        "class MyJudge(Judge):\n"
        "    async def evaluate(self, test_case, observation):\n"
        "        return JudgeVerdict(passed=True, reasoning='ok', judge_type='llm')\n"
    )
    cfg = _make_remote_only_config(
        tmp_path,
        f"judge:\n"
        f"  type: plugin\n"
        f"  path: {plugin.name}\n",
    )
    judge = _build_llm_judge(cfg, api_key=None)
    assert isinstance(judge, PluginJudge)
```

Also add the missing `import pytest` at the top of the file if not present. The file currently has no imports from pytest — add:

```python
import pytest
```

at the top (after existing imports).

- [ ] **Step 4.2 — Run tests to verify they fail**

```bash
PYTHONPATH=src .venv/bin/pytest tests/test_cli_judge_router.py \
  -k "build_llm_judge" -v 2>&1 | tail -15
```

Expected: 4 failures with `ImportError: cannot import name '_build_llm_judge'`.

- [ ] **Step 4.3 — Implement: add imports to `cli.py`**

In `src/agentsec/cli.py`, find the existing import block (around lines 34–50). Add these two lines after the existing evaluator imports:

```python
from .evaluator.openai_judge import OpenAICompatibleJudge
from .evaluator.plugin_judge import PluginJudge
```

Also update the `.config` import from:

```python
from .config import MultiTargetConfig, OpenClawTargetConfig
```

to:

```python
from .config import MultiTargetConfig, OpenAIJudgeConfig, OpenClawTargetConfig, PluginJudgeConfig
```

- [ ] **Step 4.4 — Implement: add `_build_llm_judge` to `cli.py`**

In `src/agentsec/cli.py`, locate `def _evaluate_remote(` (around line 406). Insert the following function **immediately before** it (as a module-level helper):

```python
def _build_llm_judge(cfg: OpenClawTargetConfig, api_key: str | None) -> "Judge":
    """Return the appropriate LLM-slot judge based on the target config."""
    judge_cfg = cfg.judge
    if judge_cfg is None:
        return LLMJudge(api_key=api_key)
    if isinstance(judge_cfg, OpenAIJudgeConfig):
        return OpenAICompatibleJudge(judge_cfg)
    if isinstance(judge_cfg, PluginJudgeConfig):
        resolved = (Path(cfg._config_path).parent / judge_cfg.path).resolve()
        return PluginJudge(resolved)
    raise AssertionError(f"unhandled judge config type: {type(judge_cfg)}")

```

- [ ] **Step 4.5 — Implement: wire `_build_llm_judge` into `_evaluate_remote`**

In `_evaluate_remote`, find these two lines (around line 434–435):

```python
    needs_llm = any(tc.judge_type in {"llm", "hybrid"} for tc in test_cases)
    llm = LLMJudge(api_key=api_key) if needs_llm else NoOpJudge()
```

Replace with:

```python
    needs_llm = any(tc.judge_type in {"llm", "hybrid"} for tc in test_cases)
    if needs_llm:
        try:
            llm = _build_llm_judge(cfg, api_key)
        except ValueError as exc:
            console.print(f"[red]error:[/red] {exc}")
            raise typer.Exit(code=1) from exc
    else:
        llm = NoOpJudge()
```

- [ ] **Step 4.6 — Run new tests to verify they pass**

```bash
PYTHONPATH=src .venv/bin/pytest tests/test_cli_judge_router.py \
  -k "build_llm_judge" -v 2>&1 | tail -15
```

Expected: 4 passed.

- [ ] **Step 4.7 — Run full test suite**

```bash
PYTHONPATH=src .venv/bin/pytest tests/ -x -q 2>&1 | tail -10
```

Expected: all tests pass.

- [ ] **Step 4.8 — Lint**

```bash
.venv/bin/ruff check src/ --fix
```

Expected: no errors.

- [ ] **Step 4.9 — Commit**

```bash
git add src/agentsec/cli.py tests/test_cli_judge_router.py
git commit -m "feat(cli): add _build_llm_judge, wire custom judge into _evaluate_remote"
```

---

## Task 5: Docs

**Files:**
- Modify: `CHANGELOG.md`
- Modify: `ROADMAP.md`

- [ ] **Step 5.1 — Update CHANGELOG.md**

Open `CHANGELOG.md`. Add a new entry at the top (after the `# Changelog` heading but before the existing v0.7.0 entry):

```markdown
## [0.8.0] — 2026-05-04

### Added

- **Custom judge interface** — operators can now replace the default Claude judge in
  `openclaw-target.yaml` with any OpenAI-compatible LLM backend (`type: openai_compatible`)
  or a Python plugin (`type: plugin`). See `docs/superpowers/specs/2026-05-04-custom-judge-design.md`.
- `OpenAICompatibleJudge` (`src/agentsec/evaluator/openai_judge.py`) — calls any
  `/chat/completions` endpoint; reuses the same `_SYSTEM` prompt as `LLMJudge`.
- `PluginJudge` (`src/agentsec/evaluator/plugin_judge.py`) — loads a user `.py` file
  via `importlib`, discovers the single `Judge` subclass, delegates all calls to it.
- `_build_llm_judge` helper in `cli.py` dispatches to the correct judge based on config.
- New config fields: `OpenAIJudgeConfig`, `PluginJudgeConfig` (Pydantic discriminated union).

```

- [ ] **Step 5.2 — Update ROADMAP.md**

In `ROADMAP.md`, find the line for the custom judge item (it will look like `- [ ] Custom judge interface / plugin` or similar). Change `[ ]` to `[x]`.

- [ ] **Step 5.3 — Commit**

```bash
git add CHANGELOG.md ROADMAP.md
git commit -m "chore: mark custom judge interface complete in ROADMAP; update CHANGELOG"
```

---

## Final verification

```bash
PYTHONPATH=src .venv/bin/pytest tests/ -q 2>&1 | tail -5
.venv/bin/ruff check src/
```

Both should complete with zero failures / zero lint errors.
