Verifier

A Verifier grades an agent's result against the original query and returns structured feedback — a score, specific issues, and a concrete suggestion — instead of a plain boolean. Attached to a LoopConfig, it drives both convergence and a targeted refinement prompt built from its critique, replacing the static default refine message. This page is a showcase of the ways to use it, end to end.

How-to: basic quality gate

The simplest use — stop refining once the built-in LLM grader is confident enough.

from deepcrew import Agent, run_agent, LoopConfig, Verifier, VerifierConfig

agent = Agent(
    name="researcher",
    model="openai/gpt-4o-mini",
    tools=[search_web],
    loop_config=LoopConfig(
        max_iterations=4,
        verifier=Verifier(VerifierConfig(threshold=0.85)),
    ),
)

result = await run_agent(agent, [{"role": "user", "content": "Explain CRISPR"}])
print(result.text)

How-to: a task-specific rubric

Pass grading criteria specific to your domain so the verifier checks for what actually matters — e.g. code review, not just "is this a complete sentence."

code_reviewer = Agent(
    name="code_reviewer",
    model="openai/gpt-4o",
    system_prompt="Review the given diff for bugs, security issues, and style.",
    loop_config=LoopConfig(
        max_iterations=3,
        verifier=Verifier(VerifierConfig(
            threshold=0.9,
            rubric=(
                "1. Every changed function must be covered by the review.\n"
                "2. Security issues (injection, auth, secrets) must be called out explicitly.\n"
                "3. Style nits are optional but bugs are not."
            ),
        )),
    ),
)

result = await run_agent(code_reviewer, [{"role": "user", "content": diff_text}])

How-to: fully custom grading (no LLM call)

evaluate_fn replaces the built-in LLM grader entirely — useful when you have a deterministic check (schema validation, a unit test, a regex) that's cheaper and more reliable than asking another model.

import json
from deepcrew import VerifierFeedback

async def json_schema_grader(query: str, result) -> VerifierFeedback:
    try:
        data = json.loads(result.text)
    except json.JSONDecodeError:
        return VerifierFeedback(score=0.0, issues=["Output is not valid JSON"], suggestion="Return valid JSON only.")
    missing = [k for k in ("summary", "action_items") if k not in data]
    if missing:
        return VerifierFeedback(score=0.4, issues=[f"Missing key: {k}" for k in missing], suggestion="Include all required keys.")
    return VerifierFeedback(score=1.0, converged=True)

agent = Agent(
    name="extractor",
    model="openai/gpt-4o-mini",
    loop_config=LoopConfig(
        max_iterations=3,
        verifier=Verifier(VerifierConfig(evaluate_fn=json_schema_grader)),
    ),
)

How-to: adaptive compute budget v0.2.3

By default the loop always runs max_iterations times unless it converges early. With adaptive=True, it also tracks the verifier score across iterations and stops as soon as improvement plateaus — saving compute once refinement stops paying off. max_iterations is still a hard ceiling; adaptive can only shorten the loop, never lengthen it.

agent = Agent(
    name="researcher",
    model="openai/gpt-4o-mini",
    tools=[search_web],
    loop_config=LoopConfig(
        max_iterations=8,
        verifier=Verifier(VerifierConfig(threshold=0.9)),
        adaptive=True,
        min_improvement=0.02,   # minimum score delta to still count as "improving"
        plateau_patience=2,     # stop after this many non-improving iterations in a row
    ),
)

result = await run_agent(agent, [{"role": "user", "content": "Explain CRISPR"}])
print(f"Stopped after {result.loop_iterations} iterations (cap was 8)")

When the loop stops early on a plateau, it returns the highest-scoring result seen so far — not necessarily the very last one — and emits a LOOP_ITERATION event with {"early_stop": "plateau"} in its data. Adaptive is a no-op without a verifier configured (there's no score to track).

How-to: grading something outside a loop

Verifier doesn't require a LoopConfig at all — call evaluate() directly whenever you have a query and an AgentResult you want graded, e.g. to build your own custom retry/escalation logic instead of using the built-in loop.

from deepcrew import Verifier, VerifierConfig, run_agent, Agent

verifier = Verifier(VerifierConfig(threshold=0.85))
agent = Agent(name="drafter", model="openai/gpt-4o-mini")

result = await run_agent(agent, [{"role": "user", "content": "Draft a release note"}])

feedback = await verifier.evaluate(
    "Draft a release note",
    result,
    default_model=agent.model,   # used only if VerifierConfig.model wasn't set
)

if not feedback.converged:
    print(f"Score {feedback.score:.2f} — issues: {feedback.issues}")
    print(f"Suggestion: {feedback.suggestion}")

assess_complexity() — the other Verifier method

Separate from evaluate() (which grades a finished answer), Verifier also exposes assess_complexity(task, default_model=...) — a lightweight, pre-execution judgment call: "does this task genuinely need decomposing into sub-tasks, or can one agent handle it directly?" This is what powers Orchestrator's spawn_complexity_check parameter, gating whether a newly-spawned sub-agent gets its own nested spawn tool (see Agent Spawning → Bounded nested spawning). You can call it standalone too:

verifier = Verifier()
needs_decomposition = await verifier.assess_complexity(
    "Build a full REST API with auth, rate limiting, and tests",
    default_model="openai/gpt-4o-mini",
)
print(needs_decomposition)  # bool

If the underlying LLM call fails or returns something unparseable, assess_complexity() defaults to True (permissive) rather than False — the reasoning being that a failed complexity check should not silently prevent legitimate decomposition. If you constructed the Verifier with evaluate_fn (a fully custom grader), assess_complexity() has no way to reuse that custom function for this different purpose and always returns True unconditionally.

Verifier reference

Common pitfalls

• A failed grading call returns score 0.0, not an error. If the LLM call raises, or the response can't be parsed as JSON even with the regex fallback, evaluate() swallows the exception and returns VerifierFeedback(score=0.0, ...) — indistinguishable from a genuinely terrible answer. If your loop seems to never converge, check whether the grading model itself is actually reachable before assuming your agent's output is at fault.
• rubric is appended text, not a replacement. It's inserted into the default grading prompt as extra criteria — it doesn't change the required JSON response shape (score/issues/suggestion), so a rubric describing a different output format won't be honored.
• evaluate_fn disables assess_complexity()'s real logic. Once you supply a fully custom grader, complexity assessment (used by spawn nesting) always returns True — it has no way to reuse your custom function.
• adaptive and procedural_memory both require verifier. Neither has any effect if you set them on a LoopConfig without also setting verifier.

See also

• Looping — how LoopConfig.verifier integrates with convergence, adaptive early-stop, and refinement.
• Procedural Memory — verifier feedback also feeds the evolving playbook.
• Agent Spawning — assess_complexity() gates nested spawn-tool attachment.