---
title: Tasks
llms-description: Tasks bring together datasets, solvers, and scorers to define an evaluation. Strategies for creating flexible and re-usable tasks.
---

## Overview

This article documents both basic and advanced use of Inspect tasks, which are the fundamental unit of integration for datasets, solvers, and scorers. The following topics are explored:

-   [Task Basics](#task-basics) describes the core components and options of tasks.
-   [Parameters](#parameters) covers adding parameters to tasks to make them flexible and adaptable.
-   [Solvers](#solvers) describes how to create tasks that can be used with many different solvers.
-   [Task Reuse](#task-reuse) documents how to flexibly derive new tasks from existing task definitions.
-   [Packaging](#packaging) illustrates how you can distribute tasks within Python packages.
-   [Exploratory](#exploratory) provides guidance on doing exploratory task and solver development.

## Task Basics {#task-basics}

Tasks provide a recipe for an evaluation consisting minimally of a dataset, a solver, and a scorer (and possibly other options) and is returned from a function decorated with `@task`. For example:

``` python
from inspect_ai import Task, task
from inspect_ai.dataset import json_dataset
from inspect_ai.scorer import model_graded_fact
from inspect_ai.solver import chain_of_thought, generate

@task
def security_guide():
    return Task(
        dataset=json_dataset("security_guide.json"),
        solver=[chain_of_thought(), generate()],
        scorer=model_graded_fact()
    )
```

For convenience, tasks always define a default solver. That said, it is often desirable to design tasks that can work with *any* solver so that you can experiment with different strategies. The [Solvers](#solvers) section below goes into depth on how to create tasks that can be flexibly used with any solver.

### Task Options

While many tasks can be defined with only a dataset, solver, and scorer, there are lots of other useful `Task` options. We won't describe these options in depth here, but rather provide a list along with links to other sections of the documentation that cover their usage:

+------------------+-------------------------------------------------------------------------------------------------+-----------------------------------------------------------+
| Option           | Description                                                                                     | Docs                                                      |
+==================+=================================================================================================+===========================================================+
| `epochs`         | Epochs to run for each dataset sample.                                                          | [Epochs](scorers.qmd#reducing-epochs)                     |
+------------------+-------------------------------------------------------------------------------------------------+-----------------------------------------------------------+
| `setup`          | Setup solver(s) to run prior to the main solver.                                                | [Sample Setup](#setup-parameter)                          |
+------------------+-------------------------------------------------------------------------------------------------+-----------------------------------------------------------+
| `cleanup`        | Cleanup function to call at task completion                                                     | [Task Cleanup](#task-cleanup)                             |
+------------------+-------------------------------------------------------------------------------------------------+-----------------------------------------------------------+
| `sandbox`        | Sandbox configuration for un-trusted code execution.                                            | [Sandboxing](sandboxing.qmd)                              |
+------------------+-------------------------------------------------------------------------------------------------+-----------------------------------------------------------+
| `approval`       | Approval policy for tool calls.                                                                 | [Tool Approval](approval.qmd)                             |
+------------------+-------------------------------------------------------------------------------------------------+-----------------------------------------------------------+
| `metrics`        | Metrics to use in place of scorer metrics.                                                      | [Scoring Metrics](scorers.qmd#scoring-metrics)            |
+------------------+-------------------------------------------------------------------------------------------------+-----------------------------------------------------------+
| `model`          | Model for evaluation (note that model is typically specified by `eval` rather than in the task) | [Models](models.qmd)                                      |
+------------------+-------------------------------------------------------------------------------------------------+-----------------------------------------------------------+
| `config`         | Config for model generation (also typically specified in `eval`).                               | [Generate Config](options.qmd#model-generation)           |
+------------------+-------------------------------------------------------------------------------------------------+-----------------------------------------------------------+
| `fail_on_error`  | Failure tolerance for samples.                                                                  | [Failure Threshold](handling-errors.qmd#failure-threshold) |
+------------------+-------------------------------------------------------------------------------------------------+-----------------------------------------------------------+
| `message_limit`\ | Limits to apply to sample execution.                                                            | [Sample Limits](setting-limits.qmd#sample-limits)      |
| `token_limit`\   |                                                                                                 |                                                           |
| `time_limit`\    |                                                                                                 |                                                           |
| `working_limit`  |                                                                                                 |                                                           |
+------------------+-------------------------------------------------------------------------------------------------+-----------------------------------------------------------+
| `name`\          | Eval log attributes for task.                                                                   | [Eval Logs](eval-logs.qmd)                                |
| `version`\       |                                                                                                 |                                                           |
| `metadata`       |                                                                                                 |                                                           |
+------------------+-------------------------------------------------------------------------------------------------+-----------------------------------------------------------+

: {tbl-colwidths=\[25,50,25\]}

You by and large don't need to worry about these options until you want to use the features they are linked to.

## Parameters {#parameters}

Task parameters make it easy to run variants of your task without changing its source code. Task parameters are simply the arguments to your `@task` decorated function. For example, here we provide parameters (and default values) for system and grader prompts, as well as the grader model:

``` {.python filename="security.py"}
from inspect_ai import Task, task
from inspect_ai.dataset import example_dataset
from inspect_ai.scorer import model_graded_fact
from inspect_ai.solver import generate, system_message

@task
def security_guide(
    system="devops.txt", 
    grader="expert.txt",
    grader_model="openai/gpt-4o"
):
   return Task(
      dataset=example_dataset("security_guide"),
      solver=[system_message(system), generate()],
      scorer=model_graded_fact(
          template=grader, model=grader_model
      )
   )
```

Let's say we had an alternate system prompt in a file named `"researcher.txt"`. We could run the task with this prompt as follows:

``` bash
inspect eval security.py -T system="researcher.txt"
```

The `-T` CLI flag is used to specify parameter values. You can include multiple `-T` flags. For example:

``` bash
inspect eval security.py \
   -T system="researcher.txt" -T grader="hacker.txt"
```

If you have several task parameters you want to specify together, you can put them in a YAML or JSON file and use the `--task-config` CLI option. For example:

``` {.yaml filename="config.yaml"}
system: "researcher.txt"
grader: "hacker.txt"
```

Reference this file from the CLI with:

``` bash
inspect eval security.py --task-config=config.yaml
```

For a broader view of how task parameters relate to `task_with()`, environment variables, `eval()`, and CLI overrides, see [Task Configuration](task-configuration.qmd).

## Solvers {#solvers}

While tasks always include a *default* solver, you can also vary the solver to explore other strategies and elicitation techniques. This section covers best practices for creating solver-independent tasks.

### Solver Parameter

You can substitute an alternate solver for the solver that is built in to your `Task` using the `--solver` command line parameter (or `solver` argument to the `eval()` function).

For example, let's start with a simple CTF challenge task:

``` python
from inspect_ai import Task, task
from inspect_ai.solver import generate, use_tools
from inspect_ai.tool import bash, python
from inspect_ai.scorer import includes

@task
def ctf():
    return Task(
        dataset=read_dataset(),
        solver=[
            use_tools([
                bash(timeout=180), 
                python(timeout=180)
            ]),
            generate()
        ],
        sandbox="docker",
        scorer=includes()
    )
```

This task uses the most naive solver possible (a simple tool use loop with no additional elicitation). That might be okay for initial task development, but we'll likely want to try lots of different strategies. We start by breaking the `solver` into its own function and adding an alternative solver that uses a `react()` agent

``` python
from inspect_ai import Task, task
from inspect_ai.agent import react
from inspect_ai.dataset._dataset import Sample
from inspect_ai.scorer import includes
from inspect_ai.solver import chain, generate, solver, use_tools
from inspect_ai.tool import bash, python


@solver
def ctf_tool_loop():
    return chain([
        use_tools([
            bash(timeout=180), 
            python(timeout=180)
        ]),
        generate()
    ])

@solver
def ctf_agent(attempts: int = 3):
    return react(
        tools=[bash(timeout=180), python(timeout=180)],
        attempts=attempts,
    )

 
@task
def ctf():
    # return task
    return Task(
        dataset=read_dataset(),
        solver=ctf_tool_loop(),
        sandbox="docker",
        scorer=includes(),
    )
```

Note that we use the `chain()` function to combine multiple solvers into a composite one.

You can now switch between solvers when running the evaluation:

``` bash
# run with the default solver (ctf_tool_loop)
inspect eval ctf.py

# run with the ctf agent solver
inspect eval ctf.py --solver=ctf_agent

# run with a different number of attempts
inspect eval ctf.py --solver=ctf_agent -S attempts=5
```

Note the use of the `-S` CLI option to pass an alternate value for `attempts` to the `ctf_agent()` solver.

### Setup Parameter {#setup-parameter}

In some cases, there will be important steps in the setup of a task that *should not be substituted* when another solver is used with the task. For example, you might have a step that does dynamic prompt engineering based on values in the sample `metadata` or you might have a step that initialises resources in a sample's sandbox.

In these scenarios you can define a `setup` solver that is always run even when another `solver` is substituted. For example, here we adapt our initial example to include a `setup` step:

``` python
# prompt solver which should always be run
@solver
def ctf_prompt():
    async def solve(state, generate):
        # TODO: dynamic prompt engineering
        return state

    return solve

@task
def ctf(solver: Solver | None = None):
    # use default tool loop solver if no solver specified
    if solver is None:
        solver = ctf_tool_loop()
   
    # return task
    return Task(
        dataset=read_dataset(),
        setup=ctf_prompt(),
        solver=solver,
        sandbox="docker",
        scorer=includes()
    )
```

## Task Cleanup {#task-cleanup}

You can use the `cleanup` parameter for executing code at the end of each sample run. The `cleanup` function is passed the `TaskState` and is called for both successful runs and runs where are exception is thrown. Extending the example from above:

``` python
async def ctf_cleanup(state: TaskState):
    ## perform cleanup
    ...

Task(
    dataset=read_dataset(),
    setup=ctf_prompt(),
    solver=solver,
    cleanup=ctf_cleanup,
    scorer=includes()
)
```

Note that like solvers, cleanup functions should be `async`.

## Task Reuse {#task-reuse}

The basic mechanism for task re-use is to create flexible and adaptable base `@task` functions (which often have many parameters) and then derive new higher-level tasks from them by creating additional `@task` functions that call the base function.

In some cases though you might not have full control over the base `@task` function (e.g. it's published in a Python package you aren't the maintainer of) but you nevertheless want to flexibly create derivative tasks from it. To do this, you can use the `task_with()` function, which provides a straightforward way to modify the properties of an existing task.

::: {.callout-tip}
For a comprehensive reference on all configuration and override mechanisms --- including `task_with()`, `eval()` overrides, CLI flags, and precedence rules --- see [Task Configuration](task-configuration.qmd).
:::

For example, imagine you are dealing with a `Task` that hard-codes its `sandbox` to a particular Dockerfile included with the task, and further hard codes its `solver` to a simple agent:

``` python
from inspect_ai import Task, task
from inspect_ai.agent import react
from inspect_ai.tool import bash
from inspect_ai.scorer import includes

@task
def hard_coded():
    return Task(
        dataset=read_dataset(),
        solver=react(tools=[bash()]),
        sandbox=("docker", "compose.yaml"),
        scorer=includes()
    )
```

Using `task_with()`, you can adapt this task to use a different `solver` and `sandbox` entirely. For example, here we import the original `hard_coded()` task from a hypothetical `ctf_tasks` package and provide it with a different `solver` and `sandbox`, as well as give it a `message_limit` (which we in turn also expose as a parameter of the adapted task):

``` python
from inspect_ai import task, task_with
from inspect_ai.solver import solver

from ctf_tasks import hard_coded

@solver
def my_custom_agent():
    ## custom agent implementation
    ...

@task
def adapted(message_limit: int = 20):
    return task_with(
        hard_coded(),  # original task definition
        solver=my_custom_agent(),
        sandbox=("docker", "custom-compose.yaml"),
        message_limit=message_limit
    )
```

Tasks are recipes for an evaluation and represent the convergence of many considerations (datasets, solvers, sandbox environments, limits, and scoring). Task variations often lie at the intersection of these, and the `task_with()` function is intended to help you produce exactly the variation you need for a given evaluation.

Note that `task_with()` modifies the passed task in-place, so if you want to create multiple variations of a single task using `task_with()` you should create the underlying task multiple times (once for each call to `task_with()`). For example:

``` python
adapted1 = task_with(hard_coded(), ...)
adapted2 = task_with(hard_coded(), ...)
```

## Packaging {#packaging}

A convenient way to distribute tasks is to include them in a Python package. This makes it very easy for others to run your task and ensure they have all of the required dependencies.

Tasks in packages can be *registered* such that users can easily refer to them by name from the CLI. For example, the [Inspect Evals](https://github.com/UKGovernmentBEIS/inspect_ai) package includes a suite of tasks that can be run as follows:

``` bash
inspect eval inspect_evals/gaia 
inspect eval inspect_evals/swe_bench
```

### Example

Here's an example that walks through all of the requirements for registering tasks in packages. Let's say your package is named `evals` and has a task named `mytask` in the `tasks.py` file:

```         
evals/       
  evals/
    tasks.py
    _registry.py
  pyproject.toml
```

The `_registry.py` file serves as a place to import things that you want registered with Inspect. For example:

``` {.python filename="_registry.py"}
from .tasks import mytask
```

You can then register `mytask` (and anything else imported into `_registry.py`) as a [setuptools entry point](https://setuptools.pypa.io/en/latest/userguide/entry_point.html). This will ensure that inspect can resolve references to your package from the CLI. Here is how this looks in `pyproject.toml`:

::: {.panel-tabset group="entry-points"}
## Setuptools

``` toml
[project.entry-points.inspect_ai]
evals = "evals._registry"
```

## Poetry

``` toml
[tool.poetry.plugins.inspect_ai]
evals = "evals._registry"
```
:::

Now, anyone that has installed your package can run the task as follows:

``` bash
inspect eval evals/mytask
```

## Hugging Face

Datasets hosted on Hugging Face Hub can include an `eval.yaml` file that provides Inspect task definitions. For example, the [OpenEvals/aime_24](https://huggingface.co/datasets/OpenEvals/aime_24) dataset can be evaluated with:

``` bash
inspect eval hf/OpenEvals/aime_24 --model openai/gpt-5
```

Here are the `eval.yaml` definitions for several Hugging Face datasets:

-   [OpenEvals/aime_24](https://huggingface.co/datasets/OpenEvals/aime_24/blob/main/eval.yaml)
-   [OpenEvals/SimpleQA](https://huggingface.co/datasets/OpenEvals/SimpleQA/blob/main/eval.yaml)
-   [OpenEvals/MuSR](https://huggingface.co/datasets/OpenEvals/MuSR/blob/main/eval.yaml)

A dataset's `eval.yaml` file defines a list of tasks. Here are the fields that can be included in a task definition and how they are used in constructing `Task` instances:

+-------------------+----------------+--------------------------------+
| Field             | Default        | Usage                          |
+===================+================+================================+
| `config`          | "default"      | `hf_dataset(name)`             |
+-------------------+----------------+--------------------------------+
| `split`           | "test"         | `hf_dataset(split)`            |
+-------------------+----------------+--------------------------------+
| `field_spec`      | None           | `hf_dataset(sample_fields)`    |
+-------------------+----------------+--------------------------------+
| `shuffle_choices` | None           | `dataset.shuffle_choices()`    |
+-------------------+----------------+--------------------------------+
| `epochs`          | 1              | `Epochs(epochs)`               |
+-------------------+----------------+--------------------------------+
| `epoch_reducer`   | "mean"         | `Epochs(epoch_reducer)`        |
+-------------------+----------------+--------------------------------+
| `solvers`         | None           | `Task(solver)`                 |
+-------------------+----------------+--------------------------------+
| `scorer`          | None           | `Task(scorer)`                 |
+-------------------+----------------+--------------------------------+
| `id`              | None           | `hf/org/dataset/name`          |
+-------------------+----------------+--------------------------------+

- `field_spec.choices` can be either a single string (the key for one field in each record) or a list of strings (multiple fields, whose values will form the choices list for each sample).
- `field_spec.target` can be:
    - A literal value, specified as `literal:<value>`, where `<value>` will be used directly as the target.
    - A field name corresponding to a letter, or an integer; in this case, the integer (e.g., 0, 1, 2) will be mapped to a letter (`A`, `B`, `C`, etc.) for use as the target.
- `field_spec.input_image` is an optional field name for multimodal tasks. When specified, it should reference a field containing image data as a data URI (base64 encoded). The image will be combined with the text input to create a multimodal chat message. For example:

### Multiple Tasks

Datasets can define multiple named tasks. For example, the [OpenEvals/MuSR](https://huggingface.co/datasets/OpenEvals/MuSR/blob/main/eval.yaml) dataset defines 3 tasks: `musr:murder_mysteries`, `musr:object_placements`, and `musr:team_allocation`. If you call `inspect eval` with no task qualification, all 3 tasks will be run. If you append a task name, only that task will be run:

```bash
# run all 3 tasks defined by OpenEvals/MuSR
inspect eval hf/OpenEvals/MuSR --model openai/gpt-5

# run only the musr:murder_mysteries task
inspect eval hf/OpenEvals/MuSR/musr:murder_mysteries --model openai/gpt-5
```

Note that when running multiple tasks, you may want to increase `--max-tasks` for more concurrency:

```bash
inspect eval hf/OpenEvals/MuSR --model openai/gpt-5 --max-tasks 3
```

### Revisions

All of the examples above execute evals from the `main` branch. You can alternatively execute from a branch, tag, or revision hash by appending an `@` qualifier. For example:

```bash
inspect eval hf/OpenEvals/MuSR@df154a5 --model openai/gpt-5
```

## Exploratory {#exploratory}

When developing tasks and solvers, you often want to explore how changing prompts, generation options, solvers, and models affect performance on a task. You can do this by creating multiple tasks with varying parameters and passing them all to the `eval_set()` function.

Returning to the example from above, the `system` and `grader` parameters point to files we are using as system message and grader model templates. At the outset we might want to explore every possible combination of these parameters, along with different models. We can use the `itertools.product` function to do this:

``` python
from itertools import product

# 'grid' will be a permutation of all parameters
params = {
    "system": ["devops.txt", "researcher.txt"],
    "grader": ["hacker.txt", "expert.txt"],
    "grader_model": ["openai/gpt-4o", "google/gemini-2.5-pro"],
}
grid = list(product(*(params[name] for name in params)))

# run the evals and capture the logs
logs = eval_set(
    [
        security_guide(system, grader, grader_model)
        for system, grader, grader_model in grid
    ],
    model=["google/gemini-2.5-flash", "mistral/mistral-large-latest"],
    log_dir="security-tasks"
)

# analyze the logs...
plot_results(logs)
```

Note that we also pass a list of `model` to try out the task on multiple models. This eval set will produce in total 16 tasks accounting for the parameter and model variation.

See the article on [Eval Sets](eval-sets.qmd) to learn more about using eval sets. See the article on [Eval Logs](eval-logs.qmd) for additional details on working with evaluation logs.
