---
title: Using Models
llms-description: Models provide a uniform API for evaluating a variety of large language models and using models within evaluations.
---

## Overview

Inspect has support for a wide variety of language model APIs and can be extended to support arbitrary additional ones. Support for the following providers is built in to Inspect:

{{< include _model-providers.md >}}

Below we'll describe various ways to specify and provide options to models in Inspect evaluations. Review this first, then see the provider-specific sections for additional usage details and available options.

## Selecting a Model

To select a model for an evaluation, pass it's name on the command line or use the `model` argument of the `eval()` function:

``` bash
inspect eval arc.py --model openai/gpt-4o-mini
inspect eval arc.py --model anthropic/claude-sonnet-4-0
```

Or:

``` python
eval("arc.py", model="openai/gpt-4o-mini")
eval("arc.py", model="anthropic/claude-sonnet-4-0")
```

Alternatively, you can set the `INSPECT_EVAL_MODEL` environment variable (either in the shell or a `.env` file) to select a model externally:

``` bash
INSPECT_EVAL_MODEL=google/gemini-2.5-pro
```

#### No Model

Some evaluations will either not make use of models or call the lower-level `get_model()` function to explicitly access models for different roles (see the [Model API](#model-api) section below for details on this).

In these cases, you are not required to specify a `--model`. If you happen to have an `INSPECT_EVAL_MODEL` defined and you want to prevent your evaluation from using it, you can explicitly specify no model as follows:

``` bash
inspect eval arc.py --model none
```

Or from Python:

``` python
eval("arc.py", model=None)
```

## Generation Config

There are a variety of configuration options that affect the behaviour of model generation. There are options which affect the generated tokens (`temperature`, `top_p`, etc.) as well as the connection to model providers (`timeout`, `max_retries`, etc.)

You can specify generation options either on the command line or in direct calls to `eval()`. For example:

``` bash
inspect eval arc.py --model openai/gpt-4 --temperature 0.9
inspect eval arc.py --model google/gemini-2.5-pro --max-connections 20
```

Or:

``` python
eval("arc.py", model="openai/gpt-4", temperature=0.9)
eval("arc.py", model="google/gemini-2.5-pro", max_connections=20)
```

Use `inspect eval --help` to learn about all of the available generation config options.

## Model Args

If there is an additional aspect of a model you want to tweak that isn't covered by the `GenerateConfig`, you can use model args to pass additional arguments to model clients. For example, here we specify the `location` option for a Google Gemini model:

``` bash
inspect eval arc.py --model google/gemini-2.5-pro -M location=us-east5
```

See the documentation for the requisite model provider for information on how model args are passed through to model clients.

## Max Connections

Inspect uses an asynchronous architecture to run task samples in parallel. If your model provider can handle 100 concurrent connections, then Inspect can utilise all of those connections to get the highest possible throughput. The limiting factor on parallelism is therefore not typically local parallelism (e.g. number of cores) but rather what the underlying rate limit is for your interface to the provider.

By default, Inspect uses a `max_connections` value of 10. You can increase this consistent with your account limits. If you are experiencing rate-limit errors you will need to experiment with the `max_connections` option to find the optimal value that keeps you under the rate limit (the section on [Parallelism](parallelism.qmd) includes additional documentation on how to do this).

## Model API {#model-api}

The `--model` which is set for an evaluation is automatically used by the `generate()` solver, as well as for other solvers and scorers built to use the currently evaluated model. If you are implementing a `Solver` or `Scorer` and want to use the currently evaluated model, call `get_model()` with no arguments:

``` python
from inspect_ai.model import get_model

model = get_model()
response = await model.generate("Say hello")
```

If you want to use other models in your solvers and scorers, call `get_model()` with an alternate model name, along with optional generation config. For example:

``` python
model = get_model("openai/gpt-4o")

model = get_model(
    "openai/gpt-4o",
    config=GenerateConfig(temperature=0.9)
)
```

You can also pass provider specific parameters as additional arguments to `get_model()`. For example:

``` python
model = get_model("hf/openai-community/gpt2", device="cuda:0")
```

### Model Caching

By default, calls to `get_model()` are memoized, meaning that calls with identical parameters resolve to a cached version of the model. You can disable this by passing `memoize=False`:

``` python
model = get_model("openai/gpt-4o", memoize=False)
```

Finally, if you prefer to create and fully close model clients at their place of use, you can use the async context manager built in to the `Model` class. For example:

``` python
async with get_model("openai/gpt-4o") as model:
    eval(mytask(), model=model)
```

If you are not in an async context there is also a sync context manager available:

``` python
with get_model("hf/Qwen/Qwen2.5-72B") as model:
    eval(mytask(), model=model)
```

Note though that this *won't work* with model providers that require an async close operation (OpenAI, Anthropic, Grok, Together, Groq, Ollama, llama-cpp-python, and CloudFlare).

## Model Roles

Model roles enable you to create aliases for the various models used in your tasks, and then dynamically vary those roles when running an evaluation. For example, you might have a "critic" or "monitor" role, or perhaps "red_team" and "blue_team" roles. Roles are included in the log and displayed in model events within the transcript.

Here is a scorer that utilises a "grader" role when binding to a model:

``` python
@scorer(metrics=[accuracy(), stderr()])
def model_grader() -> Scorer:
    async def score(state: TaskState, target: Target):
        model = get_model(role="grader")
        ...
```

By default if there is no "grader" role specified, the default model for the evaluation will be returned.

Model roles can be specified in several ways:

**In the task definition:**

``` python
Task(
    ...,
    model_roles={"grader": "openai/gpt-4o"}
)
```

**With generation config in the task definition:**

``` python
Task(
    ...,
    model_roles={
        "grader": {
            "model": "openai/gpt-4o",
            "temperature": 0.5,
            "max_tokens": 2048
        }
    }
)
```

**With `task_with()`:**

``` python
task_with(my_task(), model_roles={"grader": "google/gemini-2.0-flash"})
```

**With `eval()`:**

``` python
eval("math.py", model_roles={"grader": "google/gemini-2.0-flash"})
```

**On the CLI** with simple model names:

``` bash
inspect eval math.py --model-role grader=google/gemini-2.0-flash
```

**On the CLI** with inline JSON/YAML for generation config:

``` bash
# JSON
inspect eval math.py \
    --model-role 'grader={"model": "openai/gpt-4o", "temperature": 0.5}'
# YAML
inspect eval math.py \
    --model-role 'grader={model: openai/gpt-4o, temperature: 0.5}'
```

Note that the built-in [model-graded scorers](scorers.qmd#model-graded) (e.g. `model_graded_qa()`, `model_graded_fact()`) look for the `grader` role by default.

For how model roles fit into the broader override and precedence model, see [Task Configuration](task-configuration.qmd#model-roles).

### Role Resolution

Model roles are resolved based on what is passed to `eval()`. This means that if you fully construct tasks before calling `eval()` (e.g. by calling their `@task` function) then the initialization code for tasks, solvers, and scorers for can't see the model role definitions. 

Given this, you should always call `get_model()` _inside_ the implementation of your solver or scorer function rather than during initialization. For example:

**Don't do this (model role not yet visible)**

``` python
@scorer(metrics=[accuracy(), stderr()])
def model_grader() -> Scorer:
    model = get_model(role="grader") # <1> 
    async def score(state: TaskState, target: Target):   
        ...
```

1. Role is not yet visible when `@task` function is called before `eval()`.

**Rather do this (defer until role is visible)**

``` python
@scorer(metrics=[accuracy(), stderr()])
def model_grader() -> Scorer:
    async def score(state: TaskState, target: Target):  
        model = get_model(role="grader") # <1>  
        ...
```

1. Role is visible since we are calling this after `eval()`.

### Role Defaults

By default if there is a no role explicitly defined then `get_model(role="...")` will return the default model for the evaluation. You can specify an alternate default model as follows:

``` python
model = get_model(role="grader", default="openai/gpt-4o")
```

This means that you can use model roles as a means of external configurability even if you aren't yet explicitly taking advantage of them.

### Roles for Tasks

In some cases it may not be convenient to specify `model_roles` in the top level call to `eval()`. For example, you might be running an [Eval Set](eval-sets.qmd) to explore the behaviour of different models for a given role. In this case, do not specify `model_roles` at the eval level, rather, specify them at the task level.

For example, imagine we have a task named `blues_clues` that we want to vary the red and blue teams for in an eval set:

``` python
from inspect_ai import eval_set, task_with
from ctf_tasks import blues_clues 

tasks = [
    task_with(blues_clues(), model_roles = {
        "red_team": "openai/gpt-4o",
        "blue_team": "google/gemini-2.0-flash"
    }),()
    task_with(blues_clues, model_roles = {
        "red_team": "google/gemini-2.0-flash",
        "blue_team": "openai/gpt-4o"
    })
]

eval_set(tasks, log_dir="...")
```

Note that we also don't specify a `model` for this eval (it doesn't have a main model but rather just the red and blue team roles).

As illustrated above, you can define as many named roles as you need. When using `eval()` or `Task` roles are specified using a dictionary. When using `inspect eval` you can include multiple `--model-role` options on the command line:

``` bash
inspect eval math.py \
   --model-role red_team=google/gemini-2.0-flash \
   --model-role blue_team=openai/gpt-4o-mini
```

## Learning More

-   [Providers](providers.qmd) covers usage details and available options for the various supported providers.

-   [Caching](caching.qmd) explains how to cache model output to reduce the number of API calls made.

-   [Compaction](compaction.qmd) covers compacting message histories for long-running agents that exceed the context window.

-   [Multimodal](multimodal.qmd) describes the APIs available for creating multimodal evaluations (including images, audio, and video).

-   [Reasoning](reasoning.qmd) documents the additional options and data available for reasoning models.

-   [Batch Mode](models-batch.qmd) covers using batch processing APIs for model inference.

-   [Structured Output](structured.qmd) explains how to constrain model output to a particular JSON schema.