# Inspect AI

> Inspect AI is a Python framework for large language model evaluations created by the [UK AI Safety Institute](https://aisi.gov.uk). Inspect provides many built-in components, including facilities for prompt engineering, tool usage, multi-turn dialog, and model graded evaluations. Extensions to Inspect (e.g. to support new elicitation and scoring techniques) can be provided by other Python packages.

## Docs

- [Tutorial](https://inspect.ai-safety-institute.org.uk/tutorial.html.md): Step-by-step walkthroughs of several basic examples of Inspect evaluations.
- [Workflow](https://inspect.ai-safety-institute.org.uk/workflow.html.md): Covers the mechanics of running evaluations, including how to create evals in both scripts and notebooks, specifying configuration and options, how to parameterise tasks for different scenarios, and how to work with eval log files.
- [Log Viewer](https://inspect.ai-safety-institute.org.uk/log-viewer.html.md): Goes into more depth on how to use Inspect View to develop and debug evaluations, including how to provide additional log metadata and how to integrate it with Python's standard logging module.
- [VS Code](https://inspect.ai-safety-institute.org.uk/vscode.html.md) Provides documentation on using the Inspect VS Code Extension to run, tune, debug, and visualise evaluations.
- [Solvers](https://inspect.ai-safety-institute.org.uk/solvers.html.md): Solvers are the heart of Inspect, and encompass prompt engineering and various other elicitation strategies. Here we cover using the built-in solvers and creating your own more sophisticated ones.
- [Tools](https://inspect.ai-safety-institute.org.uk/tools.html.md): Tools provide a means of extending the capabilities of models by registering Python functions for them to call. This section describes how to create custom tools and use them in evaluations.
- [Scorers](https://inspect.ai-safety-institute.org.uk/scorers.html.md): Scorers evaluate the work of solvers and aggregate scores into metrics. Sophisticated evals often require custom scorers that use models to evaluate output. This section covers how to create them.
- [Datasets](https://inspect.ai-safety-institute.org.uk/datasets.html.md): Datasets provide samples to evaluation tasks. This section illustrates how to adapt various data sources for use with Inspect, as well as how to include multi-modal data (images, etc.) in your datasets.
- [Models](https://inspect.ai-safety-institute.org.uk/models.html.md): Models provide a uniform API for both evaluating a variety of large language models and using models within evaluations (e.g. for critique or grading).
- [Agents](https://inspect.ai-safety-institute.org.uk/agents.html.md): Agents combine planning, memory, and tool usage to pursue more complex, longer horizon tasks. This section describes how to build agent evaluations with Inspect.
- [Sandboxing](https://inspect.ai-safety-institute.org.uk/sandboxing.html.md): Enables you to isolate code generated by models as well as set up more complex computing environments for tasks. 
- [Agents API](https://inspect.ai-safety-institute.org.uk/agents-api.html.md): This article describes advanced Inspect APIs available for creating evaluations with agents.
- [Human Agent](https://inspect.ai-safety-institute.org.uk/human-agent.html.md): This article describes the `human_agent()` solver which enables human baselining for computing tasks.
- [Approval](https://inspect.ai-safety-institute.org.uk/approval.html.md): Approvals enable you to create fine-grained policies for approving tool calls made by models.
- [Eval Logs](https://inspect.ai-safety-institute.org.uk/eval-logs.html.md): Explores how to get the most out of evaluation logs for developing, debugging, and analyzing evaluations.
- [Eval Sets](https://inspect.ai-safety-institute.org.uk/eval-sets.html.md): Covers Inspect's features for describing, running, and analysing larger sets of evaluation tasks.
- [Errors and Limits](https://inspect.ai-safety-institute.org.uk/errors-and-limits.html.md): This article covers various techniques for dealing with unexpected errors and setting limits on evaluation tasks and samples, including retrying failed evaluations, establishing a threshold of samples to tolerate errors for before failing an evaluation, and setting a maximum number of messages, tokens, or elapsed seconds in a sample before forcing the solver to give up.
- [Typing](https://inspect.ai-safety-institute.org.uk/typing.html.md): Provides guideance on using static type checking with Inspect, including creating typed interfaces to untyped storage (i.e. sample metadata and store).
- [Tracing](https://inspect.ai-safety-institute.org.uk/tracing.html.md): Describes advanced execution tracing tools used to diagnose runtime issues.
- [Caching](https://inspect.ai-safety-institute.org.uk/caching.html.md): Caching enables you to cache model output to reduce the number of API calls made, saving both time and expense.
- [Parallelism](https://inspect.ai-safety-institute.org.uk/parallelism.html.md): Delves into how to obtain maximum performance for evaluations. Inspect uses a highly parallel async architecture---here we cover how to tune this parallelism (e.g to stay under API rate limits or to not overburden local compute) for optimal throughput.
- [Interactivity](https://inspect.ai-safety-institute.org.uk/interactivity.html.md): Covers various ways to introduce user interaction into the implementation of tasks (for example, confirming consequential actions or prompting the model dynamically based on the trajectory of the evaluation).
- [Extensions](https://inspect.ai-safety-institute.org.uk/extensions.html.md) describes the various ways you can extend Inspect, including adding support for new Model APIs, tool execution environments, and storage platforms (for datasets, prompts, and logs).


