Weco — LangSmith Setup

Connect to LangSmith

We need your API key to access your datasets and evaluators.

Select a Dataset

Choose the dataset to evaluate your target function against.

Dataset Preview

Sample rows from your dataset.

Source Files & Target

Select the files Weco will optimize, then choose the target function.

Source Files

Select the Python files that Weco will modify during optimization.

Target Function

Function (module:function)

e.g. agent:run_chain imports run_chain from agent.py

Target Adapter

How does your target function accept inputs?

Raw default

Pass the full inputs dict directly.

def target(inputs: dict) -> dict

LangChain

Your target is a LangChain Runnable. We call .invoke(inputs).

Single Input

Extracts a single string from common keys (input, question, text, query).

Evaluators

Evaluators score your target function's outputs on each dataset example.

Evaluator Files

Select Python files containing your evaluator functions.

Code Evaluators

Space-separated. Use module:function for custom evaluators, or built-in names like exact_match. Leave empty to default to your metric name ().

Dashboard Evaluators

Dashboard evaluators run asynchronously on the LangSmith server. Enter the Feedback Key of each evaluator as configured in your LangSmith dashboard.

Metric Function (optional)

A custom function that receives all aggregated evaluator scores as a dict and returns a single float. Use this to combine multiple evaluators into one composite metric (e.g. a weighted average).

Run Settings

Configure optimization parameters. All fields have sensible defaults.

Optimization

Metric

The metric name your evaluator outputs

Goal

Steps

Optimization iterations (default: 100)

Model

LLM model. Leave blank for auto.

Additional Instructions

LangSmith

Summary Aggregation

Experiment Prefix

Max Examples

Limit examples per eval

Max Concurrency

Parallel eval threads

Dashboard Evaluator Timeout

Seconds to poll for async scores (default: 900)

Execution

Eval Timeout

Seconds per evaluation

Log Directory

Save execution logs and code snapshots

Auto-apply best solution

Require manual review of each change

Review & Start

Confirm your configuration and begin optimization.

Next time, skip the wizard with: