Tradeoff Analysis

Local transformer vs LLM — confidence-thresholded routing
Dataset size
{{ data.n_total }} examples
Positive class
"{{ data.positive_label }}"
Transformer $ / call
${{ "%.7f"|format(data.transformer_cost_per_call) }}
Inference assumption
${{ "%.3f"|format(data.transformer_instance_hourly) }}/hr, {{ "%.1f"|format(data.transformer_samples_per_second) }} sps {% if data.used_measured_throughput %}(measured){% endif %}
{% if data.measured_samples_per_second %}
Measured throughput
{{ "%.2f"|format(data.measured_samples_per_second) }} samples/sec
{% endif %}
Transformer source
Trained model
100% transformer
F1 {{ "%.3f"|format(data.baseline_transformer.f1) }} · ${{ "%.4f"|format(data.baseline_transformer.cost) }}
{% for spec in data.llm_specs %} {% set m = data.llm_measurements.get(spec.name) if data.llm_measurements else None %}
{{ spec.name }} {% if m and data.used_full_llm_eval %} (measured acc {{ "%.3f"|format(m.accuracy) }}, n={{ m.n_samples }}, full eval → deterministic) {% elif m %} (measured acc {{ "%.3f"|format(m.accuracy) }} ±{{ "%.3f"|format(m.ci_half_width) }}, n={{ m.n_samples }}) {% elif spec.accuracy < 1.0 %} (assumed acc {{ "%.2f"|format(spec.accuracy) }}) {% endif %}
${{ "%.6f"|format(spec.cost_per_call) }}/call · 100% = ${{ "%.4f"|format(data.baselines_by_llm[spec.name].cost) }}
{% endfor %}

Tradeoff frontier

Each point is one (positive_threshold, negative_threshold) cell from the sweep. X axis: estimated total cost (transformer compute + LLM calls on the deferred slice). Y axis: chosen metric. Color: % of dataset handled by the transformer. The connected dashed line traces the Pareto frontier — combinations not dominated by any other. Hover any point for the full breakdown.
Hover a multi-tier cascade point (yellow line) above to see its routing mix.

Pareto frontier — recommended threshold combinations

Non-dominated cells for the currently-selected metric, sorted by cost. Set your run size and (optionally) a max cost — the best row within budget is starred. Rows over budget are dimmed but kept visible for comparison.
Pos t Neg t F1 Precision Recall Accuracy Coverage LLM calls Projected cost ($)
Generated by valtron_core tradeoff analysis