{% extends "base.html" %} {% block title %}{{ task }} side by side — Nullius{% endblock %} {% block head %} {% endblock %} {% block crumb %}/ compare / {{ task }}{% endblock %} {% import "traces/_charts.html" as ui %} {# Same shape as the trial page's `view_url`, so the shared chart macros can call it without knowing which page they are on. Only the legend state rides in it: this page has no trace filter, so `f` is always the empty one. #} {% macro view_url(f, cf=none) -%} {{ url_for('compare_task', task=task, pick=picks, cohort=cohort, dataset=dataset, **((cf if cf is not none else chart_filter).query)) }} {%- endmacro %} {% block body %}

{{ task }}

{{ panels|length }} runs side by side· back to the comparison· drag a column header to reorder

{{ 'Dataset not recorded' if dataset.startswith('[unidentified:') else dataset }} · {{ 'Matched tasks' if cohort == 'matched' else 'Whole runs' }}

{% if panels|selectattr('attempts', 'gt', 1)|list %}

A run has repeated attempts on this task. This page shows the last indexed attempt, matching the task table; all attempts remain in the comparison totals.

{% endif %}
{# One grid, sections as full-width rows. Each section's cells land on the same grid row, so every run's "Files edited" starts at the same y — which separate per-run cards can never guarantee. #}
{% for p in panels %}

{{ p.job.agent_label or p.job.job_id }}

{% if p.job.is_control %}control{% endif %} {% if p.trial %} {{ p.trial.outcome_label }} {% if p.trial.is_regression %}⚠ broke {{ p.trial.p2p_fail }}{% endif %} {% else %}not run{% endif %}
{{ p.job.job_id }} {% if p.trial %} Open full trial → {% endif %}

{{ p.selection.workspace_label }}{% if p.attempts > 1 %} · {{ p.attempts }} attempts, last indexed shown{% endif %}

{% endfor %}

Result

{% for p in panels %}
{% if not p.trial %}

Didn't include this task.

{% else %} {# Check rows come from the dataset's Vocabulary. Panels compare one task, so every panel shares a dataset and the conditional rows stay aligned across the grid — the same property the unconditional rows below rely on. #} {% set vocab = p.trial.vocab %}
{% if vocab.gained %}
{{ vocab.gained }}
{% if p.trial.f2p_pass or p.trial.f2p_fail %}{{ p.trial.f2p_pass }}/{{ p.trial.f2p_fail }}{% else %}—{% endif %}
{% endif %} {% if vocab.held %}
{{ vocab.held }}
{% if p.trial.p2p_pass or p.trial.p2p_fail %}{{ p.trial.p2p_pass }}/{{ p.trial.p2p_fail }}{% else %}—{% endif %}
{% endif %} {% if vocab.has_patch %}
Patch applied
{% if p.trial.patch_exists is none %}—{% else %}{{ 'Yes' if p.trial.patch_applied else 'No' }}{% endif %}
{% endif %}
Agent time
{{ p.trial.dur_agent|secs }}
{# Emit these two unconditionally. pane-stats is a grid, so a panel with fewer items shifts every later metric up a row and the columns stop lining up — an agent's "Tool calls" ends up level with the oracle's "Prompt size", and the oracle looks like it made tool calls. #}
Steps
{% if p.traj %}{{ p.traj.n_steps }}{% else %}—{% endif %}
Tool calls
{%- if p.traj %}{{ p.traj.calls|length }} {%- if p.traj.n_failed_calls %} ({{ p.traj.n_failed_calls }} failed){% endif %} {%- else %}—{% endif -%}
Prompt size
{{ p.trial.n_input_tokens|tokens(p.job.kind) }}
Fresh prompt
{{ (p.trial.n_input_tokens - p.trial.n_cache_tokens)|tokens(p.job.kind) }}
Cached
{{ p.trial.n_cache_tokens|tokens(p.job.kind) }}
Cached ratio
{# 0 cached over 0 prompt is undefined, not 0% — controls get "—" here even though their token counts are real zeros. #} {%- if p.trial.n_input_tokens %}{{ (100 * p.trial.n_cache_tokens / p.trial.n_input_tokens)|round|int }}%{% else %}—{% endif -%}
Output tokens
{{ p.trial.n_output_tokens|tokens(p.job.kind) }}
Cost (estimated)
{{ p.trial.cost_usd|usd(p.job.kind) }}
{% endif %}
{% endfor %}

Where the time went

{% for p in panels %}
{% if p.trial %} {# Four phases in execution order. A 0s phase emits no segment, so a control's empty setup leaves no stray sliver. #}
{% if p.trial.dur_env|phase %}{% endif %} {% if p.trial.dur_agent_setup|phase %}{% endif %} {% if p.trial.dur_agent|phase %}{% endif %} {% if p.trial.dur_verifier|phase %}{% endif %} {% if p.trial.dur_other|phase %}{% endif %}
Env {{ p.trial.dur_env|secs }} Setup {{ p.trial.dur_agent_setup|secs }} Agent {{ p.trial.dur_agent|secs }} Verify {{ p.trial.dur_verifier|secs }} Other {{ p.trial.dur_other|secs }}
{% else %}—{% endif %}
{% endfor %} {# ---- traces, side by side ---- Two sections from each side's own trace.json, and nothing across them. The charts are the trial page's, through the same macro over the same `charts_of` — a side here draws what the trial's own page draws for it. Each side is loaded alone: two trials never shared a clock, so there is no pooled population, no combined wall clock and no waterfall. A side without a trace says so in its own cell rather than vanishing. #}

Traces · each side from its own capture, nothing pooled across them

{% for p in panels %}
{% if p.charts %} {% set breakdown_tail %}Time breakdown below is this side's accounting of where its seconds went; this is the shape of the calls that spent them.{% endset %} {# `empty_note` names this page's one reason for an empty plot: there is no filter here, so "every span is filtered out" could never be it. #} {{ ui.charts_panel(p.charts, chart_filter, no_filter, view_url, tail=breakdown_tail, empty_note="None of the " ~ p.charts.total ~ " spans in this trace has a known duration — there is nothing to plot.") }} {% else %}

{{ p.absent }}

{% endif %}
{% endfor %} {# Two strips per side, never mixed and never compared by a Δ. The first is the producer's `totals`, verbatim — its own arithmetic, which differs by producer: one sums its LLM spans and can exceed its wall clock, another partitions. The second is this reader's partition over the whole tree, the same arithmetic on both sides, which is the one strip two producers' traces can honestly be read across. Shares are of each side's own wall clock; no number here adds one side's clock to the other's. #}

Time breakdown · categories, not a timeline

{% for p in panels %}
{% if p.trace and not p.trace.error %} {% set tr = p.trace %} {% set clocked = 'inferred' if tr.zero_measured else 'measured' %}

Reported by the trace {%- if tr.totals.wall_ns %} · {{ tr.totals.wall_ns|nsdur }} {{ clocked }}{% endif %} {%- if tr.zero_measured %} · inferred, not measured{% endif %}

{% if tr.totals.segments %}
{% for s in tr.totals.segments %} {% endfor %}
{% for s in tr.totals.segments %} {{ s.label }} {{ s.ns|nsdur }} {{ s.pct|share }} {% endfor %}
{% else %}

The trace reports no totals.

{% endif %}

Partitioned here · every instant charged once

{% set part = tr.baseline.segments %} {% if part %}
{% for s in part %} {% endfor %}
{% for s in part %} {{ s.label }} {{ s.ns|nsdur }} {{ s.pct|share }} {% endfor %}
{% else %}

No span carries a usable clock — nothing to partition.

{% endif %}

Two accountings of the same {{ tr.totals.wall_ns|nsdur if tr.totals.wall_ns else 'run' }}, shown apart and never mixed. The first is the producer's own and is displayed as reported — producers differ, and one that sums its LLM spans can exceed its wall clock. The second charges every instant to exactly one category (waiting on a human beats real work, which beats the call it ran inside) and is the same arithmetic on every side of this page, so it is the strip to read across. Neither is summed from the charts above: spans nest, so adding them up double-counts.

{% else %}

{{ p.absent }}

{% endif %}
{% endfor %}

Flow

{% for p in panels %}
{% if p.traj %}
{% for c in p.traj.calls %} {% endfor %}
{% for cat, n in p.traj.category_freq.most_common() %} {{ cat }} {{ n }} {% endfor %}
{% else %}—{% endif %}
{% endfor %} {% if panels|selectattr('trial')|selectattr('trial.p2p_failures')|list or panels|selectattr('trial')|selectattr('trial.f2p_failures')|list %}

Tests still failing

{% for p in panels %}
{% if p.trial and p.trial.p2p_failures %}

Broke {{ p.trial.p2p_fail }} passing:

    {% for f in p.trial.p2p_failures %}
  • {{ f }}
    • {% endfor %}
{% endif %} {% if p.trial and p.trial.f2p_failures %}
    {% for f in p.trial.f2p_failures %}
  • {{ f }}
    • {% endfor %}
{% elif not (p.trial and p.trial.p2p_failures) %}—{% endif %}
{% endfor %} {% endif %}

Files edited

{% for p in panels %}
{% if p.traj and p.traj.files_touched %}
    {% for f in p.traj.files_touched %}
  • {{ f }}
    • {% endfor %}
{% else %}None recorded.{% endif %}
{% endfor %}

Tool calls

{% for p in panels %}
{% if p.traj %}
{% for c in p.traj.calls %} {% endfor %}
#KindTargetResult
{{ c.step_id }} {{ c.category }} {{ c.target or '—' }} {% if c.failure %}× {{ c.failure }} {%- else %}ok{% endif %}
{% else %}—{% endif %}
{% endfor %}

Final message

{% for p in panels %}
{% if p.traj and p.traj.last_agent_message %}
Show 
{{ p.traj.last_agent_message }}
{% else %}—{% endif %}
{% endfor %}
{% endblock %}