{% extends "base.html" %} {% block title %}Compare - Scenario Result Viewer{% endblock %} {% block breadcrumb %} {% endblock %} {% macro outcome_badge(r) %} {% if r is none %}not run {% elif r.outcome == "pass" %}PASS {% elif r.outcome == "fail" %}FAIL {% elif r.outcome == "error" %}ERROR {% else %}N/A{% endif %} {% endmacro %} {% macro pct(value) %}{% if value is none %}-{% else %}{{ "%.0f" | format(value * 100) }}%{% endif %}{% endmacro %} {% block content %} {% if oo_styles %}{% endif %}
Cases are matched by what they ran (their sweep overrides, else the ODD values they were drawn at), not by their position; every run of a case a session repeats is kept, and a case a session both passed and failed is flaky. The oldest session picked is the baseline.
{% if comparison %} {% set last = comparison.sessions | length - 1 %}The buckets each case was drawn in. Each cell shows the passes and runs of every session (baseline first), every run of a repeated case counted; red: a lower pass rate in the latest, green: higher.
| {{ comparison.y or "" }} \ {{ comparison.x }} | {% for bx in comparison.x_labels %}{{ bx }} | {% endfor %}|
|---|---|---|
| {{ by }} | {% for bx in comparison.x_labels %} {% set cell = comparison.cells.get((bx, by)) %} {% if cell %} {% set d = cell.delta %}
{% for t in cell.tallies %}
{% if t[1] %}{{ t[0] }}/{{ t[1] }}{% else %}-{% endif %}
{% endfor %}
|
{% else %}
{% endif %} {% endfor %} |
{{ name }}{% if not loop.last %}, {% endif %}{% endfor %}No coverage files.
{% endif %}| Change | Case | {% for ref in comparison.sessions %}{{ ref.split('/', 1)[1] }} | {% endfor %} {% for key in comparison.metrics %}Δ {{ metric_names[key][0] }} | {% endfor %}|
|---|---|---|---|---|
| {% if change == "regression" %}pass → fail {% elif change == "fixed" %}fail → pass {% elif change == "flaky" %}flaky {% else %}{{ change }}{% endif %} |
{{ c.first.label }}
{% for name, bucket in c.first.buckets.items() %}{% if name in (comparison.x, comparison.y) %}{{ name.split('.')[-1] }} {{ bucket }}{% endif %}{% endfor %}
|
{% for runs in c.runs %}
{% set ref = comparison.sessions[loop.index0] %}
{% set t = c.tally(loop.index0) %}
{% for r in runs %}{{ outcome_badge(r) }}{% if not loop.last %} {% endif %}{% else %}{{ outcome_badge(none) }}{% endfor %}
{% if t[1] > 1 %} {{ t[0] }}/{{ t[1] }} passed {% endif %}
|
{% endfor %}
{% for key in comparison.metrics %}
{% set d = c.delta(key) %}
{% set better = metric_names[key][1] %}
{% if d is none %}-{% else %}{{ "%+.2f" | format(d) }}{% endif %} | {% endfor %}{# The first run of the first session that ran it, and the last run of the last. #} {% set ns = namespace(q=[]) %} {% for runs in c.runs %}{% if runs %}{% set ref = comparison.sessions[loop.index0] %}{% if not ns.q %}{% set ns.q = ["run=" ~ (ref ~ "/" ~ runs[0].index) | urlencode] %}{% else %}{% set ns.q = [ns.q[0], "run=" ~ (ref ~ "/" ~ runs[-1].index) | urlencode] %}{% endif %}{% endif %}{% endfor %} {% if ns.q | length > 1 %}side by side{% endif %} |