{% extends "base.html" %} {% block title %}Compare - Scenario Result Viewer{% endblock %} {% block breadcrumb %} {% endblock %} {% macro outcome_badge(r) %} {% if r is none %}not run {% elif r.outcome == "pass" %}PASS {% elif r.outcome == "fail" %}FAIL {% elif r.outcome == "error" %}ERROR {% else %}N/A{% endif %} {% endmacro %} {% macro pct(value) %}{% if value is none %}-{% else %}{{ "%.0f" | format(value * 100) }}%{% endif %}{% endmacro %} {% block content %} {% if oo_styles %}{% endif %}

Compare runs

Cases are matched by what they ran (their sweep overrides, else the ODD values they were drawn at), not by their position. The oldest session picked is the baseline.

Sessions{% if selected %} of the same scenarios{% endif %}
{% for ref, s in offered %} {% endfor %}
{{ s.date }} {{ s.time }} {{ s.session_type }} {{ s.scenario_name.split(',')[0].split('#')[0] }} {{ s.passed_count }}/{{ s.total_count }}
{% if attributes %} {% endif %} Pick two or more.
{% if comparison %} {% set last = comparison.sessions | length - 1 %}
{% for ref in comparison.sessions %}
{{ ref }}{% if loop.first %} (baseline){% endif %}
{{ pct(comparison.rate(loop.index0)) }}
{{ comparison.totals[loop.index0][0] }}/{{ comparison.totals[loop.index0][1] }} passed
{% endfor %}
First to last run of each case
{{ comparison.count('regression') }} regression(s)
{{ comparison.count('fixed') }} fixed, {{ comparison.count('flaky') }} flaky, {{ comparison.count('new') }} new, {{ comparison.count('gone') }} gone
{% if comparison.x and comparison.cells %}

Pass rate per ODD cell

The buckets each case was drawn in. Each cell shows the pass rate of every session (baseline first); red: lower in the latest, green: higher.

{% for bx in comparison.x_labels %}{% endfor %} {% for by in comparison.y_labels %} {% for bx in comparison.x_labels %} {% set cell = comparison.cells.get((bx, by)) %} {% if cell %} {% set d = cell.delta %} {% else %} {% endif %} {% endfor %} {% endfor %}
{{ comparison.y or "" }} \ {{ comparison.x }}{{ bx }}
{{ by }} {% for t in cell.tallies %}
{% if t[1] %}{{ t[0] }}/{{ t[1] }}{% else %}-{% endif %}
{% endfor %}
{% endif %} {% if matrices %}
Coverage matrix of each session
{% for ref, html in matrices %}
{% if html %}{{ html | safe }}{% else %}

No coverage files.

{% endif %}
{% endfor %}
{% endif %}

Cases

{% for ref in comparison.sessions %}{% endfor %} {% for key in comparison.metrics %}{% endfor %} {% for c in comparison.cases %} {% set change = c.change() %} {% for r in c.runs %} {% endfor %} {% for key in comparison.metrics %} {% set d = c.delta(key) %} {% set better = metric_names[key][1] %} {% endfor %} {% endfor %}
Change Case{{ ref.split('/', 1)[1] }}Δ {{ metric_names[key][0] }}
{% if change == "regression" %}pass → fail {% elif change == "fixed" %}fail → pass {% elif change == "flaky" %}flaky {% else %}{{ change }}{% endif %}
{{ c.first.label }}
{% for name, bucket in c.first.buckets.items() %}{% if name in (comparison.x, comparison.y) %}{{ name.split('.')[-1] }} {{ bucket }}{% endif %}{% endfor %}
{% if r %}{{ outcome_badge(r) }}{% else %}{{ outcome_badge(none) }}{% endif %} {% if d is none %}-{% else %}{{ "%+.2f" | format(d) }}{% endif %} {% set ns = namespace(q=[]) %} {% for r in c.runs %}{% if r %}{% set ns.q = ns.q + ["run=" ~ (comparison.sessions[loop.index0] ~ "/" ~ r.index) | urlencode] %}{% endif %}{% endfor %} {% if ns.q | length > 1 %}side by side{% endif %}
{% endif %} {% endblock %} {% block extra_scripts %} {% if oo_script %}{% endif %} {% endblock %}