{% extends "base.html" %} {% block title %}Compare - Scenario Result Viewer{% endblock %} {% block breadcrumb %} {% endblock %} {% macro outcome_badge(r) %} {% if r is none %}not run {% elif r.outcome == "pass" %}PASS {% elif r.outcome == "fail" %}FAIL {% elif r.outcome == "error" %}ERROR {% else %}N/A{% endif %} {% endmacro %} {% macro pct(value) %}{% if value is none %}-{% else %}{{ "%.0f" | format(value * 100) }}%{% endif %}{% endmacro %} {% block content %} {% if oo_styles %}{% endif %}

Compare runs

Cases are matched by what they ran (their sweep overrides, else the ODD values they were drawn at), not by their position; every run of a case a session repeats is kept, and a case a session both passed and failed is flaky. The oldest session picked is the baseline.

Sessions{% if selected %} of the same scenarios{% endif %}
{% for ref, s in offered %} {% endfor %}
{{ s.date }} {{ s.time }} {{ s.session_type }} {{ s.scenario_name.split(',')[0].split('#')[0] }} {{ s.passed_count }}/{{ s.total_count }}{% if s.pending_count %} {{ s.pending_count }} pending{% endif %}
{% if attributes %} {% endif %} Pick two or more.
{% if comparison %} {% set last = comparison.sessions | length - 1 %}
{% for ref in comparison.sessions %}
{{ ref }}{% if loop.first %} (baseline){% endif %}
{% set waiting = pending[loop.index0] %}
{{ pct(comparison.rate(loop.index0)) }}
{{ comparison.totals[loop.index0][0] }}/{{ comparison.totals[loop.index0][1] }} passed{% if waiting %} · {{ waiting }} pending (not collected){% endif %}
{% endfor %}
First to last session of each case
{{ comparison.count('regression') }} regression(s)
{{ comparison.count('fixed') }} fixed, {{ comparison.count('flaky') }} flaky, {{ comparison.count('new') }} new, {{ comparison.count('gone') }} gone
{% if comparison.x and comparison.cells %}

Pass rate per ODD cell

The buckets each case was drawn in. Each cell shows the passes and runs of every session (baseline first), every run of a repeated case counted; red: a lower pass rate in the latest, green: higher.

{% for bx in comparison.x_labels %}{% endfor %} {% for by in comparison.y_labels %} {% for bx in comparison.x_labels %} {% set cell = comparison.cells.get((bx, by)) %} {% if cell %} {% set d = cell.delta %} {% else %} {% endif %} {% endfor %} {% endfor %}
{{ comparison.y or "" }} \ {{ comparison.x }}{{ bx }}
{{ by }} {% for t in cell.tallies %}
{% if t[1] %}{{ t[0] }}/{{ t[1] }}{% else %}-{% endif %}
{% endfor %}
{% endif %} {% if matrices %}
Coverage matrix of each session
{% for ref, html in matrices %}
{% if html %}{{ html | safe }}{% else %}

No coverage files.

{% endif %}
{% endfor %}
{% endif %}

Cases

{% for ref in comparison.sessions %}{% endfor %} {% for key in comparison.metrics %}{% endfor %} {% for c in comparison.cases %} {% set change = c.change() %} {% for runs in c.runs %} {% set ref = comparison.sessions[loop.index0] %} {% set t = c.tally(loop.index0) %} {% endfor %} {% for key in comparison.metrics %} {% set d = c.delta(key) %} {% set better = metric_names[key][1] %} {% endfor %} {% endfor %}
Change Case{{ ref.split('/', 1)[1] }}Δ {{ metric_names[key][0] }}
{% if change == "regression" %}pass → fail {% elif change == "fixed" %}fail → pass {% elif change == "flaky" %}flaky {% else %}{{ change }}{% endif %}
{{ c.first.label }}
{% for name, bucket in c.first.buckets.items() %}{% if name in (comparison.x, comparison.y) %}{{ name.split('.')[-1] }} {{ bucket }}{% endif %}{% endfor %}
{% for r in runs %}{{ outcome_badge(r) }}{% if not loop.last %} {% endif %}{% else %}{{ outcome_badge(none) }}{% endfor %} {% if t[1] > 1 %}
{{ t[0] }}/{{ t[1] }} passed
{% endif %}
{% if d is none %}-{% else %}{{ "%+.2f" | format(d) }}{% endif %} {# The first run of the first session that ran it, and the last run of the last. #} {% set ns = namespace(q=[]) %} {% for runs in c.runs %}{% if runs %}{% set ref = comparison.sessions[loop.index0] %}{% if not ns.q %}{% set ns.q = ["run=" ~ (ref ~ "/" ~ runs[0].index) | urlencode] %}{% else %}{% set ns.q = [ns.q[0], "run=" ~ (ref ~ "/" ~ runs[-1].index) | urlencode] %}{% endif %}{% endif %}{% endfor %} {% if ns.q | length > 1 %}side by side{% endif %}
{% endif %} {% endblock %} {% block extra_scripts %} {% if oo_script %}{% endif %} {% endblock %}