{% set paraphrase_str = 'Paraphrase' %}

Robustness report

Description of the Test Suite:

{{ suite_description }}

Summarized results

{% for summary in summaries %} {% endfor %}
Perturbation Type No. of perturbations Expected behavior Robust Accuracy
{{ summary.perturbation_type }} {{ summary.total_perturbations }} {{ summary.description }} {{ "%.4f"|format(summary.robust_accuracy) }}

Detailed results

Following section contains detailed results for the perturbation that were applied to the model.

{% for summary in summaries %}

Perturbation type: {{ summary.perturbation_type }}

Description: {{ summary.description }}

Failed Perturbations

{% if paraphrase_str in summary.perturbation_type %} {% else %} {% endif %} {% for pset in summary.results %} {% for r in pset %} {% if r.result <= 0 %} {% endif %} {% endfor %} {% endfor %}
Perturbed InputPerturbation - Reference cosine similarityPerturbationOriginal Input Result
{{ r.input }} {% if r.metadata is not none %} {% if paraphrase_str in summary.perturbation_type %} {{ r.metadata[0] | round(2) }} {% else %} {{ r.metadata[0] }} → {{ r.metadata[1] }} {% endif %} {% else %} unperturbed-data {% endif %} {{ r.original_input }} {% if r.result > 0 %} ✅ {% else %} ❌ {% endif %}

Passed Perturbations

{% if paraphrase_str in summary.perturbation_type %} {% else %} {% endif %} {% for pset in summary.results %} {% for r in pset %} {% if r.result > 0 %} {% endif %} {% endfor %} {% endfor %}
Perturbed InputPerturbation - Reference cosine similarityPerturbationOriginal Input Result
{{ r.input }} {% if r.metadata is not none %} {% if paraphrase_str in summary.perturbation_type %} {{ r.metadata[0] | round(2) }} {% else %} {{ r.metadata[0] }} → {{ r.metadata[1] }} {% endif %} {% else %} unperturbed-data {% endif %} {{ r.original_input }} {% if r.result > 0 %} ✅ {% else %} ❌ {% endif %}
{% endfor %}