tooltangle report

{{ report.model }} {{ report.created_at[:16]|replace("T", " ") }} UTC

Accuracy

{{ "%.0f"|format(accuracy) }}%

095% interval {{ "%.0f"|format(low) }}–{{ "%.0f"|format(high) }}%100%

Messages

{{ "{:,}".format(report.evaluated) }}

{% if report.failed %}{{ report.failed }} more failed{% else %}all answered{% endif %}

Tools

seen by the model at once

Definitions

≈ {{ "{:,}".format(report.definition_tokens) }}

tokens on every request

Findings

{{ finding.severity|upper }}
{{ finding.title }}{% if finding.detail %}{{ finding.detail }}{% endif %} {% for example in finding.examples %}
e.g. {{ example }}
{% endfor %}
Nothing to report.

{% if matrix.labels %}

Confusion matrix

Rows are the tool a message was written for, columns the tool the model called.

correct pick fewer {% for step in range(1, 6) %}{% endfor %} more confusions

{% for label in matrix.labels %}{% endfor %} {% for expected in matrix.labels %} {% for picked in matrix.labels %} {% set count = matrix.cells.get(expected ~ "|" ~ picked, 0) %} {% if not count %} {% elif expected == picked %} {% else %} {% endif %} {% endfor %} {% endfor %}

	{{ label }}
{{ expected }}		{{ count }}	{{ count }}

{% endif %}

Tools

Sorted from the least to the most accurate.

{% for key, tool in tools %} {% endfor %}

Tool	Messages	Accuracy	Precision	Description
`{{ key }}`	{{ tool.queries }}	{% if tool.accuracy is not none %} {% if tool.accuracy < 0.8 %}low{% endif %} {{ "%.0f%%"\|format(tool.accuracy * 100) }} {% else %}-{% endif %}	{{ "%.0f%%"\|format(tool.precision * 100) if tool.precision is not none else "-" }}	{{ tool.description\|truncate(160) }}

Messages

only mistakes

{% for row in report.queries %} {% endfor %}

Message	Written for	Model called	Split
{{ row.text }}	`{{ row.tool or "no tool" }}` {% if row.accepted\|length > 1 %} also fine: {{ row.accepted\|reject("equalto", row.tool)\|join(", ") }} {% endif %}	{% if row.error %} request failed {% elif row.correct %} `{{ row.picked\|pick }}` {% else %} `{{ row.picked\|pick }}` {% endif %}	{{ row.split }}