{% extends "report_base.html" %} {# The report of the credibility assessment of a model, see `sbmlsim.credibility.report.render_html`. It extends the base of the reports of sbmlsim and fills its blocks. Sections: the overview, one card per context of use, the limitations, the unclaimed validation data, the verification, the cross-validation, the sections of the evidence extensions and the provenance. The blocks of an extension in the card of a context follow the block of the core they name (`view.blocks`, see `sbmlsim.credibility.extensions`). A verdict, a coverage and the result of a rule are words with a colour class (`verdict`, `state`), never only a colour. Nothing loads from the network. #} {% macro verdict(v) %}{{ v.label }}{% endmacro %} {% macro limitation_context(e) %}{% if e.context %}{{ e.context }}{% else %}the model{% endif %}{% endmacro %} {% macro level(l) %}{{ l.value }} {{ l.label }}{% endmacro %} {% block title %}Credibility of {{ provenance.title }}{% endblock %} {% block items %}contexts of use{% endblock %} {% block heading %}Credibility{% endblock %} {% block subtitle %}{{ provenance.title }} {{ provenance.model }}{% endblock %} {% block placeholder %}Search contexts of use, evidence, studies …{% endblock %} {% block badges %}contexts of use {{ contexts | length }}{% for v in verdicts %}{{ v.label }} {{ verdict_counts[v] }}{% endfor %}{% endblock %} {% block head %} {% endblock %} {% block nav %} Overview Contexts of use Limitations Unclaimed data Verification {% if assessment.cross_validations %}Cross-validation{% endif %} {% for section in extension_sections %}{{ section.title }} {% endfor %} Provenance {% endblock %} {% block chips %} {% for v in verdicts %} {% endfor %} {% endblock %} {% block footer %}created by sbmlsim {{ provenance.versions.get("sbmlsim", "") }} on {{ provenance.date }}{% endblock %} {% block content %}

Overview

The contexts of use of the model, their model risk from the influence of the model on the decision and the consequence of a wrong decision, and their verdict on the decisive evidence (decided on): the calibration data, the data the model was fitted to, for a description (C1), else the validation data. The evaluable points are the decisive points which the fold criteria can evaluate; a verdict needs at least the minimum of them, by default {% for risk, n in default_min_points %}{{ n }} for a {{ risk }} risk{% if loop.revindex > 2 %}, {% elif loop.revindex == 2 %} and {% endif %}{% endfor %}.

{% for view in contexts %} {% set c = view.result %} {% endfor %}
context of useverdict decided on title levelrisk decisive points evaluable points minimum points
{{ c.id }} {{ verdict(c.verdict) }} {{ c.decisive_role.value }} {{ c.title }} {{ level(c.level) }} {{ c.risk.value }} {{ c.n_decisive }} {{ c.n_evaluable }} {{ c.minimum_points if c.minimum_points else "" }}

Limitations of the model

{% if model_limitations %} {% else %}

The model declares no limitations.

{% endif %}
{% if assessment.notes %}

Notes

{{ assessment.notes }}

{% endif %}

Contexts of use

{% for view in contexts %} {% set c = view.result %}
{{ c.id }} {{ c.title }} {{ verdict(c.verdict) }}

{{ c.question }}

level
{{ level(c.level) }}
influence
{{ c.influence.value }}
consequence
{{ c.consequence.value }}
model risk
{{ c.risk.value }}
decided on
{{ c.decisive_role.value }} evidence
decisive points
{{ c.n_decisive }}
evaluable points
{{ c.n_evaluable }}{% if c.minimum_points %} (minimum {{ c.minimum_points }}){% if c.n_evaluable < c.minimum_points %} below the minimum{% endif %}{% endif %}
{% if c.rationale %}
rationale
{{ c.rationale }}
{% endif %}

Quantities of interest

A quantity is covered by the evaluable decisive points of its observable inside the claimed domain, a ratio with a pairing by the points its evidence extension derives from them.

{% for q in c.quantities %} {% endfor %}
quantitydescriptionkindcoveragepoints
{{ q.id }}{{ q.description }} {% if q.ratio and q.coverage %}ratio{% elif q.ratio %}ratio, no pairing declared{% else %}absolute{% endif %} {% if q.coverage %}{{ q.coverage }}{% else %}n/a{% endif %} {% if q.coverage %}{{ q.n_points }}{% else %}n/a{% endif %}

Claimed domain

{% if c.domain_statement %}

{{ c.domain_statement }} statement, not assessed

{% endif %} {% if view.coverage %}
{% for e in view.coverage %} {% endfor %}
fieldvalue coveragepoints
{{ e.field }}{{ e.value }} {% if e.coverage is none %}outside the claim{% else %}{{ e.coverage.value }}{% endif %} {{ e.n_points }}
{% elif not c.domain_statement %}

The domain claims no values of the metadata.

{% endif %}

Evidence

{% if view.evidence %}
{% for e in view.evidence %} {% endfor %}
evidencerole selection pointsmappings studies outliers dropped limitation subtracted
{{ e.name }}{% if e.description %}
{{ e.description }}{% endif %}
{{ e.role.value }} {{ e.selection }} {{ e.n_points }}{{ e.n_mappings }} {{ e.n_studies }} {{ e.outliers_dropped }} {{ e.limitation_subtracted }}

Metrics of the evidence

The fold errors (GMFE, AFE) and the percentage of the points within a fold or a number of standard deviations of the data, n/a without an evaluable point.

{% for name in view.metrics %}{% endfor %} {% for e in view.evidence %} {% for name in view.metrics %}{% endfor %} {% endfor %}
evidence{{ name }}
{{ e.name }}{{ e.metrics[name] | number if name in e.metrics else "" }}
{% else %}

No evidence.

{% endif %}

Acceptance

{% if c.acceptance %}
{% for a in c.acceptance %} {% endfor %}
rulevaluelimitresult
{{ a.name }}{% if a.extension %} {{ extension_labels.get(a.extension, a.extension) }}{% endif %} {{ a.value | number }} {{ a.limit }} {% if a.met is none %}not evaluable{% elif a.met %}met{% else %}not met{% endif %}
{% else %}

No acceptance rules.

{% endif %} {{ view.blocks.acceptance }} {% if c.uncertainty is not none %}{% set u = c.uncertainty %} {% set states = {"accepted": "met", "rejected": "not_met", "equivalent": "not_evaluable", "undecided": "not_evaluable", "not_evaluable": "outside"} %}

Parameter uncertainty

The adopted fits evaluated at {{ u.n_draws }} draws of their fitted parameters (the profile or Fisher uncertainty of their records{% if u.fits | selectattr("correlated") | list %}, with the correlation of the Fisher information{% endif %}). A rule is robust when it is met with a probability of at least {{ u.threshold }} over the draws, decided at the level {{ u.level }} for the rules of the context together; the verdict is the acceptance at the adopted parameters and does not change.

{% if u.rules %}
{% for r in u.rules %} {% endfor %}
rulelimitP(met)sequencedrawsdecision
{{ r.name }} {{ r.limit }} {{ r.estimate | number }} {% if r.low is not none %}{{ r.low | number }} to {{ r.high | number }}{% endif %} {{ r.n }}{% if r.n_failed %} +{{ r.n_failed }} failed{% endif %} {{ r.decision | replace("_", " ") }}{% if r.n_needed %} about {{ r.n_needed }} draws{% endif %}
{% endif %}

The {{ ((u.quantiles[1] - u.quantiles[0]) * 100) | round | int }} % bands of the predictions hold {{ "n/a" if u.coverage is none else ((u.coverage * 100) | round | int) ~ " %" }} of the {{ u.n_points }} evaluable decisive observations; a band of the parameter uncertainty holds the prediction of the mean, the variability between the subjects and the error of the measurements lie outside it.

{% if u.quantities %}
{% for q in u.quantities %} {% endfor %}
quantitypointsband foldcoverage %
{{ q.quantity }}{{ q.n_points }}{{ q.band_fold | number }}{{ "n/a" if q.coverage is none else (q.coverage * 100) | round | int }}
{% endif %} {% if u.sensitivity %}

The parameters which carry the variance of a quantity (the mean log ratio of its predictions to the adopted ones) and of the value of a rule over the draws: the first-order indices, estimated from the draws without further simulations (Plischke 2013), up to three of at least 0.05 per target; the index of a parameter includes the effect of the parameters it is correlated with.

{% for target, group in u.sensitivity | groupby("target") %} {% set leading = group | rejectattr("index", "none") | selectattr("index", "ge", 0.05) | sort(attribute="index", reverse=True) | list %} {% endfor %}
targetleading parameters
{{ target }}{% if group[0].kind == "rule" %} rule{% endif %} {% for i in leading[:3] %}{{ i.fit }}.{{ i.parameter }} {{ i.index | number }}{% if not loop.last %}, {% endif %}{% else %}none of at least 0.05{% endfor %}
{% endif %} {% for f in u.fits %}

{{ f.fit }}: {{ f.n_parameters }} {{ "parameter" if f.n_parameters == 1 else "parameters" }}, {{ f.note }}{% if f.n_failed %}, {{ f.n_failed }} failed draws{% endif %}.

{% endfor %} {% endif %} {{ view.blocks.uncertainty }} {% if c.variability is not none %}{% set v = c.variability %} {% set agreements = {"consistent": "met", "narrow": "not_met", "wide": "not_evaluable"} %}

Variability

The visual predictive check of the decisive points with the variability of the subjects the model declares ({% for t, d in v.targets.items() %}{{ t }} {{ d }}{% if not loop.last %}, {% endif %}{% endfor %}){% if v.between %} and between the studies ({% for t, d in v.between.items() %}{{ t }} {{ d }}{% if not loop.last %}, {% endif %}{% endfor %}){% endif %}: {% if v.between %}{{ v.studies }} studies of {% endif %}{{ v.subjects }} subjects {% if v.vectors > 1 %}nested in {{ v.vectors }} draws of the parameters{% else %}at the adopted parameters{% endif %}, the subjects of every study group varying around its inputs. A mean of a group of N subjects is compared with the means of {{ v.groups }} virtual groups of N subjects{% if v.between %} of a study{% endif %}, an individual with the subjects; the verdict does not change.

{% if v.coverage is not none %}
pointsmeans of groupscoverage %interval %expected %agreement
{{ v.n_points }} {{ v.n_groups }} {{ (v.coverage * 100) | round | int }} {{ (v.low * 100) | round | int }} to {{ (v.high * 100) | round | int }} {{ (v.expected * 100) | round | int }} {{ v.agreement }}{% if v.failed %} {{ v.failed }} failed subjects{% endif %}

The interval is the Wilson interval of the coverage at the level {{ v.level }}. Narrow: the bands hold fewer observations than expected, the declared variability with the uncertainty of the parameters does not explain the scatter of the data, or the typical predictions are biased; wide: they hold more, the variability is larger than the one of the data.

{% if v.quantities %}
{% for q in v.quantities %} {% endfor %}
quantitypointscoverage %
{{ q.quantity }}{{ q.n_points }}{{ "n/a" if q.coverage is none else (q.coverage * 100) | round | int }}
{% endif %} {% else %}

No evaluable decisive point has a band.

{% endif %} {% endif %} {{ view.blocks.variability }}

Verification

{% if c.verification %}
{% set battery = c.verification | rejectattr("check", "in", prediction_checks) | list %} {% for r in c.verification if r.check in prediction_checks %} {% endfor %} {% if battery %} {% set failed = battery | rejectattr("passed") | list %} {% endif %}
checkfactor pointslargest difference atacceptanceresult
{{ check_labels[r.check] }}{% for problem in r.problems %}
{{ problem }}{% endfor %}
{{ r.factor }} {{ r.n_points }} {{ r.max_difference | number if r.acceptance_changed is not none else "" }} {{ r.at }} {% if r.acceptance_changed is none %}{% elif r.acceptance_changed %}changed{% else %}unchanged{% endif %} {{ "passed" if r.passed else "failed" }}
battery of the variants, {{ battery | length }} checks{% for r in failed %}
{{ check_labels[r.check] }} failed{% endfor %}
{{ battery | map(attribute="factor") | unique | join(", ") }} {{ "failed" if failed else "passed" }}
{% else %}

The assessment was not verified.

{% endif %} {{ view.blocks.verification }}

Factors of V&V 40

{% for f in c.factors %} {% endfor %}
statusfactor
{{ f.status | replace("_", " ") }}{% if f.step is not none %}
in step {{ f.step }}{% endif %}
{{ f.name }}
{{ f.statement }}
{{ view.blocks.factors }}

Extrapolation

{% if c.extrapolation %}

{{ c.extrapolation }}

{% else %}

No extrapolation is argued.

{% endif %}
{% endfor %}

Limitations

The data the model does not describe, of the model and of a context of use. Their metrics are reported for information and are outside every acceptance; their points are subtracted from the evidence.

{% if assessment.limitations %}
{% for e in assessment.limitations %} {% endfor %}
context of uselimitation selection pointsmappings studies
{{ limitation_context(e) }} {{ e.name }}{% if e.description %}
{{ e.description }}{% endif %}
{{ e.selection }} {{ e.n_points }}{{ e.n_mappings }} {{ e.n_studies }}

Metrics of the limitations

{% for name in limitation_metrics %}{% endfor %} {% for e in assessment.limitations %} {% for name in limitation_metrics %}{% endfor %} {% endfor %}
context of uselimitation{{ name }}
{{ limitation_context(e) }} {{ e.name }}{{ e.metrics[name] | number if name in e.metrics else "" }}
{% else %}

No limitations are declared.

{% endif %}

Unclaimed validation data

The validation points of the adopted fits which no evidence and no limitation selects.

{% if assessment.unclaimed %} {% for u in assessment.unclaimed %} {% endfor %}
fitstudy mappingpoints
{{ u.fit }}{{ u.experiment }}{{ u.mapping }}{{ u.n }}
{% else %}

Every validation point is claimed.

{% endif %}

Verification

{% if verification %}

Every adopted fit is evaluated again at its reference parameters with the tolerances divided by {{ verification.tightening | number }} (discretization error) and with the BDF of CVODE of at most order 2 instead of 5 at the tightened tolerances (numerical solver error). A check of a context of use passes when the largest relative difference of its decisive predictions is at most {{ verification.limit | number }} and its acceptance does not change; a difference below {{ verification.scale_floor | number }} of the largest prediction of the mapping is none. The setup (use error) passes when the variants are valid SBML with consistent units and every decisive prediction is finite.

{% for f in verification.fits %} {% endfor %}
fitreferencetolerancessecond integration method
{{ f.fit }}{{ f.reference }}{{ f.tolerances }}{{ f.integrator }}

Variants

{% if verification.problems %} {% else %}

Every variant is valid SBML with consistent units.

{% endif %}

Battery of the variants

{% if verification.battery %}

The variants of the model are checked without data, the same for every context of use: {{ verification.battery.settings }} A check which is not applicable assesses nothing.

{% for r in verification.battery.results %} {% endfor %}
checkfactorcaseslargest deviationatresult
{{ r.check.label }}
{{ r.statement }}{% for problem in r.problems %}
{{ problem }}{% endfor %}
{{ r.check.factor }} {{ r.cases if r.passed is not none else "" }} {{ r.largest | number if r.passed is not none and r.largest == r.largest else "" }} {{ r.at }} {{ r.status.value | replace("_", " ") }}
{{ battery_html }} {% else %}

The battery of the variants did not run.

{% endif %} {% else %}

The assessment was not verified: the factors of the verification are not assessed.

{% endif %}
{% if assessment.cross_validations %}

Cross-validation

Every adopted fit is refitted once per training study without the study, from its adopted parameters, and predicts the held-out study. A GMFE of the held-out predictions close to the one of the fitted predictions of the same points is a model form which carries across studies.

{% for cv in assessment.cross_validations %}

{{ cv.subset }} {{ cv.fit }}

{{ cv.statement() }}

{% for s in cv.studies %} {% endfor %}
studypointsGMFE held outGMFE fittedwithin 2-fold held outwithin 2-fold fitted
{{ s.study }}{{ s.n_points }}{{ s.gmfe_held_out | number }}{{ s.gmfe_fitted | number }}{{ s.within_2fold_held_out | number }}{{ s.within_2fold_fitted | number }}
{% endfor %}
{% endif %} {% for section in extension_sections %}

{{ section.title }}

{{ section.html }}
{% endfor %}

Provenance

model
{{ provenance.model }}
title
{{ provenance.title }}
adopted fits
{{ provenance.fits | join("\n") if provenance.fits else "none" }}
{% for package, version in provenance.versions.items() %}
{{ package }}
{{ version }}
{% endfor %}
commit
{% if provenance.commit %}{{ provenance.commit }}{% if provenance.dirty %} with uncommitted changes{% endif %}{% else %}outside a repository{% endif %}
date
{{ provenance.date }}
lock
{% if provenance.lock == "matched" %}the evidence matches the lock of the declaration{% elif provenance.lock == "written" %}the lock of the evidence was written with this assessment{% else %}the declaration has no lock{% endif %}
{% endblock %}