Data set comparisons

Because this report contains multiple datasets, you may find it useful or interesting to compare these datasets directly. This tab contains some tools for doing so. Generally speaking, they use the same statistical tests and metrics that are used in the "Model Violation" tab to quantify how consistent each pair of datasets are with each other. If these datasets aren't even intended to be comparable, then this tab is pretty pointless. But in many cases, it's useful to ask "Which pairs of datasets are similar?" or "Which circuits differ most dramatically from one dataset to the next?"

{{ dataset_comparison_summary|render }}
Pairwise comparisons of all datasets This chart compares each pair of datasets. It quantifies how consistent they are -- that is, how plausible it is that both datasets were actually generated by the same underlying probabilities. For each pair, two likelihoods are computed: (1) the likelihood of a model in which each circuit has a fixed probability and each dataset is interpreted as a sample from it; and (2) the likelihood of a model in which each circuit has an independent probability for each dataset. If the datasets really are consistent, then the log of their ratio is a \chi^2 random variable with known mean and variance. This plot shows the number of standard deviations by which the observed statistic exceeds its expected mean. Values around 1 or less indicate consistency, while large positive values indicate that the data sets probably weren't generated by the same process.
{{ dataset_comparison_histogram|render }} {{ dscmpSwitchboard|render }}
Histogram of per-circuit 2\Delta\log(\mathcal{L}) values between two data sets. Given two datasets to compare, the frequencies observed each individual circuit can be evaluated for consistency. The result defines a loglikelihood ratio (2\Delta\log(\mathcal{L})) value, which indicates the amount the single-generating-gateset model is violated. The line shows what would be expected for perfectly consistent data.
{{ dataset_comparison_box_plot|render }}
Per-circuit 2\Delta\log(\mathcal{L}) values comparing two data sets. This plot shows the inconsistency between two datasets for every individual GST circuit. Each colored box (mouse over to read off the value) represents a single circuit, and they are arranged in exactly the same fashion as in the model violation analysis on another tab. The color in the box represents the loglikelihood ratio (2\Delta\log(\mathcal{L})) for that circuit, between a model that assumes both datasets are generated by the same underlying probabilities, and one that assigns independent probabilities to each dataset. Darker grays indicate less consistency; green indicates inconsistency detected at the 0.95 family-wise confidence level (so 95 times out of 100, perfectly consistent data should yield no green boxes anywhere in the grid).