ModelDiff-ML Report

Independent discovery and validation · schema 1.0

Samples

5,000

Task

binary classification

Primary metric

log_loss

Overall result

improvement

Overview

Metric	Baseline	Candidate
accuracy	0.6344	0.7082
log_loss	0.6429	0.6341
brier	0.2247	0.2147
roc_auc	0.6864	0.7338

Primary delta (candidate loss − baseline loss): -0.0088. Negative means improvement; positive means regression.

Prediction behavior

Mean Absolute Probability Shift	0.1187
Median Absolute Probability Shift	0.0800
P95 Absolute Probability Shift	0.5962
Hard Classification Disagreement Rate	0.2038
Mean Candidate Confidence Change	0.0722

Validated regressions

Region	Validation support	Baseline loss	Candidate loss	Delta	CI	p	q	Significant
x1 > 1.25246 AND x2 <= -0.516008	75 (3.0%)	0.4837	2.1203	+1.6366	[+1.4696, +1.7910]	0.0004998	0.0004998	yes
x1 > 1.25246 AND x2 <= -0.239355	103 (4.1%)	0.5003	2.1203	+1.6200	[+1.4957, +1.7420]	0.0004998	0.0004998	yes
x1 > 1.25246 AND x2 <= -0.0134457	131 (5.2%)	0.5356	2.1203	+1.5846	[+1.4673, +1.6919]	0.0004998	0.0004998	yes
x1 > 0.774474 AND x2 <= -0.516008	159 (6.4%)	0.5412	1.6676	+1.1264	[+0.9722, +1.2692]	0.0004998	0.0004998	yes
x1 > 0.774474 AND x2 <= -0.0134457	276 (11.0%)	0.5925	1.6862	+1.0936	[+0.9815, +1.1954]	0.0004998	0.0004998	yes
x1 > 1.25246	247 (9.9%)	0.5826	1.3722	+0.7896	[+0.6760, +0.9141]	0.0004998	0.0004998	yes
x1 > 0.471171 AND x2 <= -0.239355	334 (13.4%)	0.5881	1.2711	+0.6830	[+0.5768, +0.7848]	0.0004998	0.0004998	yes
x1 > 0.471171 AND x2 <= -0.0134457	411 (16.4%)	0.6094	1.2918	+0.6824	[+0.5941, +0.7735]	0.0004998	0.0004998	yes
x1 > 0.774474	536 (21.4%)	0.6207	1.1202	+0.4994	[+0.4165, +0.5696]	0.0004998	0.0004998	yes
x1 > 0.235402 AND x2 <= -0.0134457	544 (21.8%)	0.6199	1.0989	+0.4789	[+0.4040, +0.5493]	0.0004998	0.0004998	yes

Validated improvements

Region	Validation support	Baseline loss	Candidate loss	Delta	CI	p	q	Significant
x1 <= -0.0268108 AND x2 <= -0.0134457	597 (23.9%)	0.6823	0.5288	-0.1535	[-0.1597, -0.1487]	0.0004998	0.0004998	yes
x1 <= -0.270907	976 (39.0%)	0.6130	0.4650	-0.1480	[-0.1532, -0.1428]	0.0004998	0.0004998	yes
x1 <= -0.0268108	1214 (48.6%)	0.6216	0.4738	-0.1478	[-0.1522, -0.1436]	0.0004998	0.0004998	yes
x1 <= -0.539	742 (29.7%)	0.5896	0.4433	-0.1462	[-0.1533, -0.1400]	0.0004998	0.0004998	yes
x2 > 0.245825	974 (39.0%)	0.6067	0.4615	-0.1452	[-0.1498, -0.1406]	0.0004998	0.0004998	yes
x2 > 0.545369	735 (29.4%)	0.5925	0.4483	-0.1443	[-0.1503, -0.1388]	0.0004998	0.0004998	yes
x1 <= -0.848861	511 (20.4%)	0.5498	0.4069	-0.1429	[-0.1522, -0.1342]	0.0004998	0.0004998	yes
x1 <= -0.0268108 AND x2 > -0.0134457	617 (24.7%)	0.5629	0.4206	-0.1422	[-0.1494, -0.1360]	0.0004998	0.0004998	yes
x2 > -0.0134457	1236 (49.4%)	0.6157	0.4737	-0.1421	[-0.1484, -0.1345]	0.0004998	0.0004998	yes
x2 > -0.0134457 AND x1 > -0.0268108	619 (24.8%)	0.6684	0.5265	-0.1419	[-0.1520, -0.1297]	0.0004998	0.0004998	yes

Warnings

None

Methodology and configuration

Regions were selected only on the discovery partition, then evaluated unchanged on an independent validation partition. Confidence intervals use percentile bootstrap resampling; p-values use paired sign flips; q-values use Benjamini–Hochberg correction.

discovery_fraction	0.5
max_depth	3
beam_width	25
max_regions	10
min_region_size	None
min_region_fraction	0.02
max_region_fraction	0.5
quantiles	[0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9]
max_categories	20
jaccard_threshold	0.8
n_bootstrap	1000
n_permutations	2000
confidence_level	0.95
significance_level	0.05
random_state	42
threshold	0.5
epsilon	1e-15