### all hard tasks (20)

| feature | level | tasks | pinned cells failed | failed, answer seen | pinned cells not accepted | tasks failed (most rungs) | ladder escalated | ladder left local | ladder failed |
| -- | -- | -- | -- | -- | -- | -- | -- | -- | -- |
| files touched | 1 file | 12 | 23/60 38% (27%-51%) | 4/41 10% (4%-23%) | 28/60 47% (35%-59%) | 5/12 42% (19%-68%) | 1/12 8% (1%-35%) | 2/12 17% (5%-45%) | 5/12 42% (19%-68%) |
| files touched | 2-3 files | 8 | 24/40 60% (45%-74%) | 9/25 36% (20%-55%) | 26/40 65% (50%-78%) | 6/8 75% (41%-93%) | 0/8 0% (0%-32%) | 0/8 0% (0%-32%) | 5/8 62% (31%-86%) |
| cross-file cause | no | 12 | 31/60 52% (39%-64%) | 8/37 22% (11%-37%) | 35/60 58% (46%-70%) | 7/12 58% (32%-81%) | 1/12 8% (1%-35%) | 2/12 17% (5%-45%) | 6/12 50% (25%-75%) |
| cross-file cause | yes | 8 | 16/40 40% (26%-55%) | 5/29 17% (8%-35%) | 19/40 48% (33%-63%) | 4/8 50% (22%-78%) | 0/8 0% (0%-32%) | 0/8 0% (0%-32%) | 4/8 50% (22%-78%) |
| context size | under 60k chars | 6 | 11/30 37% (22%-54%) | 1/20 5% (1%-24%) | 17/30 57% (39%-73%) | 3/6 50% (19%-81%) | 0/6 0% (0%-39%) | 0/6 0% (0%-39%) | 2/6 33% (10%-70%) |
| context size | 60k-150k chars | 8 | 18/40 45% (31%-60%) | 5/27 19% (8%-37%) | 19/40 48% (33%-63%) | 4/8 50% (22%-78%) | 0/8 0% (0%-32%) | 0/8 0% (0%-32%) | 4/8 50% (22%-78%) |
| context size | over 150k chars | 6 | 18/30 60% (42%-75%) | 7/19 37% (19%-59%) | 18/30 60% (42%-75%) | 4/6 67% (30%-90%) | 1/6 17% (3%-56%) | 2/6 33% (10%-70%) | 4/6 67% (30%-90%) |

Difference, last level minus first, with a Newcombe 95% interval:

| feature | levels | pinned cells failed | failed, answer seen | tasks failed | ladder escalated | ladder left local |
| -- | -- | -- | -- | -- | -- | -- |
| files touched | 2-3 files vs 1 file | +22% (+2% to +39%) | +26% (+6% to +47%) | +33% (-10% to +62%) | -8% (-35% to +25%) | -17% (-45% to +18%) |
| cross-file cause | yes vs no | -12% (-30% to +8%) | -4% (-23% to +16%) | -8% (-45% to +30%) | -8% (-35% to +25%) | -17% (-45% to +18%) |
| context size | over 150k chars vs under 60k chars | +23% (-2% to +45%) | +32% (+6% to +54%) | +17% (-32% to +56%) | +17% (-25% to +56%) | +33% (-12% to +70%) |

### code_generation (10)

| feature | level | tasks | pinned cells failed | failed, answer seen | pinned cells not accepted | tasks failed (most rungs) | ladder escalated | ladder left local | ladder failed |
| -- | -- | -- | -- | -- | -- | -- | -- | -- | -- |
| files touched | 1 file | 6 | 21/30 70% (52%-83%) | 4/13 31% (13%-58%) | 26/30 87% (70%-95%) | 5/6 83% (44%-97%) | 0/6 0% (0%-39%) | 1/6 17% (3%-56%) | 5/6 83% (44%-97%) |
| files touched | 2-3 files | 4 | 17/20 85% (64%-95%) | 4/7 57% (25%-84%) | 19/20 95% (76%-99%) | 4/4 100% (51%-100%) | 0/4 0% (0%-49%) | 0/4 0% (0%-49%) | 3/4 75% (30%-95%) |
| cross-file cause | no | 6 | 23/30 77% (59%-88%) | 3/10 30% (11%-60%) | 27/30 90% (74%-97%) | 5/6 83% (44%-97%) | 0/6 0% (0%-39%) | 1/6 17% (3%-56%) | 4/6 67% (30%-90%) |
| cross-file cause | yes | 4 | 15/20 75% (53%-89%) | 5/10 50% (24%-76%) | 18/20 90% (70%-97%) | 4/4 100% (51%-100%) | 0/4 0% (0%-49%) | 0/4 0% (0%-49%) | 4/4 100% (51%-100%) |
| context size | under 60k chars | 4 | 11/20 55% (34%-74%) | 1/10 10% (2%-40%) | 17/20 85% (64%-95%) | 3/4 75% (30%-95%) | 0/4 0% (0%-49%) | 0/4 0% (0%-49%) | 2/4 50% (15%-85%) |
| context size | 60k-150k chars | 3 | 14/15 93% (70%-99%) | 3/4 75% (30%-95%) | 15/15 100% (80%-100%) | 3/3 100% (44%-100%) | 0/3 0% (0%-56%) | 0/3 0% (0%-56%) | 3/3 100% (44%-100%) |
| context size | over 150k chars | 3 | 13/15 87% (62%-96%) | 4/6 67% (30%-90%) | 13/15 87% (62%-96%) | 3/3 100% (44%-100%) | 0/3 0% (0%-56%) | 1/3 33% (6%-79%) | 3/3 100% (44%-100%) |

Difference, last level minus first, with a Newcombe 95% interval:

| feature | levels | pinned cells failed | failed, answer seen | tasks failed | ladder escalated | ladder left local |
| -- | -- | -- | -- | -- | -- | -- |
| files touched | 2-3 files vs 1 file | +15% (-10% to +35%) | +26% (-15% to +59%) | +17% (-34% to +56%) | +0% (-39% to +49%) | -17% (-56% to +34%) |
| cross-file cause | yes vs no | -2% (-26% to +21%) | +20% (-20% to +53%) | +17% (-34% to +56%) | +0% (-39% to +49%) | -17% (-56% to +34%) |
| context size | over 150k chars vs under 60k chars | +32% (+1% to +55%) | +57% (+9% to +82%) | +25% (-35% to +70%) | +0% (-49% to +56%) | +33% (-23% to +79%) |

### review (10)

| feature | level | tasks | pinned cells failed | failed, answer seen | pinned cells not accepted | tasks failed (most rungs) | ladder escalated | ladder left local | ladder failed |
| -- | -- | -- | -- | -- | -- | -- | -- | -- | -- |
| files touched | 1 file | 6 | 2/30 7% (2%-21%) | 0/28 0% (0%-12%) | 2/30 7% (2%-21%) | 0/6 0% (0%-39%) | 1/6 17% (3%-56%) | 1/6 17% (3%-56%) | 0/6 0% (0%-39%) |
| files touched | 2-3 files | 4 | 7/20 35% (18%-57%) | 5/18 28% (12%-51%) | 7/20 35% (18%-57%) | 2/4 50% (15%-85%) | 0/4 0% (0%-49%) | 0/4 0% (0%-49%) | 2/4 50% (15%-85%) |
| cross-file cause | no | 6 | 8/30 27% (14%-44%) | 5/27 19% (8%-37%) | 8/30 27% (14%-44%) | 2/6 33% (10%-70%) | 1/6 17% (3%-56%) | 1/6 17% (3%-56%) | 2/6 33% (10%-70%) |
| cross-file cause | yes | 4 | 1/20 5% (1%-24%) | 0/19 0% (0%-17%) | 1/20 5% (1%-24%) | 0/4 0% (0%-49%) | 0/4 0% (0%-49%) | 0/4 0% (0%-49%) | 0/4 0% (0%-49%) |
| context size | under 60k chars | 2 | 0/10 0% (0%-28%) | 0/10 0% (0%-28%) | 0/10 0% (0%-28%) | 0/2 0% (0%-66%) | 0/2 0% (0%-66%) | 0/2 0% (0%-66%) | 0/2 0% (0%-66%) |
| context size | 60k-150k chars | 5 | 4/25 16% (6%-35%) | 2/23 9% (2%-27%) | 4/25 16% (6%-35%) | 1/5 20% (4%-62%) | 0/5 0% (0%-43%) | 0/5 0% (0%-43%) | 1/5 20% (4%-62%) |
| context size | over 150k chars | 3 | 5/15 33% (15%-58%) | 3/13 23% (8%-50%) | 5/15 33% (15%-58%) | 1/3 33% (6%-79%) | 1/3 33% (6%-79%) | 1/3 33% (6%-79%) | 1/3 33% (6%-79%) |

Difference, last level minus first, with a Newcombe 95% interval:

| feature | levels | pinned cells failed | failed, answer seen | tasks failed | ladder escalated | ladder left local |
| -- | -- | -- | -- | -- | -- | -- |
| files touched | 2-3 files vs 1 file | +28% (+6% to +51%) | +28% (+8% to +51%) | +50% (-2% to +85%) | -17% (-56% to +34%) | -17% (-56% to +34%) |
| cross-file cause | yes vs no | -22% (-40% to +1%) | -19% (-37% to +1%) | -33% (-70% to +21%) | -17% (-56% to +34%) | -17% (-56% to +34%) |
| context size | over 150k chars vs under 60k chars | +33% (+0% to +58%) | +23% (-8% to +50%) | +33% (-38% to +79%) | +33% (-38% to +79%) | +33% (-38% to +79%) |
