optional · interactive figures

Explore: profiles and the bootstrap

A performance profile whose solvers you can remove, and a bootstrap interval you can watch settle.

optional: nothing in the lab depends on itback to the slides

28 · explore

A performance profile, with solvers switched off

Try: switch Y off and watch X's curve jump at τ = 1; then switch X off instead. Static version: a performance profile, by hand.

What it shows. The slide's three solvers on four instances (times in seconds, "fail" for a timeout). For whichever solvers are switched on, the curves are the reference performance_profile: the share of instances each solver solves within a factor τ of the fastest solver present. The table shows each ratio and the fastest solver per instance, from the reference virtual_best. All seven subsets are recorded by slides/viz/traces/u28.py; highlighted cells differ from the three-solver table.

With all three: X and Y start at 0.5 (they tie on p3), only Y reaches 1.
Without Y: X becomes the fastest on p2, its ratio there drops from 2 to 1, and its curve starts at 0.75. X's times didn't change; its competition did. That is Gould and Scott's trap.
Without X: Y's ratio on p1 becomes $4/3$ (against Z's 3 s), a step that wasn't in any three-solver curve.

28 · explore

A bootstrap interval, resample by resample

Try: drag the slider from 5 resamples to 2 000 and watch the interval settle at [0.68, 0.98]. Static version: a bootstrap interval in numbers.

What it shows. The slide's five per-instance time ratios, 0.6, 0.8, 0.9, 1.1, 0.7. Each resample draws five of them with replacement and takes the mean; the histogram shows the first $n$ resample means and the shaded band the 95% percentile interval. The resamples are drawn exactly as the reference bootstrap_ci draws them (random.Random(1), rng.choices), and each interval is bootstrap_ci's own answer for that number of resamples, checked against the recorded means.

5 resamples: [0.76, 0.84], the slide's five means. 50: [0.70, 0.92]. From about 200 on it stays at [0.68, 0.98], wobbling by one resample at the top.
The interval's width comes from the data (five instances), not from the resample count. To make it narrower, benchmark more instances.
The means can only take a few values here (sums of five of the ratios, divided by 5), so the histogram has gaps: a reminder of how little information five instances carry.