Part VI · making it real · spine
Summaries that don't lie, pictures that answer the right question, intervals and tests for claims about randomised and chaotic solvers, and an audit of this course's own earlier conclusions.
needs 27~1½ h slides + 2½ h lablab: uv run co lab 28
28 · overview
Why the arithmetic mean of times misleads; shifted geometric means; timeouts and penalties.
Performance profiles, time-to-target plots, the virtual best solver.
Seeds, paired bootstrap intervals, Wilcoxon, Holm; instance selection; negative results.
Unit 07's branching claim survives, quantified. Unit 19's median claim doesn't.
Words with a dotted underline are glossary terms: click one for its definition. G lists every term in the unit, N shows the notes. Optional extra: interactive figures, linked from the slides they support by a small ▸ chip; skip them freely.
section 1
One number per solver, chosen with care.
28 · summaries
Two solvers, ten instances:
| 9 instances | 1 instance | arith. mean | geo. mean | SGM, shift 10 | |
|---|---|---|---|---|---|
| A | 1 s each | 1 000 s | 100.9 | 2.0 | 7.3 |
| B | 2 s each | 100 s | 11.8 | 3.0 | 5.0 |
The arithmetic mean is one instance's number. The geometric mean treats 1 s → 2 s like 500 s → 1 000 s: ratios, which is what comparisons are about. The shift keeps trivial instances from dominating.
PAR-2 on the example, with a 300 s limit: A's 1 000 s run becomes 600 s, and A's arithmetic mean drops from 100.9 to 60.9. The limit and $k$ moved the number, not the solver.
$$\mathrm{SGM}_s(t) = \exp\Big(\tfrac1n \textstyle\sum_i \log(t_i + s)\Big) - s$$An unsolved run has no time, only a lower bound. PAR-k counts it as $k \times$ the limit. The result depends on the limit and on $k$, so report both, and report the solved count separately.
Unit 19 reported a mean of 153.6 nodes with a 3 000-node cap. Three capped runs contributed 90 of it: without them the mean is 65.6. A higher cap would give a higher mean. The number described the cap as much as the solver.
SCIP and most MIP benchmark reports use shift 10 s (Achterberg 2007). For sub-second runs, pick a shift on their scale: the lab's audit uses 0.1 s.
section 2
Distributions, not averages.
28 · pictures
For problem $p$ and solver $s$, the performance ratio $r_{p,s} = t_{p,s} / \min_{s'} t_{p,s'}$, or $\infty$ if $s$ failed. The performance profile:
$$\rho_s(\tau) = \tfrac{1}{|P|}\,\big|\{p : r_{p,s} \le \tau\}\big|$$Ratios are to the best solver in the comparison. Adding or removing a solver can reorder the others' curves (Gould & Scott 2016). Compare two at a time when the question is two-way.
Lab then, unit 07's branching rules on 24 knapsacks, time:
| τ | 1 | 1.25 | 1.5 | 2 | 3 | 5 |
|---|---|---|---|---|---|---|
| most-fractional | 0.08 | 0.75 | 1.00 | 1.00 | 1.00 | 1.00 |
| pseudocost | 0.92 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| strong | 0.00 | 0.00 | 0.00 | 0.04 | 0.92 | 1.00 |
Per problem, the fastest solver's time. It's the ceiling for any portfolio of these solvers, and the gap to it says how much choosing per instance could buy. Here pseudocost is the virtual best on 22 of 24.
28 · pictures · example
Three solvers, four instances, seconds ("—" = failed):
| instance | X | Y | Z | best |
|---|---|---|---|---|
| p1 | 2 | 4 | 3 | X |
| p2 | 10 | 5 | — | Y |
| p3 | 1 | 1 | 2 | X, Y |
| p4 | — | 8 | 4 | Z |
Ratios (time over the instance's best): X $1, 2, 1, \infty$; Y $2, 1, 1, 2$; Z $1.5, \infty, 2, 1$.
The virtual best takes 2, 5, 1, 4: every instance solved.
| τ | 1 | 1.5 | 2 | large |
|---|---|---|---|---|
| X | 0.50 | 0.50 | 0.75 | 0.75 |
| Y | 0.50 | 0.50 | 1.00 | 1.00 |
| Z | 0.25 | 0.50 | 0.75 | 0.75 |
X and Y tie as winners at τ = 1 (the tie on p3 counts for both). Only Y reaches 1: it solves everything, never worse than 2× the best.
Drop Y and recompute: X's ratio on p2 becomes 1 and Z's on p4 stays 1, so X's curve starts at 0.75. Removing a solver changed the others' curves: the trap on the previous slide.
section 3
How sure, and about what?
28 · inference
Paired, rank-based: do the differences tend to be positive? Its p-value answers "is there a difference", and the bootstrap interval answers "how big" (the effect size). Report both.
Three solvers mean three pairwise tests. At α = 0.05 each, the chance of at least one false positive is about 14% if nothing differs. Holm: sort the p-values, multiply the $k$-th smallest by $m - k + 1$, and keep them monotone. Uniformly more powerful than Bonferroni.
Holm (1979), Scand. J. Statistics 6.
28 · inference · example
Per-instance time ratios, new over old: $0.6, 0.8, 0.9, 1.1, 0.7$. Mean 0.82. Draw 5 values with replacement, five times:
| resample | mean |
|---|---|
| 0.6, 0.7, 1.1, 0.8, 0.9 | 0.82 |
| 0.9, 1.1, 1.1, 0.6, 0.6 | 0.86 |
| 0.7, 0.9, 1.1, 0.6, 0.9 | 0.84 |
| 1.1, 0.8, 0.7, 0.7, 0.6 | 0.78 |
| 0.6, 0.9, 0.7, 0.8, 0.8 | 0.76 |
The spread of these means is the uncertainty. With 2 000 resamples the 95% percentile interval is [0.68, 0.98]: faster, but by somewhere between 2% and 32%.
| test | raw p | rank | × (m − k + 1) | adjusted |
|---|---|---|---|---|
| A vs B | 0.01 | 1st | × 3 = 0.03 | 0.03 |
| B vs C | 0.03 | 2nd | × 2 = 0.06 | 0.06 |
| A vs C | 0.04 | 3rd | × 1 = 0.04 | 0.06 |
Unadjusted, all three are "significant" at 0.05. After Holm, only A vs B is. The last row is raised to 0.06 because adjusted values must not decrease down the ranking.
Bonferroni would multiply all three by 3: 0.03, 0.09, 0.12. Same verdict here, but Holm is never stricter.
28 · inference
Fix the instance set before running, and say how it was chosen.
"Faster on average" (an aggregate claim) and "faster on most instances" (a per-instance claim) are different. The audit of unit 07 reports both: pseudocost has the lower SGM and wins 22 of 24.
State the hypothesis, the instances and budget, the effect size with its interval, and what it rules out. "Restarts don't reliably change the median on QCP(12, 55%): 3/8 instances worse, p = 0.79, and the original instance's intervals overlap" rules out a specific claim, which is useful.
section 4
This course's own conclusions, re-run.
28 · audit
An audit. Claim (totals over 3 instances): strong branching uses the fewest nodes; pseudocost is fastest.
| rule | SGM nodes | SGM s | fewest nodes | fastest |
|---|---|---|---|---|
| most-fractional | 800 | 0.58 | 0 | 2 |
| pseudocost | 708 | 0.53 | 1 | 22 |
| strong | 543 | 1.34 | 23 | 0 |
| paired ratio | nodes, 95% CI | seconds, 95% CI |
|---|---|---|
| strong / pseudocost | 0.77 [0.72, 0.82] | 2.31 [2.15, 2.47] |
| strong / most-frac. | 0.68 [0.63, 0.74] | 2.10 [1.94, 2.28] |
| pseudocost / most-frac. | 0.89 [0.84, 0.93] | 0.91 [0.88, 0.95] |
Holm-adjusted Wilcoxon p ≤ 0.0001 for all six.
Strong branching saves about a quarter of the nodes and costs 2.3× the time. Pseudocost is 9% faster than most-fractional.
28 · audit
Claims (one instance, 100 seeds, 3 000-node cap): restarts cut the mean from 153.6 to 52.1, and make the median worse, 31 → 49.5.
| strategy | capped | mean [95% CI] | median [95% CI] |
|---|---|---|---|
| first-fail | 3 | 153.6 [66.6, 267.5] | 31.0 [28.5, 35.0] |
| Luby restarts | 0 | 52.1 [45.0, 59.7] | 49.5 [28.5, 53.5] |
Eight more instances, 30 seeds each: restarts lower the mean on 7/8 (p = 0.03) and raise the median on 3/8 (p = 0.79).
The intervals overlap on the original instance, and the family shows no consistent direction. Unit 19's lab sheet and slides have been corrected in place.
Restarts cut the tail. On the two instances where first-fail hit the cap 6 times in 30, restarts never did (means 1 067 → 194 and 1 141 → 191).
28 · lab
uv run co test 28
uv run co then 28 # audit unit 07 (24 instances) and unit 19 (intervals + 8 instances) · out/profile.png28 · checkpoint
Dolan & Moré, Math. Programming 91 (2002).
Hooker, J. Heuristics 1 (1995).
McGeoch, A Guide to Experimental Algorithmics (2012).
Beiranvand, Hare & Lucet, Optim. Eng. 18 (2017).
Part VII, the coda: 29 convexity and gradients · 30 automatic differentiation · 31 where discrete meets differentiable.