Part II · trunk A · integer programming · spine

Branch and bound

Split the problem, bound each piece with an LP, and throw away every piece that cannot win. What to split on and which piece to look at next decide everything.

needs 05~2 h slides + 3 h lablab: uv run co lab 07

07 · overview

The route

1 · the tree

Branching on a fractional variable, three reasons to prune, the global bound, and the gap.

2 · node selection

Depth-first finds solutions. Best-first proves optimality. What solvers actually ship.

3 · branching

Most-fractional (as good as random), pseudocosts, strong branching, and reliability branching.

4 · the other job

Primal heuristics, reading the gap, and a list of what makes SCIP 6–75× better than yours.

getting around

Words with a dotted underline are glossary terms: click one for its definition. G lists every term in the unit, N shows the notes. Optional extra: interactive figures, linked from the slides they support by a small ▸ chip; skip them freely.

section 1

The tree

Divide the feasible set, and conquer only the pieces that could still contain the optimum.

07 · tree · example

The running example

Dots: the 7 integer points. Ring: the root LP optimum. Branching on $y$ keeps the strip $y \le 2$ and the empty region $y \ge 3$; the gap between the dashed lines holds no integer point.

$$\begin{aligned}\min\ & -x - 5y \\ \text{s.t. } & 7x + 3y \le 22 \\ & -3x + 3y \le 2 \\ & x, y \ge 0 \text{ integer}\end{aligned}$$

Minimizing $-x - 5y$ means maximizing $x + 5y$: $y$ is worth five times as much.

what the root LP says

Dropping integrality, the optimum is the corner $(2, \tfrac83)$ with value $-\tfrac{46}{3} = -15.33$. That is a lower bound: no integer point does better.

$y = 2.67$ is fractional, so branch on it: $y \le 2$ or $y \ge 3$. The next slide is the whole tree.

The polygon. Corners $(0, 0)$, $(\tfrac{22}{7}, 0)$, $(2, \tfrac83)$ and $(0, \tfrac23)$. The top corner is where $7x + 3y = 22$ meets $y = x + \tfrac23$: substituting, $10x + 2 = 22$, so $x = 2$, $y = \tfrac83$.

The integer points. $(0,0), (1,0), (2,0), (3,0), (1,1), (2,1), (2,2)$. Their values of $x + 5y$ are 0, 1, 2, 3, 6, 7, 12, so the optimum is $(2, 2)$ with objective $-12$. Checking $(2,2)$: $14 + 6 = 20 \le 22$ and $-6 + 6 = 0 \le 2$. Nothing with $y = 3$ fits: $y \le x + \tfrac23$ needs $x \ge 3$, and then $7x + 3y \ge 30 > 22$.

In code this is a MILP with c = (-1, -5), A_ub = ((7, 3), (-3, 3)), b_ub = (22, 2), both variables integer in $[0, 10]$. The reference branch_and_bound with most-fractional branching solves it in 5 nodes best-first and 9 depth-first.

07 · tree

LP-based branch and bound

Solve the root LP. If its solution is integral, done.
Otherwise pick a fractional $x_j = v$ and create two children (branching): $x_j \le \lfloor v \rfloor$ and $x_j \ge \lceil v \rceil$. Every integer solution is in exactly one child.
Solve each child's LP, and repeat on the open nodes.

prune a node when

its LP is infeasible ($y \ge 3$);
its LP is integral: it becomes the incumbent if better ($(2,2)$);
its LP bound is $\ge$ the incumbent: nothing below it can win ($x \ge 3$).

explore interactively

Correctness in two lines. The children partition the parent's integer feasible points: no integer satisfies $\lfloor v \rfloor < x_j < \lceil v \rceil$. And each child's LP bound is a valid lower bound on every integer point in it. So pruning never discards a better solution, and the search is finite because bounded integer variables can only be tightened finitely often.

The fractional point itself is cut off in both children ($x_j = v$ violates both new bounds), so the same LP solution never repeats. That is why branching on a fractional variable makes progress, where branching on an integral one wouldn't.

The tree, node by node, from the reference solution. Root: LP $(2, \tfrac83)$, value $-\tfrac{46}{3}$; $y$ is the only fractional variable. Child $y \ge 3$: no point of the polygon has $y \ge 3$ (its top is $\tfrac83$), so the LP is infeasible. Child $y \le 2$: the LP moves right along $y = 2$ to $7x + 6 = 22$, $x = \tfrac{16}{7} = 2.29$, value $-\tfrac{86}{7} = -12.29$. Branch on $x$. Grandchild $x \le 2$: the LP optimum is $(2, 2)$, integral, value $-12$: the incumbent. Grandchild $x \ge 3$: $7 \cdot 3 + 3y \le 22$ gives $y \le \tfrac13$, optimum $(3, \tfrac13)$, value $-\tfrac{14}{3} = -4.67$, which is worse than $-12$, so pruned by bound. Five nodes, and $-12$ is proved optimal.

07 · tree

Two bounds, one gap

Upper bound $U$: the incumbent's objective value, or $+\infty$ before one is found.

Lower bound $L$: the smallest LP bound among open nodes. Every unexplored integer point lies in some open node.

$$L \le z^* \le U, \qquad \text{gap} = \frac{U - L}{|U|}$$

On the example, after the root and its two children: $U = +\infty$, $L = -12.29$. After $(2,2)$ is found: $U = -12$, and the only open node has bound $-4.67 \ge U$, so $L$ jumps to $-12$ and the gap closes to 0.

The solver stops when the gap is 0, or below a tolerance (HiGHS and Gurobi default to a relative gap of $10^{-4}$; SCIP to 0).

two hard jobs

Push $U$ down: find good integer solutions early, so pruning by bound starts early.

Push $L$ up: branch and cut so the open nodes' bounds rise toward $z^*$.

Everything in a MILP solver serves one of these two. A solver log is two columns, "primal bound" and "dual bound", converging.

section 2

Node selection

Which open node to look at next.

07 · selection

Depth-first against best-first

depth-first

Always a child of the last node. Reaches integral leaves quickly, so an incumbent exists early. Small memory. It can spend a long time in a bad subtree.

best-first

Always the open node with the smallest bound. That raises $L$ as fast as possible. The frontier can grow huge, and incumbents come late.

why best-first uses fewest nodes

A node whose bound is below the optimum can never be pruned, so every search must branch on it. Best-first branches on those nodes and (up to ties) nothing else.

full argument in the notes (N)

Lab then, most-fractional branching:

instance	DFS	best
knapsack 22×3 #0	1 373	251
knapsack 22×3 #1	3 681	1 593
UFL 8×20, M = 10C	87	79

What solvers ship: hybrids of node selection. Dive depth-first from a promising node to find a solution, then jump back to a node with a good bound or a good estimate (best estimate), with periodic plunges.

Full argument

Take any node $N$ with LP bound strictly below $z^*$. No incumbent can ever have value below $z^*$, so $N$ is never pruned by bound. It must be branched on unless its LP is integral, which is impossible because its value would then be below $z^*$. Conversely, best-first only pops nodes in bound order. Once the optimal solution has been found, every remaining node has bound at least $z^*$ and is pruned. Lab step 2's extra test checks this directly: the bounds of branched nodes come out nondecreasing and never above the optimum.

DFS's weakness in the numbers: on knapsack #0 it needed 5.5× the nodes. Its strength doesn't show in node counts, because the lab has no time limit. Under a time limit, DFS usually has a much better incumbent to report when time runs out.

07 · selection · example

Both orders, on the running example

best-first: 5 nodes

pop	node	bound	outcome
1	root	−15.33	branch on $y$
2	$y \le 2$	−12.29	branch on $x$
3	$x \le 2$	−12	integral: $U = -12$
4	$x \ge 3$	−4.67	bound $\ge U$: pruned

It takes the smaller bound first, $-12$ before $-4.67$, finds the optimum there, and the other node dies without being branched.

depth-first: 9 nodes

pop	node	bound	outcome
1	root	−15.33	branch on $y$
2	$y \le 2$	−12.29	branch on $x$
3	$x \ge 3$	−4.67	branch on $y$
4	$y \le 0$	−3.14	branch on $x$
5	$x \le 3$	−3	integral: $U = -3$
6	$x \le 2$	−12	integral: $U = -12$

It dives into the last child created, $x \ge 3$, and digs to a poor solution $(3, 0)$ before coming back. Four extra nodes, two of them infeasible children.

explore interactively

"Nodes" counts every LP solved: the root plus every child created. Best-first creates the root, $y \le 2$, $y \ge 3$ (infeasible), $x \le 2$, $x \ge 3$: five. Depth-first additionally branches $x \ge 3$ into $y \le 0$ (LP $(\tfrac{22}{7}, 0)$, value $-3.14$) and $y \ge 1$ (infeasible, since $x \ge 3$ forces $y \le \tfrac13$), then $y \le 0$ into $x \le 3$ (integral $(3, 0)$, value $-3$) and $x \ge 4$ (infeasible): nine.

Depth-first's first incumbent, $-3$, is poor, but it would be a usable answer if time ran out. Best-first has nothing to report until its third pop. That is the trade-off of the previous slide in four rows.

The pop orders are from a traced copy of the reference branch_and_bound (a heap for best-first, a stack for depth-first), with most-fractional branching.

section 3

Branching

Which variable to split on. The single biggest influence on tree size.

07 · branching

Most fractional, and why it's weak

Most-fractional branching. Intuition: a variable at 0.5 is "most undecided" (its fractionality is largest), so decide it.

Measured: about as good as branching on a random fractional variable (Achterberg, Koch & Martin 2005, on MIPLIB).

Fractionality says nothing about how much the bound will move. A variable at 0.5 with a zero-cost column changes nothing when fixed, while a variable at 0.9 on a tight constraint can end the subtree.

what a good rule estimates

For each candidate $j$, the bound improvements $\Delta_j^-$ and $\Delta_j^+$ of the two children, combined with the product score

$$\text{score}_j = \max(\Delta_j^-, \varepsilon) \cdot \max(\Delta_j^+, \varepsilon)$$

A product rewards moving both children, and a zero on one side isn't fatal because of the $\varepsilon$.

07 · branching

Strong branching and pseudocosts

strong branching

Compute $\Delta^\pm_j$ exactly: solve both child LPs for each candidate, then branch on the best. The best node counts known, and the most LP solves by far.

Lab totals, best-first: 1 271 nodes but 4 641 LP solves, 3.2 s, against most-fractional's 1 923 nodes and 1.3 s.

pseudocost branching

Estimate $\Delta^\pm_j$ from history: the average gain per unit of fractionality seen when branching on $x_j$ before, times the current fractionality. Costs nothing per node.

Weak at the top of the tree, where there's no history yet. Lab totals: 1 581 nodes, barely better than most-fractional.

reliability branching (Achterberg, Koch & Martin 2005)

Use strong branching on a variable until its pseudocosts are reliable (a few observations each way), then trust the pseudocosts. This is the default rule in SCIP, and the node count of strong branching at roughly the cost of pseudocosts.

The curriculum's checkpoint is the strong-branching trade-off. Strong branching wins on nodes because every branching decision is informed by exact child bounds. It loses on time because each node costs up to $2k + 1$ LP solves ($k$ candidates) instead of 1. In the lab, 3.7 LP solves per node on average. Whether it pays depends on how expensive an LP is compared with a node's other work, and on how much it shrinks the tree.

Real strong branching is cheaper than the lab's. Solvers limit the dual simplex iterations per probe (a partial solve gives a lower bound on the gain), probe only the most promising candidates, and reuse the parent's basis as a warm start (unit 03).

Pseudocost initialization is the whole problem with pseudocosts. The lab uses the average of what has been seen, or 1.0, for unseen variables, which is uninformative, and that is why pseudocosts barely beat most-fractional here. Reliability branching exists to fix exactly this.

07 · branching · example

Scores and pseudocosts, on the running example

a product score from the real tree

At node $y \le 2$ (bound $-12.29$) we branched on $x = 2.29$:

$\Delta^- = -12 - (-12.29) = 0.29$, $\quad \Delta^+ = -4.67 - (-12.29) = 7.62$

score $= 0.29 \times 7.62 = 2.18$

two candidates, given gains

candidate	$\Delta^-$	$\Delta^+$	sum	product
A	0.1	8	8.1	0.8
B	2	3	5	6

A sum prefers A; the product prefers B, which moves both children.

pseudocosts recorded by that tree

branch	gain	÷ fractional part	per unit
$y$ down, at root	3.05	$\div\ 0.67$	4.57
$x$ down	0.29	$\div\ 0.29$	1.00
$x$ up	7.62	$\div\ 0.71$	10.67

Next time $x$ is fractional elsewhere in the tree, with fractional part 0.5, the pseudocost predicts $\Delta^- \approx 0.5 \times 1.00$ and $\Delta^+ \approx 0.5 \times 10.67$, without solving an LP.

$y$ up was infeasible, so no gain was recorded for it.

Exact values. Node $y \le 2$ has bound $-\tfrac{86}{7}$. Its children have $-12$ and $-\tfrac{14}{3}$, so $\Delta^- = \tfrac27$ and $\Delta^+ = \tfrac{160}{21}$, and the product is $\tfrac{320}{147} \approx 2.18$.

Pseudocosts divide a gain by how far the variable had to move. At the root $y = \tfrac83$ moves down by $\tfrac23$, gain $-\tfrac{86}{7} + \tfrac{46}{3} = \tfrac{64}{21}$, per unit $\tfrac{32}{7} = 4.57$. At node $y \le 2$, $x = \tfrac{16}{7}$ moves down by $\tfrac27$ (gain $\tfrac27$, per unit 1) and up by $\tfrac57$ (gain $\tfrac{160}{21}$, per unit $\tfrac{32}{3} = 10.67$). These are exactly the lists the reference tree stores in stats["pseudo"].

Candidates A and B are made up to show the two scores disagreeing; A is the kind of variable that makes one child much better and leaves the other essentially the parent.

section 4

The other job, and the gap to SCIP

Finding good solutions early, and what else solvers do.

07 · primal side

Primal heuristics

A good incumbent early lets bound pruning start early. Solvers spend a share of their time on primal heuristics that find one:

Rounding: round the LP solution in a way that keeps feasibility where possible (simple rounding, ZI rounding).
Diving: fix a variable, re-solve the LP, repeat, down one branch without keeping the tree.
RINS (Danna, Rothberg & Le Pape 2005): fix the variables where the incumbent and the LP agree, and solve the small sub-MILP that remains.
Feasibility pump (Fischetti, Glover & Lodi 2005): alternate between the LP polytope and the nearest integer point.

in the lab's tree

No heuristics: incumbents only come from integral LP leaves. That is one reason the DFS runs in then are so much larger than SCIP's. SCIP typically has a near-optimal solution before its tree has a dozen nodes.

On the running example, rounding the root LP $(2, 2.67)$ down gives $(2, 2)$, the optimum, before any branching.

Unit 26's local search and unit 19's LNS are the same idea used as stand-alone methods.

07 · the gap

Your factor of shame

instance	yours, best	SCIP bare*	SCIP	HiGHS
knapsack 22×3 #0	157	87	8	9
knapsack 22×3 #1	1 039	203	161	208
UFL 8×20, M = 10C	75	41	1	1

"Yours, best": the fewest nodes among your six variants. * SCIP bare: presolve and cuts off. The ratio of your column to SCIP's is the factor of shame.

hypotheses to test

Cutting planes raise the root bound (unit 08). Default vs bare SCIP isolates them plus presolve.
Presolve tightens bounds and coefficients (units 05, 27).
Branching: reliability branching vs yours. Bare SCIP still wins by 2–5×.
Heuristics give an early incumbent, and more pruning.
Conflict analysis learns from infeasible nodes (the SAT idea of unit 20).

07 · lab

Lab 07 — what you build

about 3 hours · uv run co lab 07 · LPs solved for you

Most fractional and the two children.
The tree: prune, accept incumbents, branch, best-first or DFS, and record pseudocost history.
The product score.
Pseudocost branching from the recorded history.
Strong branching, with its probes counted.

uv run co test 07
uv run co then 07     # the 3x2 matrix, SCIP bare, SCIP and HiGHS default, factor of shame

07 · checkpoint

Done when

checkpoint

All five lab steps are ✓.
Your strong-branching variant beats your most-fractional variant on node count while losing on wall time, and you can explain that trade-off in terms of LP solves per node.
You can say what reliability branching is and why it exists.

Reading

Wolsey, ch. 7.
Achterberg, Koch & Martin, "Branching rules revisited", ORL 33 (2005).
Achterberg (2007), thesis, ch. 5–6 (optional).

08 cutting planes, which test the first hypothesis · 16 DP, where a third exact method meets this one · 31 learns a branching rule for this tree.