Part II · trunk A · integer programming · branch · deep theory

Separation ⇔ optimization

You can optimize over a polytope in polynomial time exactly when you can find a violated inequality in polynomial time. Exponentially many constraints are fine.

needs 04, 08~2 h slides + 2½ h lablab: uv run co lab 09

The curriculum calls this the most beautiful result in the course, and the branch unit it would fight to keep. It turns the ellipsoid method (unit 04), which is useless for solving LPs, into the reason we can trust formulations with exponentially many constraints. It also explains why lazy constraints, which every solver supports, are sound engineering and not a hack.

The lab makes it concrete on the oldest example: Dantzig, Fulkerson and Johnson's 1954 subtour formulation of the TSP.

How to read this deck

The theorem is given as two ideas, each with a picture (the second one is a square and its polar, a diamond); the ellipsoid details are in the notes. The subtour section runs on six cities, where the separation oracle is carried out by hand: components on an integer point, a minimum cut on a fractional one, and Stoer–Wagner on four vertices. Every cut value was computed with the lab's reference solution.

09 · overview

The route

1 · the theorem

Optimization and separation oracles, the Grötschel–Lovász–Schrijver equivalence, and why the ellipsoid method is the bridge.

2 · subtours

The subtour LP, separation by minimum cut, and Stoer–Wagner.

3 · in practice

Lazy constraints, cut callbacks, how good the bound is, and Concorde.

getting around

Words with a dotted underline are glossary terms: click one for its definition. G lists every term in the unit, N shows the notes. Optional extra: interactive figures, linked from the slides they support by a small ▸ chip; skip them freely.

Lab then: at 200 cities there are 8 × 10⁵⁹ subtour constraints. The LP bound needed 56, the optimal tour 70.

section 1

The theorem

Two oracles for a polytope, and the fact that they are equally powerful.

09 · theorem

Two ways to be given a polytope

optimization oracle

Given $c$: return $\arg\max\{c^\top x : x \in P\}$, or report $P = \emptyset$.

"Solve the problem for any objective."

separation oracle

Given $y$: either certify $y \in P$, or return an inequality $a^\top x \le \beta$ valid for $P$ with $a^\top y > \beta$.

"Find a violated constraint, if there is one."

A polytope with exponentially many facets can have a short separation oracle, one that searches its facets implicitly. For subtour constraints it is a minimum cut, and for matching's odd-set constraints a minimum odd cut.

For the theorem, $P$ must be "well-described": its facets have bounded encoding size $\varphi$, and the running time may be polynomial in $\varphi$ and $n$.

09 · theorem

Grötschel, Lovász & Schrijver (1981)

theorem

For a family of well-described polyhedra, the optimization problem is solvable in polynomial time if and only if the separation problem is.

separation ⇒ optimization, the idea

The ellipsoid method (unit 04) only ever asks "is this centre inside, and if not, which inequality fails?". A separation oracle answers exactly that, so the ellipsoid method runs, and with the objective added as a sliding objective it optimizes.

optimization ⇒ separation, the idea

Flip the picture with the polar $P^\circ$: facets of $P$ become vertices of $P^\circ$, and vertices of $P$ become facets. "Find a violated facet of $P$" becomes "optimize over $P^\circ$", and the next slide shows it on a square.

details in the notes (N)

Details

Separation ⇒ optimization. Run the ellipsoid method on $P \cap \{c^\top x \ge \gamma\}$. Its cuts come from the separation oracle, or from the objective constraint $c^\top x \ge \gamma$ when the centre is feasible (a "sliding objective"). The volume argument bounds the number of iterations, and rounding at the end gives an exact optimal vertex.

Optimization ⇒ separation. With $0$ in the interior of $P$, the polar is $P^\circ = \{a : a^\top x \le 1\ \forall x \in P\}$. Separating $y$ from $P$ means finding $a \in P^\circ$ with $a^\top y > 1$, which is optimization over $P^\circ$. A separation oracle for $P^\circ$ is "maximize $a^\top x$ over $P$", which is exactly the optimization oracle we have. Apply the first direction to $P^\circ$.

The first direction is the one the lab's intuition uses. The ellipsoid method never needs the constraint list, only an answer to "is this centre feasible, and if not, which inequality does it violate?" (unit 04, slide "Useless in practice, indispensable in theory").

The second direction is the elegant one. The polar of a polytope has a vertex for each facet of $P$ and a facet for each vertex of $P$. An optimization oracle for $P$ answers "which vertex of $P$ maximizes this direction", which is exactly "which facet of $P^\circ$ is most violated by this point". So optimizing over $P$ is separating over $P^\circ$. Running ellipsoid over $P^\circ$ with that oracle finds a most-violated facet of $P$.

Practical meaning of the second direction: if a problem is NP-hard to optimize over, like the TSP, then separating over its full convex hull is NP-hard too. That is why TSP codes separate only chosen families (subtours, combs, …), and never "all facets".

09 · theorem · example

A square and its polar

Left: $P = [-1,1]^2$ and $y = (1.5, 0.5)$. Right: $P^\circ = \{a : |a_1| + |a_2| \le 1\}$. The facet $x_1 \le 1$ of $P$ is the vertex $(1, 0)$ of $P^\circ$.

$P$ (square)	$P^\circ$ (diamond)
4 facets $\pm x_1 \le 1$, $\pm x_2 \le 1$	4 vertices $(\pm1, 0)$, $(0, \pm1)$
4 vertices $(\pm1, \pm1)$	4 facets $\pm a_1 \pm a_2 \le 1$

separating the point by optimizing over the diamond

Is $y = (1.5, 0.5)$ in $P$? Maximize $a^\top y = 1.5a_1 + 0.5a_2$ over $P^\circ$. The best vertex is $a = (1, 0)$, value $1.5$.

$1.5 > 1$, so $a^\top x \le 1$, i.e. $x_1 \le 1$, is valid for $P$ and violated by $y$. A cut, found by an optimization.

If the maximum had been $\le 1$, every facet of $P$ would hold at $y$, so $y \in P$.

explore interactively

Why the polar of the square is the diamond. $a$ is in $P^\circ$ when $a^\top x \le 1$ for every $x$ in the square. The worst $x$ is a vertex, $(\pm1, \pm1)$, giving $|a_1| + |a_2| \le 1$. So each vertex of $P$ contributes one facet of $P^\circ$.

Why a vertex of $P^\circ$ is a facet of $P$. The vertex $(1, 0)$ stands for the inequality $1 \cdot x_1 + 0 \cdot x_2 \le 1$, and that is a facet of the square. Every valid inequality of the form $a^\top x \le 1$ is a point of $P^\circ$, and the strongest ones are its vertices.

For the square the polar is written out, so this is a toy. The theorem's power is when $P$ has exponentially many facets: $P^\circ$ then has exponentially many vertices, and "optimize over $P^\circ$" is done by the ellipsoid method with the optimization oracle for $P$ as its separation oracle.

09 · theorem

What it bought

result	via
Maximum-weight matching as an LP in polynomial time	odd-set inequalities (unit 06) separated by a minimum odd cut (Padberg & Rao 1982)
The first polynomial algorithm for submodular function minimization	separation over the base polytope by a greedy algorithm (unit 13)
Maximum stable set in perfect graphs	Lovász's theta function, an SDP, separated over the theta body (unit 25)
Subtour LP for the TSP in polynomial time	subtour constraints separated by a global minimum cut (this lab)
NP-hardness of separation for hard polytopes	the other direction, contrapositive

Some of these (stable sets in perfect graphs, submodular minimization, for a long time) had no other known polynomial algorithm. The ellipsoid method was the only route.

section 2

Subtours

The constraint family that started it all, and its minimum-cut oracle.

09 · subtours

The subtour elimination LP

Solid edges at 1, dashed at ½. Every degree is 2 and the graph is connected, yet the cut around $\{3,4,5\}$ weighs 1: a subtour constraint is violated.

Edge variables $x_e$ on the complete graph, the DFJ formulation (1954), with $\delta(S)$ the edges leaving $S$:

$$\begin{aligned}\min\ & \textstyle\sum_e c_e x_e \\ \text{s.t. } & \textstyle\sum_{e \in \delta(v)} x_e = 2 \quad \forall v \\ & \textstyle\sum_{e \in \delta(S)} x_e \ge 2 \quad \forall\, \emptyset \ne S \subsetneq V \\ & 0 \le x_e \le 1\end{aligned}$$

The degree constraints alone allow subtours: several disjoint cycles. The subtour constraints say every vertex set must be entered and left. There are $2^{n-1}-1$ of them.

09 · subtours

Separation is a minimum cut

claim

$x^*$ violates some subtour constraint iff the global minimum cut of the support graph with edge weights $x^*$ has value below 2.

The constraint for $S$ is exactly "the cut $\delta(S)$ has weight $\ge 2$", so the most violated one is the lightest cut. It is a restatement, not a trick.

Integer points are easier: a 0/1 solution of the degree constraints is a union of cycles, and each cycle's vertex set has cut weight 0. The connected components check suffices (lab step 5).

Stoer–Wagner (1997)

Build a maximum-adjacency order: repeatedly add the vertex most strongly attached to those already added.
The last vertex's attachment, the cut of the phase, is a minimum cut separating the last two vertices, $s$ and $t$.
Record it, merge $s$ and $t$, and repeat until one vertex is left.

$n - 1$ phases, $O(nm + n^2 \log n)$ in total. No flows needed.

why merging is safe: notes (N)

explore interactively

09 · subtours · example

The oracle, on six cities

an integer point: components are enough

$x = 1$ on the edges of triangles $\{0,1,2\}$ and $\{3,4,5\}$, 0 elsewhere. Every degree is 2.

Support graph components: $\{0,1,2\}$ and $\{3,4,5\}$. Two components, so both constraints $x(\delta(\{0,1,2\})) \ge 2$ and $x(\delta(\{3,4,5\})) \ge 2$ are violated, with left side 0.

a fractional point: connected, still violated

In each triangle two edges at 1 and one at $\tfrac12$: $01 = 12 = 1$, $02 = \tfrac12$, and the same on $\{3,4,5\}$. Two bridges at $\tfrac12$: $03$ and $25$.

Degrees: vertex 0 has $1 + \tfrac12 + \tfrac12 = 2$, vertex 1 has $1 + 1 = 2$, and so on. All degree constraints hold, and the support graph is connected.

Minimum cut: $\{3,4,5\}$, weight $\tfrac12 + \tfrac12 = 1 < 2$. The constraint for $S = \{3,4,5\}$ is violated.

Stoer–Wagner on four vertices

Weights $01 = 3$, $23 = 3$, $02 = 12 = 13 = 1$.

phase	order (attachment when added)	cut of phase
1	0, 1 (3), 2 (2), 3 (4)	$\{3\}$: 4
2	0, 1 (3), $\{2,3\}$ (3)	$\{2,3\}$: 3
3	0, $\{1,2,3\}$ (4)	$\{1,2,3\}$: 4

After each phase the last two are merged. The best phase cut, 3, is the global minimum: $\{0,1\}$ against $\{2,3\}$.

Why the fractional point needs a real min cut: components see one piece, and every degree is right. Only a cut computation finds the thin connection.

explore interactively

The reference oracle. separate_subtours on the integer point returns both components; on the fractional point components returns one piece and min_cut returns $(1.0, (3, 4, 5))$. The cut edges are exactly the two bridges. The constraint for $\{0,1,2\}$ is the same one, since a set and its complement share a cut.

The Stoer–Wagner phases on the four-vertex graph. Phase 1 starts at 0. Vertex 1 is attached by 3, vertex 2 by 1, vertex 3 by 0: add 1. Now 2 is attached by $1 + 1 = 2$ and 3 by 1: add 2. Then 3 is attached by $1 + 3 = 4$: add 3, cut of the phase 4, merge 2 and 3. Phase 2: 1 is attached by 3, $\{2,3\}$ by $1$; add 1; then $\{2,3\}$ is attached by $1 + 1 + 1 = 3$: cut of the phase 3. Phase 3: one merge left, cut 4. The best is 3, matching the reference min_cut, which returns $(3.0, (2, 3))$.

Why the cut of a phase is a minimum cut between its last two vertices is the lemma in the paper. The consequence used above is simpler: any global minimum cut either separates that pair, and then the phase found something at least as good, or keeps them together, and then it survives their merge.

09 · subtours

A few dozen constraints out of 10⁵⁹

$n$	constraints that exist	used by the LP	LP bound	optimum	MIP cuts	gap
30	5.4 × 10⁸	6	480.5	482	5	0.31%
80	6.0 × 10²³	16	725.0	726	24	0.14%
120	6.7 × 10³⁵	39	813.0	819	52	0.73%
200	8.0 × 10⁵⁹	56	1083.5	1091	70	0.69%

Lab then, random Euclidean instances. The exact tour at $n = 200$ took 10 MIP solves and 46 s with HiGHS as the integer solver.

section 3

In practice

Lazy constraints, and how far the idea goes.

09 · practice

Lazy constraints and callbacks

Every MIP solver can take constraints it doesn't see up front:

Lazy constraints: checked on every candidate integer solution; if one is violated, the solution is rejected and the constraint is added. Required for correctness.
User cuts: separated at fractional LP points to tighten the bound. Optional; only for speed.

Gurobi: a callback with cbLazy and cbCut. SCIP: a constraint handler with check, enforce and separate methods. That is the same distinction.

the classic bug

Adding subtour constraints only as user cuts means the solver can accept an integer solution made of subtours, if no fractional node happened to trigger separation. Lazy constraints need enforcement on integer solutions, which is what lab step 5 does, clumsily.

09 · practice

How good is the bound, and how far has this gone?

integrality gap of the subtour LP

For metric TSP the optimal tour is at most $\tfrac32$ times the subtour LP bound (Wolsey 1980; Shmoys & Williamson 1990), and instances with gap close to $\tfrac43$ exist. The 4/3 conjecture says that is the truth; it is open.

On random Euclidean instances the gap is tiny, under 1% in the lab.

Unit 11 computes the same bound a completely different way (the Held–Karp bound from 1-trees), and gets the same number.

Concorde

Concorde: Applegate, Bixby, Chvátal & Cook. Branch-and-cut with subtour constraints, comb inequalities, and "local cuts" found by solving small projected subproblems.

It solved an 85 900-city instance from a chip-layout application (pla85900) to proven optimality in 2006, with about 136 CPU-years of computation.

09 · lab

Lab 09 — what you build

about 2½ hours · uv run co lab 09

A subtour constraint, and the connected components of a support graph.
Stoer–Wagner global minimum cut.
The separation oracle: components, else min cut.
The subtour LP by lazy constraints, which equals the LP with every one written out.
An exact TSP by lazy cuts on the integer program.

uv run co test 09
uv run co then 09     # constraints used vs constraints that exist, LP bound vs optimum, up to 200 cities

09 · checkpoint

Done when

checkpoint

All five lab steps are ✓.
You can state the separation–optimization theorem precisely, and you have solved a TSP over an exponential constraint family while only ever writing down a few hundred of its constraints.
You can explain how matching's odd-set inequalities are separated, and what that implies about maximum-weight matching.

Reading

Grötschel, Lovász & Schrijver, ch. 6 (consulted).
Schrijver, Combinatorial Optimization, §5.10–5.11 and ch. 58.
Applegate, Bixby, Chvátal & Cook (2006), ch. 5–6.
Stoer & Wagner, JACM 44 (1997).

10 column generation, where the same idea applies to columns · 11 Lagrangian relaxation and the Held–Karp bound · 12 Benders, where constraints are generated from a subproblem's duals.

Separation ⇔ optimization

The route

The theorem

Two ways to be given a polytope

Grötschel, Lovász & Schrijver (1981)

A square and its polar

What it bought

Subtours

The subtour elimination LP

Separation is a minimum cut

The oracle, on six cities

A few dozen constraints out of 10⁵⁹

In practice

Lazy constraints and callbacks

How good is the bound, and how far has this gone?

Lab 09 — what you build

Done when

Reading

Next