Part II · trunk A · integer programming · spine

Lagrangian relaxation and subgradient methods

Price the hard constraints into the objective and solve what is left. Every price gives a bound; the best price gives exactly the bound you can predict in advance.

needs 03, 16~1¾ h slides + 2½ h lablab: uv run co lab 11

Unit 10 reached a strong bound by building an LP with exponentially many columns. This unit reaches the same bound without any LP solver: pick prices for the complicating constraints, solve an easy combinatorial problem, and adjust the prices. The two methods are dual views of one convexification, and the unit's main theorem (Geoffrion) says precisely which bound you get.

Lagrangian relaxation was the dominant bounding technique for combinatorial optimisation from the 1970s until LP solvers became fast in the 1990s. It is still the method of choice when the relaxed problem has a fast specialised algorithm: spanning trees, shortest paths, knapsacks.

How to read this deck

Two small examples carry it. A three-variable integer program with one dualised constraint, whose $L(u)$ is drawn exactly and climbed by hand; and five cities, where one subgradient step turns a 1-tree into an optimal tour and two edges are fixed. All values were computed with the lab's reference solution. The proof of Geoffrion's theorem is in the notes.

11 · overview

The route

1 · dualising

The Lagrangian function, weak duality, concavity; Geoffrion's theorem and the integrality property.

2 · subgradients

Step rules, why it is not an ascent method, and the zig-zag.

3 · Held–Karp

1-trees, the bound that equals the subtour LP, and what LKH does with it.

4 · using the multipliers

Lagrangian heuristics, and fixing variables by reduced cost.

getting around

Words with a dotted underline are glossary terms: click one for its definition. G lists every term in the unit, N shows the notes. Optional extra: interactive figures, linked from the slides they support by a small ▸ chip; skip them freely.

section 1

Dualising

Move the hard constraints into the objective.

11 · dualising

The Lagrangian function

$$z = \min \{\, c^\top x : Ax \ge b,\ x \in X \,\}$$

$Ax \ge b$ are the complicating constraints; optimising over $X$ alone is easy. Dualise them with multipliers $u \ge 0$, giving the Lagrangian function:

$$L(u) = \min_{x \in X}\ c^\top x + u^\top (b - Ax)$$

weak duality

For any feasible $x$ and $u \ge 0$, $u^\top(b - Ax) \le 0$, so $L(u) \le c^\top x$. Hence $L(u) \le z$.

Equality constraints get free-sign multipliers. The GAP lab dualises "each job exactly once" with $u \in \mathbb R^n$.

$L$ is the lower envelope of the affine functions $u \mapsto c^\top x + u^\top(b - Ax)$, one per $x \in X$. So it is concave and piecewise linear, and non-differentiable exactly at its maximum.

11 · dualising · example

L(u) on a three-variable example

$L(u)$ for the example. It rises while the items bought weigh less than 5, peaks at $u = \tfrac43$, and falls once they weigh more.

$$\begin{aligned}\min\ & 3x_1 + 5x_2 + 4x_3 \\ \text{s.t. } & 2x_1 + 4x_2 + 3x_3 \ge 5, \quad x \in \{0,1\}^3\end{aligned}$$

Dualise the one constraint. For a price $u$ each item costs $c_j - u\,w_j$; take it if that is negative:

$$L(u) = 5u + \textstyle\sum_j \min(0,\ c_j - u\,w_j)$$

two prices by hand

$u = 1$: item costs $3-2,\ 5-4,\ 4-3 = 1, 1, 1$, all positive, buy nothing. $L = 5$. Violation $g = 5 - 0 = 5$: the price is too low.

$u = 2$: costs $-1, -3, -2$, buy all three. $L = 10 - 6 = 4$. $g = 5 - 9 = -4$: too high.

The best price, $u = \tfrac43$, gives $\tfrac{19}{3} = 6.33$, below the true optimum 7 (buy items 1 and 3).

explore interactively

The pieces. Item $j$ becomes worth buying when $u > c_j / w_j$: item 2 at $\tfrac54$, item 3 at $\tfrac43$, item 1 at $\tfrac32$. So $L = 5u$ up to $\tfrac54$, then $5u + 5 - 4u = u + 5$, then $u + 5 + 4 - 3u = 9 - 2u$ after $\tfrac43$, then $12 - 4u$ after $\tfrac32$. The maximum is at $\tfrac43$: $\tfrac43 + 5 = \tfrac{19}{3}$. The slope changes from $+1$ to $-2$ there, and the subgradients at the kink are the violations $g = 1$ (buy item 2 only) and $g = -2$ (buy items 2 and 3).

The integer optimum. Sets that cover weight 5: $\{1,2\}$ cost 8, $\{1,3\}$ cost 7, $\{2,3\}$ cost 9, all three cost 12. So $z = 7$.

Why the bound equals the LP. $X = \{0,1\}^3$ is a box, whose LP is integral: the integrality property of the next slide. The LP relaxation takes $x_2 = 1$ and $x_3 = \tfrac13$, cost $5 + \tfrac43 = \tfrac{19}{3}$, the same number.

11 · dualising

Geoffrion 1974: which bound you get

theorem $$z_{LD} = \min \{\, c^\top x : Ax \ge b,\ x \in \mathrm{conv}(X) \,\}$$

So $z_{LP} \le z_{LD} \le z$, where $z_{LP}$ also relaxes $X$ to its LP description $\bar X$.

integrality property

If $\mathrm{conv}(X) = \bar X$ (the easy problem's LP has integral vertices), then $z_{LD} = z_{LP}$. The Lagrangian relaxation gives nothing the LP didn't, though it may compute it faster. The example's box $\{0,1\}^3$ is such an $X$: 6.33 both ways.

why: notes (N)

It is the same bound as Dantzig–Wolfe (unit 10), whose master convexifies $X$ explicitly.

Lab then, GAP, 300 iterations:

m × n	opt	LP	A: knapsacks	B: one agent per job
5 × 25	452	444.13	452.00	444.12
5 × 40	682	671.18	682.00	671.12
8 × 60	905	892.37	902.83	892.33
10 × 80	1084	1072.32	1083.71	1072.27

A leaves one knapsack per agent, which has no integrality property, so it closes 83–100% of the gap. B leaves "each job picks an agent", whose LP is integral, so it stops at the LP, as predicted.

section 2

Subgradients

Climbing a function with no gradient.

11 · subgradients

Subgradient ascent

subgradient

If $x(u)$ attains $L(u)$, then $g = b - Ax(u)$ satisfies $L(v) \le L(u) + g^\top (v - u)$ for all $v$. Computing it is free: the violation of the dualised constraints.

$$u \leftarrow \bigl[u + t_k\, g\bigr]_+$$

Diminishing $t_k$ with $\sum t_k = \infty$ and $\sum t_k^2 < \infty$ converges, slowly: $O(1/\varepsilon^2)$ iterations.
Polyak: $t_k = \lambda_k (\bar z - L(u))/\lVert g\rVert^2$, with $\bar z$ an upper bound. Held–Wolfe–Crowder: halve $\lambda$ when progress stalls. That's lab step 2.

not an ascent method

A subgradient step can decrease $L$, even with a tiny step. The subgradient guarantees getting closer to the optimal multipliers, not a higher value. Always keep the best bound so far.

Lab then, bound A on GAP 8 × 60, 300 iterations:

rule	best	steps down
λ = 2, halve after 20 fails	902.83	144 / 299
λ = 2 fixed	900.40	142 / 299
λ = 0.1 fixed	895.93	96 / 299

11 · subgradients · example

Subgradient steps, on the example

Polyak steps $t = \lambda\,(7 - L(u)) / g^2$, with the incumbent 7 as $\bar z$, starting at $u = 0$, on the example.

λ = 1 fixed: the zig-zag

$k$	$u$	$L(u)$	$g$	step
0	0	0	5	+1.4
1	1.4	6.2	−2	−0.4
2	1.0	5.0	5	+0.4
3	1.4	6.2	−2	−0.4

Step 2 lowers $L$ from 6.2 to 5, then it repeats forever: 1.4, 1.0, 1.4, … The best bound stays 6.2.

halve λ after every step without a new best

$k$	$u$	$L(u)$	λ	best
3	1.4	6.2	0.5	6.2
4	1.2	6.0	0.25	6.2
5	1.25	6.25	0.125	6.25
6	1.344	6.3125	0.125	6.3125

Same first three steps, then shrinking steps close in on $u = \tfrac43$, $L = 6.33$.

explore interactively

Step 0. $L(0) = 0$ (buy nothing), $g = 5$, $t = (7 - 0)/25 = 0.28$, so $u$ moves by $0.28 \times 5 = 1.4$. Step 1. At $u = 1.4$ items 2 and 3 are worth buying ($5 - 5.6 < 0$, $4 - 4.2 < 0$), weight 7, $g = -2$; $L = 7 + (-0.6) + (-0.2) = 6.2$; $t = 0.8 / 4 = 0.2$, move $-0.4$. Step 2. At $u = 1$ nothing is bought, $L = 5$, $g = 5$, $t = 2 / 25 = 0.08$, move $+0.4$: back to 1.4.

The halving column uses a stricter rule than the lab (which waits 20 failures) so that the effect shows within a few steps. From $k = 3$ onward: $\lambda = 0.5$ gives a move of $-0.2$ to 1.2; there $L = 6$, $g = 5$, and with $\lambda = 0.25$ the move is $+0.05$ to 1.25, where item 2 is exactly indifferent and $L = 6.25$ whichever way it is decided. The steps continue toward the kink at $\tfrac43$.

Every iterate's $L$ is a valid lower bound on 7, including the bad ones. That is why the method only has to remember the best, not climb monotonically.

11 · subgradients

Beyond plain subgradients

cutting planes on the dual = column generation

Every $x \in X$ you've seen gives an affine upper piece of $L$. Maximising the envelope of the pieces so far is an LP: Kelley's cutting-plane method. Its dual is unit 10's restricted master, and its columns are the $x$'s. It has the same wild oscillation that unit 10 stabilised.

Bundle methods (Lemaréchal, Kiwiel): the cutting-plane model plus a quadratic proximal term. The stabilisation that works, with convergence guarantees.
Volume algorithm (Barahona & Anbil 2000): a subgradient method that also averages the relaxed solutions into an approximate primal LP solution.
Primal recovery from subgradients: averages of $x(u_k)$ with suitable weights converge to an optimal solution over $\mathrm{conv}(X)$.

Rule of thumb: subgradients for speed and simplicity with a cheap oracle; bundle or column generation when you need the exact bound or primal information.

section 3

Held–Karp

A spanning tree, priced into a tour.

11 · Held–Karp

1-trees and the Held–Karp bound (1970–71)

A 1-tree is a spanning tree on $\{1, \dots, n-1\}$ plus two edges at vertex 0. Every tour is a 1-tree with all degrees 2.

Dualise "degree = 2" with vertex penalties $\pi_v$: edge costs become $d_{ij} + \pi_i + \pi_j$, and

$$L(\pi) = \min_{T \text{ 1-tree}} \sum_{ij \in T} (d_{ij} + \pi_i + \pi_j) - 2 \sum_v \pi_v$$

Subgradient: $g_v = \deg_T(v) - 2$. Raise the price of vertices with too many edges. If $g = 0$, the 1-tree is a tour, and it's optimal.

the Held–Karp bound equals the subtour LP

The 1-tree polytope's LP description is integral (a matroid-type polytope, unit 13), so by Geoffrion

$$\max_\pi L(\pi) = z_{\text{subtour LP}}$$

n	opt	degree LP	subtour LP	Held–Karp
30	482	470.5	480.50	480.50
60	627	572.5	621.50	621.50
100	776	744.0	771.50	771.50
150	921	873.0	918.75	918.27

This is the curriculum's "then" step: the Held–Karp bound and the root LP bound are close, and the reason is Geoffrion's theorem. The subtour LP is what a MIP solver reaches at the root after separating subtour cuts (unit 09), and the Lagrangian dual of the 1-tree relaxation is the same number. At $n = 150$ the ascent stopped at the 1000-iteration limit, 0.05% short.

Historically, Held and Karp's bound let branch-and-bound solve TSPs of 50–100 cities in the early 1970s. Concorde now uses LP with cutting planes instead, because an LP gives duals for cut separation and is exact. But LKH, the best heuristic, uses the Held–Karp multipliers to build its candidate edge lists (α-nearness, Helsgaun 2000). An edge's α-value is lab step 3's "cost of the cheapest 1-tree containing it" minus the minimum 1-tree's cost.

11 · Held–Karp · example

A 1-tree on five cities

	0	1	2	3	4
0	·	4	5	3	2
1		·	3	5	2
2			·	4	3
3				·	3

π = 0: the plain 1-tree

Tree on 1–4: $14$ (2), $12$ (3), $34$ (3). Plus the two cheapest edges at 0: $04$ (2), $03$ (3). Cost 13.

Degrees $(2, 2, 1, 2, 3)$, so $g = (0, 0, -1, 0, 1)$: vertex 2 is a leaf, vertex 4 a hub.

one step, and it is a tour

From the plain 1-tree, with $\bar z = 14$: $t = 2 \cdot (14 - 13) / 2 = 1$, so $\pi = (0, 0, -1, 0, 1)$. Edges at 2 get cheaper by 1, edges at 4 dearer by 1.

The new minimum 1-tree is the tour $0\!-\!3\!-\!2\!-\!1\!-\!4\!-\!0$, all degrees 2, length $3 + 4 + 3 + 2 + 2 = 14$. $L(\pi) = 14 = \bar z$: optimal, proved.

reduced-cost fixing, at π = 0

Force edge $13$ (length 5) into the plain 1-tree: it must replace the heaviest edge on the tree path $1\!-\!4\!-\!3$, which is 3. Cost $13 + 5 - 3 = 15 > 14$: no tour of length 14 uses edge 13. Fixed to 0. Edge $02$ is fixed the same way.

explore interactively

The minimum 1-tree. Prim's algorithm on vertices 1–4 takes $14$ (2), then the cheaper of $12$, $24$, $34$ (all 3): it picks $12$, then $34$ ($24$ would close a cycle). The reference one_tree returns bound 13, degrees $[2, 2, 1, 2, 3]$ and edges $14, 12, 34, 04, 03$.

The step. From the plain 1-tree, $\lVert g \rVert^2 = 2$, and with the lab's $\lambda = 2$ the Polyak step is $2 \cdot 1 / 2 = 1$. The reference held_karp_bound records exactly two values, 13 then 14. The optimal tour of the five cities, by brute force over the 24 orderings, has length 14.

Fixing. The reference fix_edges with $\bar z = 14$ returns $\{02, 13\}$. For $02$ (length 5): adding it at vertex 0 displaces the dearer of the two vertex-0 edges, 3, giving $13 + 5 - 3 = 15 > 14$. In a 5-city instance two edges out of ten is modest; the lab's larger instances fix around 90%.

section 4

Using the multipliers

A bound is also a guide.

11 · using the multipliers

Fixing variables, and repairing solutions

reduced-cost fixing

If every solution with $x_e = 1$ has Lagrangian value at least $L(u) + \rho_e$, and $L(u) + \rho_e > \bar z$, then $x_e = 0$ in every solution better than the incumbent $\bar z$.

For 1-trees: $\rho_e$ $= c_e - (\text{heaviest edge on the tree path})$, the matroid exchange of unit 13.

Lab then, with the optimum as $\bar z$: 89% / 89% / 93% / 96% of edges eliminated at $n$ = 30 / 60 / 100 / 150.

Lagrangian heuristics

The relaxed solution violates only the dualised constraints, and often only slightly. Repair it. For GAP from relaxation A: every agent is within capacity, jobs are over- or under-assigned, so keep the cheapest copy and place the rest by regret (lab step 5).

m × n	opt	bound A	repair
5 × 25	452	452.00	452
5 × 40	682	682.00	682
8 × 60	905	902.83	stuck
10 × 80	1084	1083.71	1178

11 · lab

Lab 11 — what you build

about 2½ hours · uv run co lab 11

The minimum 1-tree under multipliers.
Subgradient ascent (Polyak step, halving) and the Held–Karp bound.
Edge fixing by 1-tree reduced costs.
Two relaxations of GAP: one that can beat the LP, one that cannot.
A Lagrangian repair heuristic.

uv run co test 11
uv run co then 11     # HK vs degree LP and subtour LP · GAP bounds · the zig-zag

11 · checkpoint

Done when

checkpoint

All five lab steps are ✓.
Before writing any code, you can predict whether dualising a given constraint set will beat the LP relaxation, using the integrality property.
You can say why the Held–Karp bound equals the subtour LP.

Reading

Geoffrion, Math. Programming Study 2 (1974).
Held & Karp, Operations Research 18 (1970); Math. Prog. 1 (1971).
Fisher, Management Science 27 (1981).
Held, Wolfe & Crowder, Math. Prog. 6 (1974).

12 Benders decomposition, which generates constraints as unit 10 generated columns · 23 Christofides, whose analysis uses the subtour LP bound again · 26 LKH's candidate sets.

Lagrangian relaxation and subgradient methods

The route

Dualising

The Lagrangian function

L(u) on a three-variable example

Geoffrion 1974: which bound you get

Subgradients

Subgradient ascent

Subgradient steps, on the example

Beyond plain subgradients

Held–Karp

1-trees and the Held–Karp bound (1970–71)

A 1-tree on five cities

Using the multipliers

Fixing variables, and repairing solutions

Lab 11 — what you build

Done when

Reading

Next