Part I · trunk A · linear programming · spine

Simplex, honestly

Walking from vertex to better vertex: why it works, when it loops forever, when it takes exponentially long, and why it is still the workhorse.

needs 01~2 h slides + 3 h lablab: uv run co lab 02

Unit 01 ended with the fundamental theorem: a bounded, feasible LP over a pointed polyhedron is optimized at a vertex, and vertices are basic feasible solutions. Lab 01 enumerated all $\binom{m}{n}$ bases to find them. This unit replaces enumeration by local search on the graph of vertices. Because the objective is linear and the feasible region convex, local search here is exact: a vertex with no improving neighbour is optimal.

"Honestly" in the title means the unit does not stop at the version that works on textbook examples. It covers the three ways the naive method fails: cycling (degeneracy), exponential running time (Klee–Minty), and floating point. Each of them is reproduced in the lab.

Convention for the whole unit: $\max\{c^\top x : Ax \le b,\ x \ge 0\}$, $A \in \mathbb{Q}^{m\times n}$, with slacks $s = b - Ax \ge 0$.

How to read this deck

The running example is unit 01's polygon, maximizing $2x + y$: its constraints $x + y \le 4$ and $x - y \le 1$ get slacks $s_1$ and $s_2$ (constraints 1 and 2 of unit 01, $x, y \ge 0$, are the sign constraints). Most slides show the idea on it with actual numbers. Where a statement has a proof worth knowing, the slide says why in plain words and the notes carry the argument under Full proof.

02 · overview

The route

1 · the method

Bases and dictionaries. Reduced costs certify optimality. A pivot moves along an edge.

2 · how it fails

Degenerate pivots and cycling (Beale). Anti-cycling (Bland). Exponential paths (Klee–Minty).

3 · why it wins anyway

Two phases. Revised simplex with LU. Steepest-edge pricing. Tolerances, and what they hide.

Lab: pivot, pricing rules, the loop with cycle detection, two phases, and Klee–Minty. Then 1000 LPs against HiGHS.

getting around

Words with a dotted underline are glossary terms: click one for its definition. G lists every term in the unit, N shows the notes. Unit 01's polygon, maximizing $2x + y$, is the running example: a solid underline like that one shows it on hover. Optional extra: interactive figures, linked from the slides they support by a small ▸ chip; skip them freely.

section 1

From vertices to bases

An algebraic handle on a geometric walk.

02 · bases

Bases and basic solutions

Add slacks: $\;Ax + s = b,\; x, s \ge 0$. Write all $n+m$ variables as one vector with matrix $[A \mid I]$, which has rank $m$.

definition

A basis $B$ is a set of $m$ columns of $[A\mid I]$ that are linearly independent. Setting the other ($N$, nonbasic) variables to 0 gives the basic solution $x_B = A_B^{-1} b$.

It is feasible when $x_B \ge 0$.

Basic feasible solutions are exactly the vertices of unit 01. Different bases can give the same vertex: a degenerate vertex.

the starting basis

If $b \ge 0$, the slacks form a feasible basis: $x = 0,\ s = b$. The origin is a vertex.

If some $b_i < 0$ it is not feasible. That is what phase 1 (section 3) is for.

Picture it: "nonbasic" means "held at 0", and a slack at 0 means its constraint is tight. So choosing a basis is choosing which walls to stand on. The next slide does all six choices for the polygon.

02 · bases · example

Bases on the example

$\max 2x + y$ with $x + y + s_1 = 4$, $\ x - y + s_2 = 1$, all four variables $\ge 0$. Here $m = 2$, so a basis is 2 of the 4 columns: $\binom42 = 6$ choices, all independent.

basis	set to 0	solve for the basis	point $(x,y)$
$s_1, s_2$	$x, y$	$s_1 = 4,\ s_2 = 1$	$(0, 0)$	✓
$x, s_1$	$y, s_2$	$x = 1,\ s_1 = 3$	$(1, 0)$	✓
$x, s_2$	$y, s_1$	$x = 4,\ s_2 = -3$	$(4, 0)$	✗
$y, s_1$	$x, s_2$	$y = -1,\ s_1 = 5$	$(0, -1)$	✗
$y, s_2$	$x, s_1$	$y = 4,\ s_2 = 5$	$(0, 4)$	✓
$x, y$	$s_1, s_2$	$x = 2.5,\ y = 1.5$	$(2.5, 1.5)$	✓

Four feasible bases, four corners: the same table as unit 01's "Finding corners by trying pairs of constraints", now in the language of variables. A negative basic value is a violated constraint.

Each corner labelled by its nonbasic variables, the two walls it stands on. $s_1 = 0$ is the line $x + y = 4$; $s_2 = 0$ is $x - y = 1$.

How each row was solved. $\{x, s_1\}$: set $y = s_2 = 0$; the second equation gives $x = 1$, the first $s_1 = 4 - 1 = 3$. $\{x, s_2\}$: set $y = s_1 = 0$; the first gives $x = 4$, the second $s_2 = 1 - 4 = -3 < 0$, which says $(4,0)$ breaks $x - y \le 1$. $\{y, s_1\}$: $x = s_2 = 0$ gives $-y = 1$, so $y = -1$, and $s_1 = 4 - (-1) = 5$. $\{y, s_2\}$: $x = s_1 = 0$ gives $y = 4$ and $s_2 = 1 + 4 = 5$. $\{x, y\}$: both slacks 0, so the two constraints are equalities, adding gives $2x = 5$.

The dictionary between the two languages: in unit 01, a vertex had $n = 2$ tight constraints. Here the sign constraints $x \ge 0$, $y \ge 0$ count too, and "constraint tight" becomes "its variable is 0", for a structural variable or a slack alike. The two nonbasic variables are the two tight constraints.

02 · bases

Reduced costs: the optimality certificate

Solve the constraints for the basic variables and substitute into the objective:

$$x_B = A_B^{-1} b - A_B^{-1} A_N\, x_N, \qquad c^\top x = \underbrace{c_B^\top A_B^{-1} b}_{\text{current value}} + \underbrace{(c_N^\top - c_B^\top A_B^{-1} A_N)}_{\bar c_N^\top}\, x_N$$

optimality criterion

If the BFS is feasible and every reduced cost $\bar c_j \le 0$, it is optimal.

why

Any other feasible point has some nonbasic variables above 0, and each one changes the value by $\bar c_j \le 0$ per unit. Nothing can raise it.

If some $\bar c_j > 0$, raising $x_j$ from 0 improves the objective at rate $\bar c_j$. That $j$ may be the entering variable.

the dictionary, on the example

At the origin (basis $s_1, s_2$):

$s_1 = 4 - x - y,\quad s_2 = 1 - x + y$
$z = 0 + \mathbf{2}\,x + \mathbf{1}\,y$

Reduced costs $2$ and $1$ are positive: not optimal, and either may enter.

At $(2.5, 1.5)$ (basis $x, y$):

$z = 6.5 - \tfrac32\, s_1 - \tfrac12\, s_2$

Both negative: raising either slack, $s_1$ or $s_2$ (leaving a wall), only lowers $z$. Optimal, certified by two signs.

The example's dictionary at the optimum, for checking. From $x + y + s_1 = 4$ and $x - y + s_2 = 1$: adding gives $x = \tfrac52 - \tfrac12 s_1 - \tfrac12 s_2$, subtracting gives $y = \tfrac32 - \tfrac12 s_1 + \tfrac12 s_2$. Then $2x + y = 5 - s_1 - s_2 + \tfrac32 - \tfrac12 s_1 + \tfrac12 s_2 = \tfrac{13}{2} - \tfrac32 s_1 - \tfrac12 s_2$. The reference solution's final tableau has exactly $\tfrac32, \tfrac12$ in the slack columns of its objective row (it stores $-\bar c$).

Full proof

Every feasible point satisfies the dictionary's objective equation $c^\top x = c_B^\top A_B^{-1} b + \bar c_N^\top x_N$ (it is the objective with the constraints substituted in) and has $x_N \ge 0$. With $\bar c_N \le 0$ the second term is $\le 0$, so no feasible point beats the current value. $\square$

This is the "dictionary" view (Chvátal's term): each basis rewrites the same LP as "basic variables = constants − combinations of nonbasic ones" and "objective = constant + reduced costs times nonbasic ones". A pivot moves to an equivalent rewriting. The optimality test reads the sign pattern of the objective row.

The quantity $y^\top = c_B^\top A_B^{-1}$ is the vector of simplex multipliers. Reduced costs are $\bar c_j = c_j - y^\top a_j$, and at an optimal basis $y$ is an optimal dual solution. Unit 03 is about exactly this. The criterion "all $\bar c_j \le 0$" says $A^\top y \ge c$ and $y \ge 0$ (reading the slack columns), which is dual feasibility.

Sign convention used by the lab's tableau: the objective row stores $-\bar c$, so optimality is "no negative entries in the objective row". That is the convention of most textbooks that start from the $z - c^\top x = 0$ row.

The criterion is sufficient but not necessary for optimality of the point. At a degenerate optimal vertex some bases can show a positive reduced cost even though the vertex is optimal, because the improving direction is blocked at step length 0. That is why simplex sometimes makes degenerate pivots at an optimal vertex before it can prove optimality.

02 · bases

The ratio test: how far to move

Increase entering $x_j$ to $t \ge 0$ with the other nonbasic variables held at 0. The basic variables change linearly:

$$x_B(t) = \bar b - t\, d, \qquad \bar b = A_B^{-1}b,\ \ d = A_B^{-1} a_j$$

The ratio test. Feasibility needs $\bar b_i - t d_i \ge 0$ for every row, which binds only where $d_i > 0$:

$$t^* = \min_{i:\,d_i > 0} \frac{\bar b_i}{d_i}$$

The row attaining the minimum names the leaving variable: it hits 0 first.

nothing blocks the move

If no $d_i > 0$, then $x_j$ can grow forever, and the LP is unbounded. The ray $x_B = \bar b - t d$, $x_j = t$ is a certificate: feasible for all $t$, with objective growing at rate $\bar c_j > 0$.

degenerate step

If the minimizing $\bar b_i = 0$, then $t^* = 0$. The basis changes but the point does not move: a degenerate pivot.

on the example, at the origin

Enter $x$: $d = (1, 1)$, $\bar b = (4, 1)$. Ratios $4/1$ and $1/1$: $t^* = 1$, $s_2$ leaves. Move to $(1, 0)$.

Enter $y$ instead: $d = (1, -1)$. Only the $s_1$ row counts: $t^* = 4$, move to $(0, 4)$. The $s_2$ row has $d_2 < 0$: $s_2 = 1 + y$ only grows.

Geometrically, the nonbasic variables other than $x_j$ stay at their bound 0, so $n-1$ of the $n$ tight constraints at the current vertex stay tight. The solutions of $n-1$ independent equations form a line, and the move is along the edge of the polyhedron in that line. The ratio test stops at the first new constraint that becomes tight, which is the vertex at the other end of the edge. A pivot is therefore an edge step, and the simplex path is a walk on the graph of the polyhedron.

The example box. At the origin $A_B = I$, so $d$ is just the entering column: $x$ has column $(1, 1)$ and $y$ has $(1, -1)$ in the rows of $x + y \le 4$ and $x - y \le 1$. Entering $x$, both rows shrink: $s_1 = 4 - t$, $s_2 = 1 - t$, and $s_2$ reaches 0 first. Entering $y$, $s_1 = 4 - t$ but $s_2 = 1 + t$. Dantzig's rule picks $x$ (reduced cost 2 against 1), which is the walk on the next slide.

Why only $d_i > 0$: rows with $d_i \le 0$ get larger or stay as $t$ grows, so they never become infeasible. Pivoting on a negative $d_i$ would make the right-hand side of that row negative, which is an infeasible basis. Lab step 2's "think about" asks exactly this.

02 · bases

A walk on the example polygon

$\max\ 2x+y$ over unit 01's polygon. Dantzig's rule: enter $x$ (coefficient 2), then $y$.

Origin. Slack basis. $\bar c = (2, 1)$. Enter $x$. Ratio test: $x-y\le 1$ binds first, at $x=1$.
$(1, 0)$. Now $x = 1 + y - s_2$ and $z = 2 + 3y - 2s_2$. Enter $y$. $x + y \le 4$ binds at $y = 1.5$.
$(2.5, 1.5)$. Both reduced costs $\le 0$: optimal, $z = 6.5$.

Two pivots, and the vertex $(0,4)$ was never visited. Simplex visits only an improving path, not the whole vertex set.

explore interactively

02 · bases

The tableau: pivots as row operations

A second LP, the lab's SMALL: $\max 3x + 2y$, $\ x + y \le 4$, $\ 2x + y \le 6$. Columns $x, y, s_1, s_2 \mid$ rhs. Bottom row: $-\bar c$ and $z$. Boxed: the pivot element.

s₁	1	1	1	0	4
s₂	2	1	0	1	6
z	−3	−2	0	0	0

s₁	0	½	1	−½	1
x	1	½	0	½	3
z	0	−½	0	3⁄2	9

y	0	1	2	−1	2
x	1	0	−1	1	2
z	0	0	1	1	10

Pivot 1. Most negative $-\bar c$ is column $x$. Ratios $4/1,\ 6/2$: row 2 leaves.
Pivot 2. Column $y$. Ratios $1/\tfrac12 = 2,\ 3/\tfrac12 = 6$: row 1 leaves.

End. No negative entries in the bottom row: $x = y = 2$, $z = 10$.
The slack columns of the bottom row, $(1, 1)$, are the dual values. Remember them for unit 03.

This exact LP is SMALL in the lab's tests, and the second tableau is the one test_step1_pivot_makes_a_unit_column_and_updates_basis expects after the first pivot. If your pivot disagrees, compare against these.

The tableau is the dictionary written as a matrix. Row $i$ says "basic variable $i$ + (nonbasic combination) = rhs", and the bottom row says "$z - \bar c^\top x_N = $ current value". A pivot is one step of Gauss–Jordan elimination that makes the entering column a unit vector. That is all the algebra there is.

Everything in the tableau is $A_B^{-1}$ applied to the original data: the body is $A_B^{-1}[A\mid I]$ and the rhs is $A_B^{-1}b$. In particular the slack columns of the body are $A_B^{-1}$. The revised simplex method (section 3) stores only a factorization of $A_B$ and computes the one column and one row a pivot needs.

section 2

How the naive method fails

Twice: by never stopping, and by taking exponentially long to stop.

02 · failure

Degeneracy and cycling

A degenerate pivot has $t^* = 0$: the basis changes and the point stays. The objective does not increase, so the usual termination argument (strict improvement, finitely many bases) fails.

Degenerate pivots in a row can bring the basis back to where it started. That is cycling, and the method then never stops. The smallest classic case is Beale's example, two slides on.

where degeneracy comes from, on the example

Add unit 01's constraint 5, $3x - y \le 6$, with slack $s_5$. It passes through $(2.5, 1.5)$.

Simplex reaches $(1, 0)$ as before and enters $y$. The ratio test now ties: $s_1$ row $\tfrac32$, $s_5$ row $\tfrac32$.

$s_1$ leaves. $s_5$ stays basic, at value 0. The corner $(2.5, 1.5)$ now has three bases, and a pivot among them would have $t^* = 0$.

Here the reduced costs are already $\le 0$, so it stops. Beale's example is built so that it doesn't.

explore interactively

The example box. At $(1,0)$ with basis $\{s_1, x, s_5\}$: $x = 1 + y - s_2$, $s_1 = 3 - 2y + s_2$ and $s_5 = 6 - 3x + y = 3 - 2y + 3s_2$. Raising $y$ empties $s_1$ and $s_5$ both at $y = \tfrac32$. The reference solution breaks the tie by lowest row, so $s_1$ leaves and the final basis is $\{y, x, s_5\}$ with $s_5 = 0$. The other bases of the same point are $\{x, y, s_1\}$ and $\{x, y, s_2\}$, each with one zero-valued basic slack.

Finiteness argument without degeneracy: each pivot strictly increases the objective, so no basis repeats, and there are at most $\binom{n+m}{m}$ bases. With degeneracy the objective can stay flat, and a sequence of distinct bases at the same vertex can close into a loop.

Cycling was considered a theoretical curiosity for decades: early codes rarely cycled because floating-point noise broke the ties. It is not purely theoretical. Highly degenerate combinatorial LPs stall (long runs of degenerate pivots) in practice, and exact-arithmetic solvers, which have no noise to break ties, do cycle without a rule.

The lab detects cycling by remembering bases, which costs memory proportional to the number of pivots. A production code never does this. It prevents cycling instead, by one of the rules two slides on.

02 · failure · example

Beale's cycle, pivot by pivot

Beale, 1955

$\max\ \tfrac34 x_1 - 20x_2 + \tfrac12 x_3 - 6x_4$
$\tfrac14 x_1 - 8x_2 - x_3 + 9x_4 \le 0$
$\tfrac12 x_1 - 12x_2 - \tfrac12 x_3 + 3x_4 \le 0$
$x_3 \le 1$

Two right-hand sides are 0, so the origin is a degenerate vertex: two slacks sit in the basis at value 0.

optimum

$\tfrac54$ at $x = (1, 0, 1, 0)$. Bland's rule reaches it in 6 pivots.

pivot	enters	leaves	basis after
1	$x_1$	$s_1$	$x_1, s_2, s_3$
2	$x_2$	$s_2$	$x_1, x_2, s_3$
3	$x_3$	$x_1$	$x_3, x_2, s_3$
4	$x_4$	$x_2$	$x_3, x_4, s_3$
5	$s_1$	$x_3$	$s_1, x_4, s_3$
6	$s_2$	$x_4$	$s_1, s_2, s_3$

Dantzig's rule, ties to the lowest row. Six pivots, $z$ never moves, and pivot 6 returns to the starting basis. Then it repeats forever.

where Bland escapes: pivot 5

Improving columns: $x_1$ with $\bar c = \tfrac12$ and $s_1$ with $\bar c = 1$. Dantzig takes the larger, $s_1$. Bland takes the lower index, $x_1$. Then $s_3$ leaves at $t^* = \tfrac25$: the first real move, $z = \tfrac15$. One more pivot gives $\tfrac54$.

Every row of the table comes from the lab's reference solution, and your step 3 test reproduces both counts: Cycling after 6 pivots under Dantzig, and the optimum $\tfrac54$ under Bland.

Why the first four pivots agree under both rules: at each of them there is only one improving column, or the most improving one also has the lowest index. The objective rows before pivots 1 to 4 have improving columns $\{x_1, x_3\}$, $\{x_2, x_3\}$, $\{x_3\}$ and $\{x_4, s_1\}$ (in the tableau these are the negative entries), and both rules pick the same one each time. At pivot 5 they differ for the first time.

The ratios in the degenerate pivots are all $0/d_i$: every pivot 1–6 happens in a row whose right-hand side is 0, so the point stays at $x = 0$. Row 3 ($x_3 \le 1$, right-hand side 1) is the only one that could produce a real move, and Dantzig's choices never make it the minimum ratio. At Bland's pivot 5 it is the only row with a positive entry in column $x_1$, which is why the move is real: ratio $1 / \tfrac52 = \tfrac25$.

02 · failure

Bland's rule

theorem (Bland, 1977)

If the entering variable is always the lowest-index improving one, and ties in the ratio test go to the lowest-index basic variable, the simplex method never cycles.

why, in words

In a cycle some variables leave and later come back. Watch the one with the largest index among them. When it left, and when it came back, the rule chose by lowest index both times. Those two choices say opposite things about the same reduced costs, so a cycle can't close.

full proof in the notes (N)

On Beale's example it breaks the loop at pivot 5, by preferring $x_1$ over $s_1$.

Other cures (anti-cycling rules): the lexicographic ratio test (Dantzig, Orden, Wolfe 1955); perturbation of $b$, symbolic or numerical.

The price: Bland's rule ignores how much a column improves, so it is usually slow. Production codes use good pricing plus perturbation, and fall back to anti-cycling measures only when they detect stalling.

Full proof, sketched

Suppose Bland's rule cycles, and let $x_t$ be the largest-index variable that enters and leaves during the cycle. Compare the dictionary where $x_t$ leaves (with $x_s$ entering, $s < t$) with the one where $x_t$ re-enters. Reduced costs from the second, applied to the move of the first, give an identity whose terms are all $\le 0$ by the index choices, except one that must be $> 0$. Contradiction.

The full proof is one page in Chvátal ch. 3 or Bertsimas & Tsitsiklis §3.4 (they use the lexicographic rule and leave Bland as an exercise with hints). The key identity: in the later dictionary the objective is $z = z_0 + \sum_j \bar c^{*}_j x_j$. The earlier move increases $x_s$ by $t$ and changes basic variables linearly. Since both dictionaries describe the same function, the coefficients of $t$ must agree, and the sign constraints from "lowest index" choices at both pivots make them disagree.

Perturbation in practice: add tiny random amounts to the right-hand side (primal simplex) or to the costs (dual simplex) so that ties essentially never happen, solve, then remove the perturbation and clean up with a few more pivots. Real dual simplex codes, HiGHS among them, do this, which is part of why they barely ever cycle.

Lab check: in then, Bland's rule takes 465 pivots on the $n=12$ Klee–Minty cube, against Dantzig's 4095. Bland is not a speed rule; it just happens to do better on an instance built to defeat Dantzig.

02 · failure

Klee–Minty: exponentially many pivots

The Klee–Minty cube: a squashed $n$-cube (Klee & Minty, 1972). In Chvátal's form:

$$\max \sum_{j=1}^{n} 10^{n-j} x_j \quad\text{s.t.}\quad 2\sum_{jFrom the origin, Dantzig's rule visits all $2^n$ vertices: exactly $2^n - 1$ pivots.

The cube is deformed so that each coordinate's "high" and "low" faces are nearly parallel, and the largest-coefficient choice always starts the long way round.

$n$	Dantzig	Bland	HiGHS
4	15	9	1
6	63	25	1
8	255	67	9
10	1 023	177	9
12	4 095	465	9

Reference run of uv run co then 02. HiGHS: primal simplex, presolve off.

The next slide lists the vertices for $n = 2$ and $n = 3$.

Bland's counts on this family satisfy $p_n = p_{n-1} + p_{n-2} + 1$, a Fibonacci recurrence, so they grow like $\varphi^n \approx 1.618^n$. That is exponential, just slower. For essentially every deterministic pivoting rule that has been proposed (Dantzig, Bland, steepest edge, greatest improvement, shadow vertex), someone has built a Klee–Minty-style family that forces exponentially many pivots. For randomized rules the best known upper bounds are subexponential (Kalai 1992; Matoušek, Sharir & Welzl), and Friedmann, Hansen & Zwick (2011) proved subexponential lower bounds for some of them.

Why HiGHS needs so few: pricing that measures improvement relative to edge length (next section), and a different starting basis ("crash"), neither of which falls for a construction aimed at Dantzig's rule. Don't read "9 iterations" as "HiGHS is polynomial". Read it as "the lower bound is for one rule".

A related open question: the polynomial Hirsch conjecture, whether the diameter of the vertex graph of every polytope is polynomial in $m$ and $n$. The original Hirsch conjecture (diameter $\le m - n$) was disproved by Santos (2012). If the polynomial version is false, no pivoting rule at all can be polynomial.

02 · failure · example

Klee–Minty for n = 2 and 3, vertex by vertex

$n = 2$: $\max 10x_1 + x_2$, $\ x_1 \le 1$, $\ 20x_1 + x_2 \le 100$. Dantzig visits all 4 corners: 3 pivots. Not to scale horizontally.

pivot	enters	vertex $(x_1, x_2, x_3)$	$z$
0		$(0, 0, 0)$	0
1	$x_1$	$(1, 0, 0)$	100
2	$x_2$	$(1, 80, 0)$	900
3	$s_1$	$(0, 100, 0)$	1 000
4	$x_3$	$(0, 100, 8000)$	9 000
5	$x_1$	$(1, 80, 8200)$	9 100
6	$s_2$	$(1, 0, 9800)$	9 900
7	$s_1$	$(0, 0, 10000)$	10 000

$n = 3$, Dantzig: all $2^3 = 8$ vertices, $z$ rising every time. Rows 0–3 are the $n = 2$ walk on the floor $x_3 = 0$; rows 4–7 walk it backwards on the top.

Bland: $(0,0,0) \to (1,0,0) \to (1,80,0) \to (1,80,8200) \to (1,0,9800) \to (0,0,10000)$, 5 pivots. It climbs to the top one step earlier.

explore interactively

Why Dantzig takes the long way. At the origin of the $n = 2$ cube the reduced costs are $10$ for $x_1$ and $1$ for $x_2$, so $x_1$ enters, although the optimum $(0, 100)$ has $x_1 = 0$. From $(1, 80)$ the dictionary says $z = 90 + 10\,s_1 - s_2$: raising the slack $s_1$, i.e. walking $x_1$ back down to 0, is what improves.

The recursive shape. For $n = 3$ the path does the whole $n = 2$ walk with $x_3 = 0$ (4 vertices), steps up in $x_3$ once, then does the $n = 2$ walk in reverse order on the top face (4 more). So $p_n = 2p_{n-1} + 1$ with $p_1 = 1$, giving $2^n - 1$. The objective weights $10^{n-j}$ and right-hand sides $100^{i-1}$ are chosen to make each level's gain dwarf the whole walk below it, so every step still improves.

All vertices and counts on this slide are from the lab's reference solution (klee_minty(2), klee_minty(3)) run under both rules.

02 · failure

So why is simplex the workhorse?

in practice

On real LPs the number of iterations is typically a small multiple of the number of constraints. Worst cases are artificial.

in theory, eventually

Average case (Borgwardt 1982): polynomial expected pivots for the shadow-vertex rule on a random model.

Smoothed analysis (Spielman & Teng 2004): add small Gaussian noise to any LP, and shadow-vertex simplex takes polynomial expected time.

And a structural reason, which is the one that matters for this course: simplex warm-starts. Change the LP slightly and the old basis is nearly optimal. Branch-and-bound (unit 07) and cutting planes (unit 08) re-solve thousands of slightly changed LPs.

section 3

Starting, and what real codes do

Two phases, the revised method, better pricing, and tolerances.

02 · practice

Getting started: two phases

If some $b_i < 0$, the slack basis is infeasible. Phase 1: for each such constraint, negate it and add an artificial $a_i \ge 0$ with coefficient $+1$, then solve

$$\max\ -\textstyle\sum_i a_i$$

starting from the basis of slacks and artificials, which is feasible by construction.

Optimum $< 0$: the original LP is infeasible. The phase-1 duals give a Farkas certificate, as in unit 01.
Optimum $= 0$: pivot remaining artificials out of the basis, drop them, restore the true objective. Phase 2 continues from that feasible basis.

big-M, and why not

Alternative: one phase with objective $c^\top x - M\sum a_i$ for a huge $M$. It is simple to state and numerically poor, because $M$ mixes magnitudes in the reduced costs. The same disease comes back in unit 05 as big-M constraints.

redundant rows

An artificial stuck in the basis at 0 whose row has no nonzero real entry means that row is a combination of the others. Delete it. The lab's step 4 test includes one.

The phase-1 objective row must be "priced out" before it is a valid dictionary. The artificials are basic, so their reduced costs must be zero: subtract each artificial's row from the objective row. Forgetting this is the most common phase-1 bug, and hint rung 2 of step 4 is about it.

Connection to unit 01: phase 1 is an LP whose optimum is 0 exactly when the original system is feasible. When the optimum is negative, the optimal dual solution of phase 1 is a Farkas vector for the original system. So LP software proves infeasibility with the same object you built by Fourier–Motzkin, obtained much more cheaply.

Real codes start differently. A "crash" procedure picks a starting basis with as many structural variables as possible, and the dual simplex method starts from a dual feasible basis instead, which is often easier to find.

02 · practice · example

Phase 1, on the example

Add $x + y \ge 1$ to the polygon. The origin breaks it, so the slack basis is infeasible. As a $\le$ constraint: $-x - y \le -1$, a negative $b$.

Set up. Negate and add an artificial: $x + y - s_3 + a_1 = 1$. Start from $s_1 = 4,\ s_2 = 1,\ a_1 = 1$ and maximize $-a_1$, now $-1$.

pivot	enters	leaves	at $(x,y)$	what happened
1	$x$	$s_2$	$(1, 0)$	ratios $4, 1, 1$: tie, lowest row. $a_1 = 0$ now
2	$y$	$a_1$	$(1, 0)$	degenerate: $a_1$ was already 0
phase 1 ends at value 0: feasible. Drop $a_1$, restore $2x + y$, price out: $z = 2 + \tfrac32 s_3 + \dots$
3	$s_3$	$s_1$	$(2.5, 1.5)$	phase 2, optimal at $z = 6.5$

Three pivots in all. Phase 1 found a corner, $(1, 0)$, not the optimum; phase 2 walked from there. Had the constraint been $x + y \ge 5$, phase 1 would end below 0: infeasible.

Shaded corner: cut off by $x + y \ge 1$. Ring: the infeasible start. Phase 1 moves to $(1,0)$, phase 2 on to the optimum.

The phase-1 objective row, priced out. The objective is $\max -a_1$, which as a tableau row has $+1$ under $a_1$. But $a_1$ is basic, so its entry must be 0: subtract row 3, $(1, 1, 0, 0, -1, 1 \mid 1)$ in the columns $x, y, s_1, s_2, s_3, a_1$. The priced-out row is $(-1, -1, 0, 0, 1, 0 \mid -1)$: phase-1 value $-1$, and $x$ and $y$ both improving.

Pivot 1. Dantzig ties between $x$ and $y$ and takes the lower index, $x$. Column $x$ is $(1, 1, 1)$, right-hand sides $(4, 1, 1)$: ratios $4, 1, 1$. The tie goes to the lower row, $s_2$. After the pivot $x = 1$ and $a_1 = 0$: the point $(1, 0)$ satisfies $x + y \ge 1$ with equality, and the phase-1 value is already 0.

Pivot 2. The objective row still has a negative entry under $y$ (the row is $(0, -2, 0, 1, 1, 0 \mid 0)$), so the loop continues. Column $y$ has positive entries in the $s_1$ row (ratio $3/2$) and the $a_1$ row (ratio $0/2 = 0$): $a_1$ leaves at step 0. This is the "drive the artificials out" step happening on its own, as a degenerate pivot.

Phase 2. Basis $\{s_1, x, y\}$ at $(1, 0)$ with $y = 0$ basic. Restoring $2x + y$ and pricing out gives the row $(0, 0, 0, \tfrac12, -\tfrac32 \mid 2)$ over $x, y, s_1, s_2, s_3$: value 2, and $s_3$ improving at rate $\tfrac32$ (moving away from the wall $x + y = 1$). Column $s_3$ is positive only in the $s_1$ row, ratio $3/1$, so $s_1$ leaves and the point moves to $(2.5, 1.5)$, $z = \tfrac{13}{2}$.

All pivots and rows are from the reference two_phase with Dantzig's rule; Bland's rule gives the same three pivots here.

02 · practice

The revised simplex method

The revised simplex method starts from a complaint. The tableau stores $A_B^{-1}[A \mid I]$, which is dense, $m \times (n+m)$, and is updated in full at every pivot. A pivot actually needs only:

$y^\top = c_B^\top A_B^{-1}$: one solve with $A_B^\top$ (BTRAN);
reduced costs $\bar c_j = c_j - y^\top a_j$ for the candidates (pricing);
$d = A_B^{-1} a_j$ for the chosen column: one solve (FTRAN);
the ratio test, then an update of the factorization of $A_B$.

in a real solver

$A_B$ is kept as sparse LU factors (Markowitz ordering), not an inverse.
Each pivot updates the factors (Forrest–Tomlin), and periodically they are refactorized from scratch to control fill-in and error growth.
Real LPs are very sparse, so solves cost far less than $m^2$.

02 · practice

Pricing: which column enters

Dantzig: the largest $\bar c_j$. It measures improvement per unit of $x_j$, so it depends on the units $x_j$ is measured in.

Steepest edge (Goldfarb & Reid 1977): the largest improvement per unit of distance along the edge,

$$\max_j \frac{\bar c_j}{\|\eta_j\|}, \qquad \eta_j = \begin{pmatrix} -A_B^{-1} a_j \\ e_j \end{pmatrix}$$

where $\eta_j$ is the edge direction in the space of all variables.

on the example: change the units of one variable

Write $y = 10y'$. Column of $y'$: $(10, -10)$, cost $10$. At the origin:

Dantzig: $x$ scores $2$, $y'$ scores $10$ → picks $y'$.
Steepest edge: $x$ scores $2/\sqrt3 = 1.15$, $y'$ scores $10/\sqrt{201} = 0.71$ → still picks $x$, as it did before rescaling.

what it buys

Typically far fewer pivots, and much less sensitivity to how variables are scaled. (The one-sentence answer the checkpoint asks for.)

what it costs

The norms $\|\eta_j\|$ must be kept current, which takes an extra solve per pivot with clever update formulas. Devex (Harris 1973) approximates them cheaply.

The example box. At the origin $A_B = I$, so $\eta_j$ is the entering column negated, with a 1 appended for $x_j$ itself. For $x$: $\eta = (-1, -1, 1)$, norm $\sqrt3$. For $y'$: $\eta = (-10, 10, 1)$, norm $\sqrt{201} \approx 14.18$. Before rescaling $y$ had $\eta = (-1, 1, 1)$ and score $1/\sqrt3 = 0.58$, so steepest edge picked $x$ then too. Rescaling a variable is a change of units, not of geometry, and only Dantzig's choice moved.

Why scaling fools Dantzig: substitute $x_j = 1000\,x'_j$. The column and the cost of $x'_j$ are multiplied by 1000, so $\bar c'_j = 1000\,\bar c_j$ and Dantzig now strongly prefers $x'_j$, although the geometry is unchanged. Steepest edge divides by the edge length, which scales with the column, and so it is far less affected.

Empirically, dual steepest edge in the dual simplex is the default in most commercial and open codes for good reason: on benchmark sets it reduces iteration counts substantially compared with Dantzig pricing, enough to more than pay for the extra work per iteration. Forrest & Goldfarb (1992), "Steepest-edge simplex algorithms for linear programming", is the classic study.

02 · practice

Tolerances, and what they hide

Floating-point simplex needs tolerances: a reduced cost counts as negative only below $-10^{-7}$, a pivot element as usable only above some threshold, a constraint as satisfied within $10^{-7}$ or so.

The lab's then solves $Hx = H\mathbf{1}$ with $H$ the Hilbert matrix. The only solution is $x = \mathbf 1$, objective $n$:

$n$	cond $H$	floats: $\max\|x-1\|$	HiGHS: $\max\|x-1\|$
6	1.5e7	1.3e-10	4.4e-10
7	4.8e8	3.9e-9	1.2
8	1.5e10	1.3	1.2

the lesson

At $n = 8$ both report objective $\approx 8.000$, so the value looks right. The point is wrong by more than 1 in some coordinate.

A wrong $x$ at the right value satisfies every constraint to within tolerance. From the solver's side nothing failed.

The Hilbert matrix is the standard ill-conditioned example: its condition number grows like $e^{3.5n}$. Once the condition number times machine epsilon ($\approx 10^{-16}$) reaches the feasibility tolerance ($\approx 10^{-7}$), points far from the true solution satisfy the constraints "within tolerance". Here that happens at $n \approx 7$–$8$.

Real LPs are rarely this badly conditioned, but badly scaled models (big-M constraints with $M = 10^6$ next to coefficients of $10^{-3}$) create the same effect. Solvers scale the model internally, and HiGHS and Gurobi both report warnings about coefficient ranges. Unit 27 comes back to modelling habits that avoid this.

Exact rational LP solvers exist (QSopt_ex, SoPlex in exact mode), and they are what exact verification of MIP results relies on. They typically solve in floating point first and then repair and verify the basis in rational arithmetic.

02 · lab

Lab 02 — what you build

about 3 hours · uv run co lab 02 · exact Fractions

Tableau and pivot, the Gauss–Jordan step.
Pricing and ratio test under Dantzig's and Bland's rules, ties included.
The loop with cycle detection. Agrees with HiGHS, cycles on Beale, and Bland fixes it.
Two phases for any $b$, with redundant rows and infeasibility.
Klee–Minty, and exactly $2^n - 1$ pivots under Dantzig's rule.

uv run co test 02
uv run co then 02     # 1000 LPs vs HiGHS · Klee–Minty table · floats on the Hilbert system

02 · checkpoint

Done when

checkpoint

All five lab steps are ✓.
Your simplex agrees with HiGHS on 1000 random LPs, and you can exhibit an instance where it cycles without Bland's rule.
You can say in one sentence what steepest-edge pricing buys over Dantzig's rule.

Reading

Bertsimas & Tsitsiklis, ch. 3 (§3.1–3.7).
Chvátal, Linear Programming, ch. 3–4 and 8.
Koberstein (2005), thesis, ch. 1–3, for real implementations (optional).

03 Duality: the bottom row of your final tableau, read as a solution to a different LP. 04 (branch): the polynomial-time alternatives.

Simplex, honestly

The route

From vertices to bases

Bases and basic solutions

Bases on the example

Reduced costs: the optimality certificate

The ratio test: how far to move

A walk on the example polygon

The tableau: pivots as row operations

How the naive method fails

Degeneracy and cycling

Beale's cycle, pivot by pivot

Bland's rule

Klee–Minty: exponentially many pivots

Klee–Minty for n = 2 and 3, vertex by vertex

So why is simplex the workhorse?

Starting, and what real codes do

Getting started: two phases

Phase 1, on the example

The revised simplex method

Pricing: which column enters

Tolerances, and what they hide

Lab 02 — what you build

Done when

Reading

Next