Part I · trunk A · linear programming · spine

Polyhedra, cones, and Farkas' lemma

The shape of a feasible region, three names for its corners, and a short proof that a linear system has no solution.

needs 00~2 h slides + 2½ h lablab: uv run co lab 01

Why the course starts here

Integer programming, Part II, is the study of what happens when you take the LP relaxation seriously: the LP you get by dropping integrality. To take it seriously you need to know what its feasible region looks like (a polyhedron), where optima live (at corners), and what kind of object certifies a statement about it. This unit covers those three things. It is geometry and one lemma, with no algorithm for optimizing yet: that is unit 02.

Everything is in $\mathbb{R}^n$ with rational data. Rationality matters exactly once, when we ask how large a certificate is.

Notation

$A \in \mathbb{Q}^{m\times n}$ with rows $a_1,\dots,a_m$, $b \in \mathbb{Q}^m$. Vectors are columns. $a_i x$ means the inner product of row $a_i$ with $x$. Inequalities between vectors hold componentwise.

How to read this deck

The slides aim at the ideas and how to use them. Where a statement has a proof worth knowing, the slide gives the idea in a sentence or two and the notes carry the full argument under Full proof. Skip those on a first pass; nothing in the lab depends on them.

01 · overview

The route

The destination is Farkas' lemma. Unit 03 builds LP duality on it, and through duality it reaches most of the course.

The surprise: the constructive proof of Farkas is a bookkeeping exercise on Fourier–Motzkin elimination. You will do it in lab step 5.

getting around

Words with a dotted underline are glossary terms: click one for its definition. G lists every term in the unit, N shows the notes, where the full proofs live. Optional extra: interactive figures, linked from the slides they support by a small ▸ chip; skip them freely.

section 1

Polyhedra

Finitely many linear inequalities, and the shapes they cut out.

01 · polyhedra

Halfspaces, polyhedra, polytopes

definitions

A halfspace is $\{x \in \mathbb{R}^n : a x \le \beta\}$ with $a \ne 0$: one side of a line (in 2-D) or a plane. Its boundary $\{a x = \beta\}$ is a hyperplane.

A polyhedron is what finitely many halfspaces have in common: $$P = \{x : Ax \le b\}.$$

A polytope is a polyhedron that doesn't run off to infinity.

Every polyhedron is convex (the segment between two of its points stays inside) and closed. Equalities, $\ge$, and sign constraints all fit this form.

one form to another

$a x = \beta \;\to\; a x \le \beta,\ -a x \le -\beta$

$a x \ge \beta \;\to\; -a x \le -\beta$

free $x_j \;\to\; x_j^+ - x_j^-,\ x_j^\pm \ge 0$

$a x \le \beta \;\to\; a x + s = \beta,\ s \ge 0$ (a slack variable)

Standard form $\{x : Ax = b,\ x \ge 0\}$ is what simplex works in (unit 02). This unit uses $Ax \le b$.

Convexity: if $Ax \le b$ and $Ay \le b$ then $A(\lambda x + (1-\lambda)y) \le \lambda b + (1-\lambda) b = b$. Closedness: each halfspace is the preimage of a closed set under a continuous map.

The conversions change the dimension and the number of constraints, but not the problem. They are not free for algorithms, though. Splitting a free variable into $x^+ - x^-$ doubles a column, and every value of $x_j$ is then represented in infinitely many ways, which is why solvers handle free variables directly. Adding slacks is what turns the inequality view of this unit into the equality view simplex uses.

Beware one convention difference between textbooks: Bertsimas & Tsitsiklis write $Ax \ge b$ as the general form, and many others use $Ax \le b$. Nothing changes except signs, but duality (unit 03) is a place where sign conventions cost an afternoon.

01 · polyhedra · example

The running example

Four constraints in $\mathbb{R}^2$, numbered 1–4 for the rest of the unit (constraint $i$ is row $i$ of $Ax \le b$). Every later slide comes back to this $P$. Underlined like constraint 3: hover to see it.

constraint	$a_i x \le b_i$	at $(1,1)$	at $(3,2)$	at $(2.5,1.5)$
1	$-x \le 0$	$-1$ ✓	$-3$ ✓	$-2.5$ ✓
2	$-y \le 0$	$-1$ ✓	$-2$ ✓	$-1.5$ ✓
3	$x + y \le 4$	$2$ ✓	$5$ ✗	$4$ =
4	$x - y \le 1$	$0$ ✓	$1$ =	$1$ =

Membership is plugging in. $(1,1)$ satisfies every constraint with room to spare: it is inside. $(3,2)$ breaks constraint 3: it is outside. $(2.5,1.5)$ satisfies everything, two constraints with no room at all: it sits on two walls at once.

Hold on to that last observation. "No room at all" is the idea the whole corners section is built on.

$P$ has corners $(0,0)$, $(1,0)$, $(2.5,1.5)$, $(0,4)$. Dashed: the lines where constraints 3 and 4 hold with equality.

explore interactively

01 · polyhedra

Faces

definition

An inequality $c x \le \delta$ is a valid inequality for $P$ if every point of $P$ satisfies it. The face it defines is the part of $P$ where it holds with equality: $F = P \cap \{x : c x = \delta\}$.

Picture pressing a ruler against $P$ from outside: the face is what the ruler touches.

valid inequality	face of the example	kind
$x - y \le 1$	segment $(1,0)$–$(2.5,1.5)$	edge, a facet
$-x + y \le 4$	the single point $(0,4)$	vertex
$x + y \le 5$	nothing: never tight	empty face
$0x + 0y \le 0$	all of $P$	$P$ itself

A 2-D polytope's facets are its edges. In general facets have dimension $\dim P - 1$, and a full-dimensional polyhedron needs one inequality per facet; any other constraint is a redundant inequality.

Thick: the edge from $x - y \le 1$. Large dot: the vertex $(0,4)$, the only point where $-x + y \le 4$ is tight (dotted line).

Check the vertex constraint: $-x + y$ is $0$ at $(0,0)$, $-1$ at $(1,0)$, $-1$ at $(2.5, 1.5)$ and $4$ at $(0,4)$. So $-x + y \le 4$ is valid and touches $P$ at one point. Note that it is not one of the four constraints defining $P$: faces come from any valid inequality, not just the ones you wrote down.

The empty set and $P$ itself are faces, called improper: take $0x \le 1$ (face $\emptyset$) or $0x \le 0$ (face $P$). Some texts exclude them.

The face lattice of a polyhedron (all faces ordered by inclusion) is finite, since each face is obtained by turning a subset of the defining inequalities into equalities, and there are at most $2^m$ subsets. That bound is the first hint of why corners can be enumerated: lab step 3 does it by trying subsets of size $n$.

The claim about minimal descriptions: if $P$ is full-dimensional, every inequality in $Ax \le b$ either defines a facet or can be deleted without changing $P$, and the facet-defining inequalities are unique up to positive scaling. For lower-dimensional $P$ there is freedom to add multiples of the equations. This is Schrijver §8.4, and it is why "how many facets does this polytope have" is the honest measure of how hard it is to describe. Unit 05 comes back to it: the TSP polytope has exponentially many facets, and no small extended formulation.

section 2

Three names for a corner

A geometric one, a convexity one, and an algebraic one, and why they agree.

01 · corners

Tight constraints, and counting them

definition

Constraint $i$ is tight at a point $x$ if $x$ lies on its line: $a_i x = b_i$, no slack. It depends on the point: constraint 3 is tight at $(2.5, 1.5)$ but not at $(1, 1)$. Write $I(x)$ for the set of constraints tight at $x$.

The rank of the tight constraints counts how many different directions those walls face: constraints 3 and 4 give rank 2, and a wall listed twice counts once.

point	tight constraints	rank	where it is
$(1, 1)$	none	0	inside
$(0, 2)$	constraint 1	1	on an edge
$(1.75, 0.75)$	constraint 4	1	on an edge
$(2.5, 1.5)$	constraints 3, 4	2	a corner

The pattern: with rank below $n$ you can still slide along the tight walls. With rank $n$ (here $n = 2$), the walls pin the point down completely. That is what a corner is.

Thick: the directions a point on one wall can still slide in. At the corner there is none left.

explore interactively

Why rank and not simply "how many constraints are tight": a corner can have more tight constraints than $n$ (two slides on, a degenerate corner has three in the plane). And two tight constraints that are multiples of each other, like $x + y \le 4$ written twice, are really one wall. Rank counts independent walls, which is the thing that matters.

The sliding picture is exact. If the tight constraints have rank below $n$, some direction $d \ne 0$ has $a_i d = 0$ for every tight constraint: moving along $d$ keeps them tight. The other constraints have slack, so a small enough move in either direction $\pm d$ stays inside $P$. At $(0,2)$, $d = (0, 1)$ slides up and down the left edge.

You will compute exactly this in the lab: the step 3 tests check that every vertex you return has tight constraints of rank $n$.

01 · corners

Vertex, extreme point, basic feasible solution

Three ways to say what a corner of $P = \{x : Ax \le b\}$ is:

vertex · geometry

Some objective $c$ is maximized over $P$ at $x$ and nowhere else.

A ruler touches $P$ only at $x$.

extreme point · convexity

$x$ is not the midpoint (or any in-between point) of two other points of $P$.

No segment in $P$ passes through $x$.

BFS · algebra

The tight constraints at $x$ have rank $n$.

Enough walls meet to pin $x$ down.

theorem

For $x \in P$ the three are the same thing: a vertex is an extreme point is a basic feasible solution.

The geometric name is what optimization needs. The algebraic one is what a computer can check, and simplex walks on it.

On the example. $(2.5, 1.5)$ is all three. Vertex: $c = (2, 0)$ is maximized there and nowhere else (next slide). Extreme: any segment through it would have to leave $P$ on one side. BFS: constraints 3 and 4 are tight, rank 2. $(1.75, 0.75)$ is none of the three: it is the midpoint of $(1,0)$ and $(2.5,1.5)$, and only one constraint is tight.

Terminology. "Basic feasible solution" is usually defined in standard form: $x \ge 0$, $Ax = b$, and the columns of $A$ for the positive entries of $x$ are linearly independent. The rank condition here is the same idea in inequality form. Converting one form into the other maps BFSs to BFSs. A basic solution (not necessarily feasible) is a point where $n$ linearly independent constraints are tight, feasible or not. Lab step 3 enumerates basic solutions and throws away the infeasible ones.

01 · corners

Why the three agree

vertex ⇒ extreme

If $x$ were between two other points of $P$, the objective at $x$ would be the average of its values there. One of them would be at least as good as $x$, so $x$ couldn't be the only best point.

extreme ⇒ BFS

If the tight walls had rank below $n$, $x$ could slide a little both ways along them, as on the tight-constraints slide. Then $x$ is the midpoint of the two ends of that move: not extreme.

BFS ⇒ vertex: add up the tight constraints

Take $c$ = the sum of the tight constraints. Every point of $P$ scores at most the sum of their right-hand sides, and only a point where all of them are tight scores exactly that. Rank $n$ means that point is $x$ alone.

full proof in the notes (N)

At $(2.5, 1.5)$ constraints 3 and 4 are tight. Their sum $c = (2,0)$, i.e. maximize $2x$, is best at this corner and nowhere else: check the other corners, they score $0$, $2$ and $0$ against $5$.

The step to remember is the third, because it is constructive. Adding up the tight constraints gives an objective that certifies the point is a vertex. Adding constraints with nonnegative weights to bound an objective is the move behind LP duality, and it appears here first.

Full proof

Vertex ⇒ extreme. Let $c$ make $x$ the unique maximizer. If $x = \lambda y + (1-\lambda) z$ with $y \ne z$ in $P$ and $0 < \lambda < 1$, then one of $y, z$ differs from $x$, so $c y < c x$ or $c z < c x$, and neither exceeds $c x$. Then $c x = \lambda c y + (1-\lambda) c z < c x$, a contradiction.

Extreme ⇒ BFS. If the tight constraints have rank $< n$, pick $d \ne 0$ with $a_i d = 0$ for all $i \in I(x)$. Constraints not tight have slack, so $x \pm \varepsilon d \in P$ for small $\varepsilon > 0$: for each constraint $j \notin I(x)$ with $a_j d \ne 0$ you need $\varepsilon |a_j d| \le b_j - a_j x$, and the minimum of those finitely many positive bounds works. Then $x$ is the midpoint of $x + \varepsilon d$ and $x - \varepsilon d$, a contradiction.

BFS ⇒ vertex. Take $c = \sum_{i \in I(x)} a_i$. For $y \in P$: $c y = \sum_{I} a_i y \le \sum_I b_i = c x$, with equality only if every $i \in I(x)$ is tight at $y$. The tight constraints have rank $n$, so the system $a_i y = b_i,\ i \in I(x)$ has the unique solution $y = x$. $\square$

Corollary, and a count

A BFS is determined by its tight set: it is the unique solution of $a_i y = b_i,\ i \in I(x)$. So there are at most $\binom{m}{n}$ vertices, and they can be found by trying every $n$-subset of constraints. That is the next slide and lab step 3, and it is lab 00's brute force again, over bases instead of 0/1 vectors.

The real maximum is much smaller than $\binom{m}{n}$ but still exponential. By McMullen's upper bound theorem, a polytope in $\mathbb{R}^n$ with $m$ facets has $O(m^{\lfloor n/2 \rfloor})$ vertices, attained by duals of cyclic polytopes. The $n$-cube has $2n$ facets and $2^n$ vertices.

01 · corners · example

Finding corners by trying pairs of constraints

In $\mathbb{R}^2$ a corner needs 2 independent tight constraints. So try every pair: set both to equality, solve, and keep the solution if it satisfies the other constraints.

constraints	solve as equalities	other constraints
1, 2	$(0, 0)$	✓ corner
1, 3	$(0, 4)$	✓ corner
1, 4	$(0, -1)$	✗ constraint 2
2, 3	$(4, 0)$	✗ constraint 4
2, 4	$(1, 0)$	✓ corner
3, 4	$(2.5, 1.5)$	✓ corner

$\binom{4}{2} = 6$ basic solutions, 4 of them feasible: the 4 corners. Each pair of constraints is a basis. This is lab step 3.

one more constraint: a degenerate corner

Add constraint 5: $3x - y \le 6$. It cuts nothing off (check the four corners), but it also passes through $(2.5, 1.5)$.

Now three constraints are tight there, still rank 2. The pairs 3–4, 3–5 and 4–5 all solve to the same point: three bases, one corner.

That corner is a degenerate vertex. Enumerating pairs finds it three times, so the lab must deduplicate, and in unit 02 it is what can make simplex go round in circles.

In $\mathbb{R}^3$ the same thing happens at the apex of a square pyramid: four facets meet where three would do. The lab's pyramid test is this slide in one more dimension.

01 · corners

Not every polyhedron has a corner

$\{(x, y) : y \ge 0,\ y \le 1\}$ is a horizontal strip. Every point can slide left and right forever, so no point is extreme.

theorem

For nonempty $P = \{x : Ax \le b\}$, the following are equivalent:

$P$ has a vertex (it is pointed);
$P$ contains no whole line;
$\operatorname{rank} A = n$.

The strip: its constraints $-y \le 0$ and $y \le 1$ have normals $(0, -1)$ and $(0, 1)$: rank 1 < 2. The direction $(1, 0)$ is invisible to every constraint, and that is the line.

why you rarely notice

Any polyhedron inside $x \ge 0$ is pointed: the constraints $x_j \ge 0$ already have rank $n$. Standard form is pointed by construction. So is the running example, thanks to constraints 1 and 2.

So "the optimum is at a vertex" needs a hypothesis. The fundamental theorem at the end of the unit states it.

The idea in one line: a line inside $P$ is a direction no constraint can see, and the constraints see every direction exactly when they have rank $n$.

Full proof sketch

The lineality space is $\{d : Ad = 0\}$. If it contains $d \ne 0$, then $x + t d \in P$ for all real $t$: a line. Conversely, a line $x + t d$ in $P$ needs $A d \le 0$ and $-A d \le 0$, so $A d = 0$. And $\{d : Ad = 0\} = \{0\}$ exactly when $\operatorname{rank} A = n$.

For the vertex direction: if $\operatorname{rank} A = n$, start at any $x \in P$. While the tight constraints have rank $< n$, pick $d$ in their null space but not in the null space of $A$, and move along $\pm d$ until a new constraint becomes tight. Each move increases the rank of the tight set, so after at most $n$ moves you are at a BFS. The same walk is how crossover turns an interior-point solution into a basic one (unit 04).

On the example: start at $(1,1)$ with nothing tight, move in direction $(1,0)$ until constraint 4 becomes tight at $(2,1)$, then slide along constraint 4, direction $(1,1)$, until constraint 3 becomes tight at $(2.5, 1.5)$. Two moves, one corner.

01 · corners

Two descriptions: Minkowski–Weyl

theorem (Minkowski 1896, Weyl 1935)

$P$ is a polyhedron if and only if $\;P = \operatorname{conv}(V) + \operatorname{cone}(R)\;$ for finite sets $V, R$: every point is a blend of corners plus a push along some rays.

In particular, a polytope is the convex hull of its vertices, and the convex hull of finitely many points is a polytope.

	H-description	V-description
$n$-cube	$2n$ facets	$2^n$ vertices
$n$-cross-polytope	$2^n$ facets	$2n$ vertices

Either description can be exponentially larger than the other.

definitions

$\operatorname{conv}(V) = \{\sum \lambda_v v : \lambda \ge 0,\ \sum \lambda_v = 1\}$: weighted averages.

$\operatorname{cone}(R) = \{\sum \mu_r r : \mu \ge 0\}$: nonnegative combinations, the conic hull.

For pointed $P$: $V$ = vertices, $R$ = extreme rays of the recession cone $\{d : Ad \le 0\}$.

This is the theorem that lets integer programming exist as a subject. The set of integer points in a polytope is finite, so its convex hull is a polytope by the "if" direction, and therefore has some H-description $\{x : Cx \le d\}$. Optimizing a linear function over the integer points is the same as optimizing it over that polytope, which is an LP. The catch is that $C$ is unknown and typically enormous. Cutting planes (unit 08) try to find the parts of $C$ that matter.

Proof idea, not needed for the lab. H ⇒ V: vertices and extreme rays are finite in number (bases), and every point decomposes. V ⇒ H: write $\operatorname{conv}(V)+\operatorname{cone}(R)$ as the projection of a polyhedron in the variables $(x, \lambda, \mu)$, then eliminate $\lambda$ and $\mu$ with Fourier–Motzkin, the next section's tool. Projections of polyhedra are polyhedra, and that gives the H-description.

Computing one description from the other is what qhull and cddlib do. For polytopes, whether vertex enumeration can be done in time polynomial in input plus output size is still open. For unbounded polyhedra it is NP-hard to decide whether a given list of vertices is complete (Khachiyan, Boros, Borys, Elbassioni & Gurvich, 2008). Your lab compares a $\binom{m}{n}$ enumeration with qhull.

01 · corners · example

Minkowski–Weyl, on two examples

a polytope: blend the corners

$(1, 1)$ in the running example is a weighted average of its corners:

$$(1,1) = 0.5\,(0,0) + 0.4\,(2.5,1.5) + 0.1\,(0,4)$$

Weights are $\ge 0$ and sum to 1. Check: $x = 0.4 \cdot 2.5 = 1$, $y = 0.4 \cdot 1.5 + 0.1 \cdot 4 = 1$.

unbounded: corners plus rays

$Q = \{x \ge 0,\ y \ge 0,\ x + y \ge 1\}$ has corners $(1,0)$, $(0,1)$ and runs off along the rays $(1,0)$ and $(0,1)$.

$$(2, 3) = \underbrace{(1, 0)}_{\text{a corner}} + \underbrace{1\,(1,0) + 3\,(0,1)}_{\text{a push along the rays}}$$

section 3

Fourier–Motzkin elimination

Gaussian elimination for inequalities. It is correct and complete, and its output explodes.

01 · fourier–motzkin

Eliminating one variable

Goal: the projection $\{x' : \exists x_k,\ (x', x_k) \in P\}$, the shadow of $P$ with $x_k$ dropped, described without $x_k$.

the idea

Fix the other variables. Every constraint that mentions $x_k$ is either a lower bound or an upper bound on it. A value of $x_k$ exists exactly when every lower bound is at most every upper bound.

So write down "lower ≤ upper" for each pair, and $x_k$ is gone.

tiny example

$x \ge 0,\quad x \le 3,\quad x \le 5 - y$

Lower: $0$. Upper: $3$ and $5 - y$. Pairs: $0 \le 3$ (always true) and $0 \le 5 - y$. So some $x$ exists iff $y \le 5$.

the rule (Fourier 1826, Motzkin 1936)

Sort the constraints by the sign of their $x_k$ coefficient: $Z$ (zero: keep as is), $U$ (positive: upper bounds), $L$ (negative: lower bounds).

For each $p \in U$, $q \in L$, add the two constraints with positive multipliers chosen so $x_k$ cancels: $$(-a_{qk})\,(a_p x \le b_p) \;+\; a_{pk}\,(a_q x \le b_q)$$

The projection is: the $Z$ constraints, plus one combined constraint per pair.

Both multipliers are positive. Inequalities survive nonnegative combinations and nothing else, and that is the whole difference from Gaussian elimination.

The name for the whole procedure is Fourier–Motzkin elimination, FM for short. Repeating it variable by variable is the next-but-one slide.

Full proof

Fix $x'$. The feasible values of $x_k$ form the interval $\max_{q\in L} \ell_q(x') \le x_k \le \min_{p \in U} u_p(x')$, intersected with the requirement that the $Z$ constraints hold. That interval is nonempty exactly when every lower bound is at most every upper bound: $\ell_q(x') \le u_p(x')$ for all pairs. Clear the denominators. Multiplying by $a_{pk} > 0$ and by $-a_{qk} > 0$ preserves the direction, and you get the combined constraint $(-a_{qk})\,(a_p x) + a_{pk}\,(a_q x) \le (-a_{qk})\, b_p + a_{pk}\, b_q$, in which $x_k$ cancels.

If $U$ or $L$ is empty there are no pairs, the interval is unbounded on one side and nonempty, and only $Z$ survives. The lab's hint ladder calls this out because it is the case people special-case wrongly.

Integer data stays integer. The multipliers are the coefficients themselves, so no division ever happens. That is why the lab can do everything in exact integers and divide by a gcd only to tidy up.

Count. The new system has $|Z| + |U|\cdot|L|$ constraints. With $m$ constraints split evenly that is about $m^2/4$.

Fourier described the method in 1826. Motzkin rediscovered it in his 1936 thesis, and Dines had it in 1919; the name "Fourier–Motzkin" stuck.

01 · fourier–motzkin

On the example: eliminate $x$

constraint	as a bound on $x$	class
1: $-x \le 0$	$x \ge 0$	$L$
2: $-y \le 0$	no $x$	$Z$
3: $x + y \le 4$	$x \le 4 - y$	$U$
4: $x - y \le 1$	$x \le 1 + y$	$U$

Pairs, with multipliers $1$ and $1$:

$(x+y \le 4) + (-x \le 0) \;\Rightarrow\; y \le 4$ — "$0 \le 4 - y$"

$(x-y \le 1) + (-x \le 0) \;\Rightarrow\; -y \le 1$ — redundant

Keep $Z$: $-y \le 0$. Projection: $\;0 \le y \le 4$. ✓

Projecting along $x$: every $y \in [0, 4]$ has some $x$ that puts $(x,y)$ in $P$.

explore interactively

Check the redundant constraint: $-y \le 1$ says $y \ge -1$, which $-y \le 0$ already implies. FM has no idea it is redundant. Detecting redundancy exactly means solving an LP per constraint (is $\max\,(-y)$ over the other constraints at most $1$?), so FM carries every such constraint forward and pairs it again at the next step. That is where the explosion comes from.

Two refinements keep most of the junk out. Chernikov's rule: track which original constraints each derived constraint came from, and discard a derived constraint whose history is a superset of another's. After $k$ eliminations, a constraint built from more than $k+1$ original constraints is redundant. Imbert's acceleration theorems sharpen this. With them the growth is singly exponential, which is the best possible, since projections can have exponentially many facets.

The lab doesn't implement these. It deduplicates normalized constraints, and then shows you how far that is from enough.

01 · fourier–motzkin

Deciding feasibility, and the explosion

algorithm

Eliminate $x_1, \dots, x_n$ in turn. Every remaining constraint reads $0 \le c$.

$Ax \le b$ is feasible $\iff$ every remaining $c \ge 0$.

On the example: eliminating $y$ from $0 \le y \le 4$ pairs $y \le 4$ with $-y \le 0$ and leaves $0 \le 4$. True, so $P$ is nonempty.

A complete decision procedure for linear feasibility, in ten lines of Python, needing no LP theory.

Nobody uses it for LP. The table shows why: the FM blow-up.

$n=6$, 14 random cuts + box. Pairwise worst case: $m \to m^2/4$ per step.

eliminated	FM constraints	true facets
0	26	—
1	64	29
2	601	38
3	70 996	39
4	~10⁹ pairs	—

Reference run of uv run co then 01. True facet counts from qhull.

Iterating $m \mapsto m^2/4$ gives roughly $4\,(m/4)^{2^k}$ after $k$ steps: doubly exponential in the number of eliminated variables. Even with perfect redundancy removal the output can be exponential, so the problem is not only bad bookkeeping.

Look at the table's third column. The projections themselves stay small (under 40 facets) while FM's description of them grows past seventy thousand constraints. The gap between "the object" and "a description FM can produce" is enormous.

FM is still useful. It proves theorems (projection of a polyhedron is a polyhedron; Minkowski–Weyl; Farkas), and it is used in practice where variables are few: in compilers, for loop-bound analysis in the polyhedral model, and in decision procedures for linear arithmetic inside SMT solvers (unit 21), where it is one of the options alongside simplex.

section 4

Farkas' lemma

A system with no solution always has a short, checkable reason why.

01 · farkas

A certificate of infeasibility

definition

$y \in \mathbb{R}^m$ is a Farkas certificate for $Ax \le b$ if $$y \ge 0, \qquad y^\top A = 0, \qquad y^\top b < 0.$$

In words: weights for the constraints that make every variable cancel and leave a negative right-hand side.

why it proves there is no solution

Adding true inequalities with weights $\ge 0$ gives a true inequality. These weights produce $0 \le (\text{something negative})$, which is false. So the constraints can't all be true at once.

example

$x \le 1$ and $-x \le -2$ (that is, $x \ge 2$).

Take $y = (1, 1)$ and add: $\;0 \le -1$. No $x$ can exist.

a two-variable one

$x + y \le 2,\quad -x \le -1,\quad -y \le -2$

$y = (1, 1, 1)$: $\;(x + y) - x - y \le 2 - 1 - 2$, i.e. $0 \le -1$.

Checking a certificate needs one matrix–vector product. No solver and no trust: lab step 4.

All three conditions matter. $y=0$ gives $0 \le 0$, and a negative $y_i$ flips an inequality.

The one-line algebra, for the record: if $Ax \le b$ and $y \ge 0$, then $0 = (y^\top A)\,x = y^\top (Ax) \le y^\top b < 0$. $\square$

This is the easy direction. The content of Farkas' lemma is the converse: whenever the system is infeasible, such a $y$ exists. There is never an infeasible linear system whose infeasibility cannot be shown by adding up its constraints.

Contrast with unit 00. For vertex cover, a short lower-bound certificate sometimes did not exist (the triangle). For linear systems over the reals, the certificate always exists. That difference, integers versus reals, is roughly the difference between NP-hard and polynomial.

In the language of unit 00: linear feasibility has short "yes" certificates (a solution $x$) and short "no" certificates (a Farkas $y$). So it is in NP ∩ coNP. The size argument is in the notes of "Farkas, constructively".

01 · farkas

Farkas' lemma

theorem (Farkas, 1902) · inequality form

Exactly one of these holds:

$\quad$(I) $\;\exists\, x:\ Ax \le b$ $\qquad\qquad$ (II) $\;\exists\, y \ge 0:\ y^\top A = 0,\ y^\top b < 0$

Either the system has a solution, or a certificate shows it has none. Never both, never neither.

equivalent · standard form

Exactly one of these holds:

$\quad$(I′) $\;\exists\, x \ge 0:\ Ax = b$ $\qquad\qquad$ (II′) $\;\exists\, y:\ A^\top y \ge 0,\ b^\top y < 0$

Either $b$ is a nonnegative combination of $A$'s columns, or some direction $y$ sees every column as nonnegative and $b$ as negative.

"Not both" is the previous slide. "At least one" is the real theorem. Two ways to see it: by computation (run FM and watch), and by geometry (separate a point from a cone). Both are next.

01 · farkas

Farkas, constructively

why a certificate always exists, by running FM

1. Every constraint FM produces is a nonnegative combination of the original constraints: it only ever adds constraints with positive multipliers.

2. Run FM until no variable is left. Every final constraint reads $0 \le c$, and each carries the weights $y \ge 0$ that built it.

3. If some $c < 0$, its weights are a certificate. If every $c \ge 0$, FM says the system is feasible. One of the two always happens.

example

$x \le 1$, $-x \le -2$. Eliminating $x$ pairs the two constraints with multipliers $1, 1$: $\;0 \le -1$, built from $y = (1, 1)$. That is the certificate.

in code · lab step 5

To remember the weights, append the unit vector $e_i$ to constraint $i$ as $m$ extra columns that are never eliminated. FM combines constraints linearly, so the extra columns accumulate exactly $y$.

When a constraint with zero $x$-part and $b < 0$ appears, its tail is the certificate: the tail trick.

With tails, the example's constraints are $(1 \mid 1, 0) \le 1$ and $(-1 \mid 0, 1) \le -2$, and their sum is $(0 \mid 1, 1) \le -1$.

The lab asks you to see the proof and the algorithm as the same thing. That habit is useful throughout the course. Many existence theorems in optimization come with an algorithm that finds the object, and many algorithms, run to completion, prove a theorem.

Full proof

Invariant. Every constraint FM produces is $y^\top(A, b)$ for some $y \ge 0$. By induction: the original constraint $i$ is $e_i^\top(A, b)$, and each new constraint is a positive combination of two constraints that already have this form, so its $y$ is the same positive combination of theirs. Termination. After eliminating all $n$ variables each constraint is $0 \le c$ with $y^\top A = 0$ and $y^\top b = c$. If some $c < 0$, that $y$ satisfies (II). If every $c \ge 0$, FM's correctness (each projection is exact) lifts a solution back up one variable at a time, so (I) holds. $\square$

How large is the certificate?

FM's multipliers can have many bits. But if a certificate exists, then $\{y \ge 0 : y^\top A = 0,\ y^\top b = -1\}$ is a nonempty polyhedron. It is pointed, because it lies in $y \ge 0$, so it has a vertex. A vertex solves a square subsystem, so by Cramer's rule its entries have bit size polynomial in that of $(A, b)$. So short certificates always exist, and linear feasibility is in coNP. Together with the solution itself as a "yes" certificate, which also has polynomial size by the same vertex argument, this placed LP in NP ∩ coNP decades before Khachiyan put it in P (unit 04).

01 · farkas

Farkas, geometrically

$b \notin \operatorname{cone}(a_1, a_2)$, so a line through the origin separates them. Its normal $y$ has $y \cdot a_j \ge 0$ and $y \cdot b < 0$.

Read the standard form as a picture. With $a_j$ the columns of $A$:

(I′) says you can reach $b$ by stretching and adding the $a_j$: $b \in \operatorname{cone}(a_1, \dots, a_n)$.

(II′) says some line (a hyperplane $\{z : y^\top z = 0\}$) has every $a_j$ on one side and $b$ strictly on the other.

so farkas is

a point outside a cone built from finitely many arrows can be cut off from it by a separating hyperplane.

Here $a_1=(2,1)$, $a_2=(1,3)$, $b=(-1,2)$, $y=(3,-1)$: $y\cdot a_1 = 5$, $y \cdot a_2 = 0$, $y\cdot b = -5$.

Full proof, geometric

A finitely generated cone $K$ is closed and convex. If $b \notin K$, let $p$ be the point of $K$ nearest to $b$. The vector $y = p - b$ then satisfies $y^\top z \ge y^\top p$ for all $z \in K$, with $y^\top p = 0$ because $K$ is a cone, and $y^\top b < y^\top p$. That is (II′).

The subtle step is closedness. It is obvious for a polyhedral cone, but a finitely generated cone is only known to be polyhedral after Minkowski–Weyl, or after Carathéodory's theorem, which writes it as a finite union of cones on linearly independent generators, each of which is closed. For a general convex cone that is not closed the statement fails, which is why duality in conic programming (unit 25, SDP) needs extra "constraint qualification" hypotheses that LP never does.

Keep both pictures. FM is the one you compute with. Separation is the one that generalizes to convex optimization (unit 29): KKT conditions are what this picture becomes when the cone is generated by gradients of the active constraints.

01 · farkas

Why this is the root of the trunk

LP duality (unit 03). "No $x$ has $c^\top x > z$" is an infeasibility statement about $Ax \le b,\ -c^\top x \le -z$ with a small perturbation. Its Farkas certificate is a dual solution. Strong duality is Farkas applied once.

Solvers return certificates. Gurobi: set InfUnbdInfo=1 and read FarkasDual. HiGHS: getDualRay. An infeasible model comes with the $y$ you are building.

Cutting planes (unit 08). A Chvátal–Gomory cut is a nonnegative combination of constraints whose coefficients happen to be integers, with the right side rounded down. It is Farkas plus rounding.

The KKT conditions (unit 29) are the separation picture for curved sets.

Solvers compute in floating point, so their $y$ satisfies $y^\top A \approx 0$. Your integer $y$ satisfies it exactly. Exact verification of solver claims is a live research area.

The duality sketch, to be done properly in unit 03. Suppose $\max\{c^\top x : Ax \le b\}$ is feasible with optimum $z^*$. For any $\varepsilon > 0$ the system $Ax \le b,\ -c^\top x \le -(z^* + \varepsilon)$ is infeasible. Farkas gives $(y, t) \ge 0$ with $y^\top A - t\,c^\top = 0$ and $y^\top b - t\,(z^* + \varepsilon) < 0$. Feasibility of $Ax \le b$ forces $t > 0$ (otherwise $y$ alone would be a certificate for $Ax \le b$). Scale to $t = 1$: $y^\top A = c^\top$, $y \ge 0$, and $y^\top b < z^* + \varepsilon$. That $y$ is a dual feasible solution with value within $\varepsilon$ of $z^*$, and a finiteness argument removes the $\varepsilon$.

On exactness: the SCIP team's exact SCIP and the VIPR certificate format let a MIP solver emit a proof of optimality that an independent checker verifies in rational arithmetic. For LP, the certificate is exactly the kind of object this unit builds.

01 · closing the loop

The fundamental theorem of LP

theorem

Let $P = \{x : Ax \le b\}$ be nonempty and pointed, and $c \in \mathbb{R}^n$. Then either $\max\{c^\top x : x \in P\}$ is unbounded, or it is attained at a vertex of $P$.

why

Every point of $P$ is a blend of corners plus a push along rays (Minkowski–Weyl). A linear objective of a blend is the same blend of the corners' values, so it can't beat the best corner. A push along a ray either helps forever (unbounded) or doesn't help at all.

full proof in the notes (N)

example: the best corner of the running example

Maximize $2x + y$ over $P$:

corner	$2x + y$
$(0, 0)$	0
$(1, 0)$	2
$(2.5, 1.5)$	6.5
$(0, 4)$	4

The optimum is 6.5, at $(2.5, 1.5)$. Four evaluations, no calculus.

So an LP is infeasible, unbounded, or solved at a corner. There is no fourth case, and in particular no finite best value that is never reached.

explore interactively

Full proof, via Minkowski–Weyl

Write $P = \operatorname{conv}(V) + \operatorname{cone}(R)$ with $V$ the vertices. If $c^\top r > 0$ for some $r \in R$, then $x + t r$ is feasible for every $t \ge 0$ and the objective is unbounded. Otherwise, for $x = \sum \lambda_v v + \sum \mu_r r$, $c^\top x = \sum \lambda_v\, c^\top v + \sum \mu_r\, c^\top r \le \max_{v \in V} c^\top v$, using $\sum \lambda_v = 1$, $\lambda, \mu \ge 0$ and $c^\top r \le 0$. $\square$

This is the theorem that makes LP a combinatorial problem in disguise. The optimum lies in a finite set, the vertices, which lab step 3 can enumerate in $\binom{m}{n}$ tries. Simplex (unit 02) is the idea of not enumerating: move from a vertex to an adjacent, better one until none is better. That works because the objective is linear and $P$ is convex, so a local optimum over the vertex graph is global. On the example, simplex from $(0,0)$ goes to $(1,0)$ and then to $(2.5, 1.5)$, never visiting $(0,4)$.

"No fourth case" is not true for nonlinear problems. $\inf\{e^{-x} : x \ge 0\} = 0$ is finite and never attained. Finitely many linear constraints rule this out.

Pointedness is needed only for the words "at a vertex". Without it, a bounded LP still attains its optimum, on a face that contains a line.

01 · lab

Lab 01 — what you build

about 2½ hours · uv run co lab 01 · exact integers, no solver

Eliminate one variable by Fourier–Motzkin. Tested by sampling points against the exact projection.
Decide feasibility by eliminating everything. Agrees with HiGHS.
Enumerate vertices over all $\binom{m}{n}$ bases, as on the pairs-of-constraints slide. Agrees with qhull, and survives a degenerate pyramid.
Check a Farkas certificate.
Find one, using your own eliminate and the tail trick.

uv run co test 01
uv run co then 01      # FM blow-up vs true facets · your vertices vs qhull · a proof printed

01 · checkpoint

Done when

checkpoint

All five lab steps are ✓.
Given the infeasible system that then prints, with the certificate hidden, you can produce a Farkas vector by hand and explain without reference to any solver why it proves infeasibility.
You can say, on a small example, why vertex, extreme point and BFS pick out the same points, and which of the three your vertices function computes.

Reading

Bertsimas & Tsitsiklis, ch. 2 (§2.1–2.6) and §4.6.
Gärtner & Matoušek, Understanding and Using LP, §6.1–6.4 (Farkas via FM).
Ziegler, Lectures on Polytopes, lecture 1 (optional).

02 Simplex, honestly: walking between the vertices you just enumerated, instead of listing all of them.

Polyhedra, cones, and Farkas' lemma

The route

Polyhedra

Halfspaces, polyhedra, polytopes

The running example

Faces

Three names for a corner

Tight constraints, and counting them

Vertex, extreme point, basic feasible solution

Why the three agree

Finding corners by trying pairs of constraints

Not every polyhedron has a corner

Two descriptions: Minkowski–Weyl

Minkowski–Weyl, on two examples

Fourier–Motzkin elimination

Eliminating one variable

On the example: eliminate $x$

Deciding feasibility, and the explosion

Farkas' lemma

A certificate of infeasibility

Farkas' lemma

Farkas, constructively

Farkas, geometrically

Why this is the root of the trunk

The fundamental theorem of LP

Lab 01 — what you build

Done when

Reading

Next