Convexity, gradients, and what continuity changes

property	definition	for $\frac12 x^\top Q x$
$L$-smooth	$\|\nabla f(x) - \nabla f(y)\| \le L \|x - y\|$	$L = \lambda_{\max}(Q)$
$\mu$-strongly convex	$f(y) \ge f(x) + \nabla f(x)^\top(y-x) + \frac\mu2 \|y-x\|^2$	$\mu = \lambda_{\min}(Q)$
condition number	$\kappa = L / \mu$	eigenvalue spread

$L$ bounds how far a gradient step can safely go (step $1/L$). $\mu$ bounds how flat the bottom can be. Their ratio $\kappa$ decides how many steps you need.

Example: $f = \tfrac12(x^2 + 10y^2)$ has $Q = \mathrm{diag}(1, 10)$, so $L = 10$, $\mu = 1$, $\kappa = 10$: a valley ten times steeper across than along.

Lab step 1 also builds the gradient check that every later unit relies on: central differences, error $O(h^2)$.

method	update	rate, $L$-smooth convex	rate, strongly convex
GD, step $1/L$	$x - \frac1L \nabla f(x)$	$\frac{L R^2}{2k}$	$(1 - \frac1\kappa)^k$
heavy ball (Polyak 1964)	$+\ \beta(x_k - x_{k-1})$	no general guarantee	$(\frac{\sqrt\kappa - 1}{\sqrt\kappa + 1})^k$ on quadratics
Nesterov (1983)	gradient at $x_k + \frac{k-1}{k+2}(x_k - x_{k-1})$	$\frac{2 L R^2}{(k+1)^2}$	$(1 - \frac{1}{\sqrt\kappa})^k$ with constant momentum

$R = \|x_0 - x^*\|$. Heavy ball and Nesterov add momentum; $(1-\frac1\kappa)^k$ is linear convergence. Lab tests check GD's and Nesterov's bounds at every iterate on 12 quadratics.

Lab then, quadratics with n = 100 (relative suboptimality):

κ	GD	heavy ball	Nesterov	CG	L-BFGS
100	760	52	435	103	94
10 000	> 20 000	542	12 048	1 329	749

Predicted: GD 917 and 92 099; heavy ball 46 and 461. Nesterov's worst-case bound is 140 043 and 1.3 million: loose by two orders of magnitude.

Rosenbrock to within 1e−6 of the minimum: GD 34 374, Nesterov 5 019, BFGS 34.

• Tuned heavy ball wins, and isn't a fair entry: it was given μ and L.
• Nesterov's convex schedule is ahead of GD to 1e−8, but its tail flattens: at κ = 100, GD catches up near machine precision. That schedule doesn't use strong convexity.
• CG and L-BFGS build curvature from gradients and need no parameters. On quadratics CG is optimal among Krylov methods.

Plot: out/convergence.png.

Lab then, least squares on the simplex, n = 200, f − f*:

method	100 it.	1 000 it.	5 000 it.
projected gradient	6.9e−4	8.8e−11	at solver precision
Frank–Wolfe	1.0e−1	3.5e−3	1.9e−4

Over a random polytope (n = 30), your simplex method as the oracle cost 2.1 ms per call, HiGHS 1.2 ms, with identical iterates.

$\min f(x)$ s.t. $g_i(x) \le 0$, with $f$, $g_i$ convex and a Slater point (Slater's condition). $x^*$ is optimal iff there are multipliers $\lambda \ge 0$ with the KKT conditions

$$\begin{aligned} &\nabla f(x^*) + \textstyle\sum_i \lambda_i \nabla g_i(x^*) = 0 \\ &g_i(x^*) \le 0,\ \ \lambda_i \ge 0 \\ &\lambda_i\, g_i(x^*) = 0 \end{aligned}$$

Karush (1939); Kuhn & Tucker (1951). With linear constraints no Slater point is needed. The lines are stationarity, feasibility and complementarity.

Lab then, HiGHS on an LP with m = 30, n = 50: residuals 3.6e−13, 0, 3.3e−13, 1.8e−13. Unit 03's exact certificate rejects the same pair rounded to 9 decimals: "primal constraint 12 violated".

$\max 2x + y$ over $x + y \le 4$, $x - y \le 1$, $x, y \ge 0$. Optimum $(2.5, 1.5)$, value 6.5. Try multipliers $y = (1.5,\ 0.5)$ on the two constraints.

The multipliers $1.5$ and $0.5$ are unit 02's shadow prices: loosening $x + y \le 4$ by one unit raises the optimum by 1.5. In the continuous world they are called Lagrange multipliers, and they are the same numbers.

Reading

Boyd & Vandenberghe, Convex Optimization (2004), ch. 5, 9.
Nesterov, Lectures on Convex Optimization (2018).
Bubeck, Found. Trends ML 8 (2015).
Jaggi, ICML 2013.

Next

30 automatic differentiation: where the gradients come from · 31 where discrete meets differentiable.