Part IV · trunk C · constraint programming and SAT · spine

Search, restarts, and large neighbourhood search

Which variable next, which value first, when to give up and start again — and how destroy-and-repair turns an exact engine into a heuristic that scales.

needs 18~1¾ h slides + 2½ h lablab: uv run co lab 19

Units 17 and 18 were about inference, what propagation can deduce. This unit is about decisions: where the tree branches, and when a search should abandon its tree entirely. The unit's claim is empirical, and the lab measures it. Backtracking search effort on a single instance, across random seeds, often has a heavy tail. Restarts are justified by that tail, and only where it exists.

The second half, large neighbourhood search, is the bridge to Part V. It uses the exact engine as a subroutine inside a heuristic loop, and it's how CP solvers find good solutions to problems they cannot prove optimal.

How to read this deck

Three small worked examples sit next to the lab's measurements: dom/wdeg's weights changing over three failures on four variables, the first fifteen Luby cutoffs, and one destroy-and-repair step on a three-job shop drawn as a Gantt chart. All three were computed with the lab's reference solution. The Luby optimality theorem is stated as an idea, with the precise statement in the notes.

19 · overview

The route

1 · heuristics

Variable and value ordering: first-fail, dom/wdeg, impact, activity; symmetry.

2 · heavy tails

Runtime distributions, and why they justify randomisation and restarts.

3 · restarts and nogoods

Luby's universal schedule; what to remember across restarts.

4 · LNS

Destroy and repair with the propagator as the repair engine.

getting around

Words with a dotted underline are glossary terms: click one for its definition. G lists every term in the unit, N shows the notes. Optional extra: interactive figures, linked from the slides they support by a small ▸ chip; skip them freely.

section 1

Heuristics

Fail first; succeed first.

19 · heuristics

Variable and value ordering

Variables: fail first (Haralick & Elliott 1980). Branch where a dead end is most likely, so it is found near the root.

first-fail: smallest domain.
dom/deg: domain size over degree.
dom/wdeg (Boussemart et al. 2004): each constraint's weight counts its failures, so search is drawn to the hard core of the problem.
impact-based (Refalo 2004): prefer variables whose assignments shrank the search space most in the past.
activity-based (Michel & Van Hentenryck 2012): a VSIDS-like count of how often each variable's domain changes.

Values: succeed first. Try the value most likely to lead to a solution. It matters only for satisfiable subtrees; for proofs, every value is tried anyway.

lab step 2: a hidden core

14 easy three-valued variables in a chain, plus 5 variables that must all differ over 4 values. first-fail branches on the chain (smaller domains): more than 20 000 nodes, unfinished. dom/wdeg learns from the first failures that the ≠ clique is where they come from: under 500 nodes to prove infeasibility.

On QCP, where all constraints look alike, dom/wdeg is worse than randomised first-fail (lab then). Heuristics are instance-family bets.

19 · heuristics · example

dom/wdeg, on four variables

Four variables on a cycle of $\ne$ constraints: $c_1: x \ne y$, $c_2: y \ne z$, $c_3: z \ne w$, $c_4: x \ne w$. Domains: $x, y \in \{0,1,2\}$, $z, w \in \{0,1\}$. Every weight starts at 1.

Score $= |D| / \text{wdeg}$, where wdeg adds the weights of the variable's constraints. Branch on the smallest score (ties: lowest index).

what changes, and what doesn't

At the start plain first-fail and dom/wdeg agree: $z$, the smallest domain. After failures in $c_1$, the two variables of $c_1$ get cheaper to pick, and by the second failure $x$ wins outright. The variables far from the failures, $z$ and $w$, never move.

moment	weight of $c_1$	$x$	$y$	$z$	$w$	branch on
start	1	3/2	3/2	2/2	2/2	$z$
$c_1$ fails once	2	3/3	3/3	2/2	2/2	$x$ (4-way tie)
twice	3	3/4	3/4	2/2	2/2	$x$
three times	4	3/5	3/5	2/2	2/2	$x$

Three failures turn a variable with a larger domain into the preferred branch: search moves to where the conflicts are.

section 2

Heavy tails

Most runs are fast. A few never end.

19 · heavy tails

The runtime distribution of one instance

Lab then: quasigroup completion, order 12, 55% holes, 100 seeds of randomised first-fail, nodes to a solution (its runtime distribution):

      16–31 ███████████████████████████████████ 51
      32–63 ████████████████ 24
     64–127 ████████ 12
    128–255 █████ 8
   512–1023 █ 2
   cap 3000 ██ 3

Median 31, p90 174, and 3 runs never finished within 3 000 nodes. The only difference between runs is tie-breaking.

Gomes, Selman, Crato & Kautz 2000

Backtracking search on many CSP and SAT families has runtimes with Pareto-like tails, a heavy tail: $P(T > t) \sim C t^{-\alpha}$, often with $\alpha < 1$, which means infinite mean in the limit.

Cause: an early mistake puts search in a subtree with no solution that it must exhaust. Its size is exponential in how early the mistake happened.

A heavy tail makes the mean a poor summary and a single run a lottery ticket. It also points to the fix: don't wait.

section 3

Restarts and nogoods

Cut the tail off.

19 · restarts

Luby's universal schedule

$$1, 1, 2, 1, 1, 2, 4, 1, 1, 2, 1, 1, 2, 4, 8, \dots$$

why this sequence (Luby, Sinclair & Zuckerman 1993)

You don't know the right cutoff. The Luby sequence tries every power of two, and spends about the same total effort on each, so whichever cutoff was right, it is used often enough. Nothing that doesn't know the distribution does much better.

precise statement in the notes (N)

Geometric schedules ($c \cdot r^i$) are the common alternative. SAT solvers now also restart adaptively, when recent conflicts look worse than average (Audemard & Simon 2012).

Lab then, 100 seeds, base 30:

instance	strategy	mean	median	max
heavy (55%)	plain	153.6	31	3 000 cap
	Luby	52.1	49.5	202
light (58%)	plain	48.7	35	668
	Luby	92.3	93	217

A restart cuts the tail, and with it the (capped) mean, but only where there is a tail to cut: without one, restarts hurt. On the light instance they double the mean. Corrected after unit 28: the median difference on the heavy instance is within its bootstrap interval.

explore interactively

The theorem, precisely

For a Las Vegas algorithm (always correct, random running time) with an unknown runtime distribution, running it with cutoffs $c \cdot \text{luby}(i)$, restarting after each, has expected total running time within an $O(\log)$ factor of the best fixed-cutoff strategy, which knows the distribution. No universal strategy can do better by more than a constant factor.

Why restarts need randomisation, or learning: an identical deterministic restart repeats the same search. The lab's restarts vary through random tie-breaking or through dom/wdeg's weights, which change with every failure. SAT solvers restart deterministically but keep their VSIDS activities and learned clauses, so the new search differs.

The maximum dropping from over 3 000 to 202 is the practical win: restarts turn a heavy-tailed distribution into one with an exponentially decaying tail, so worst cases become predictable.

19 · restarts · example

Luby cutoffs, term by term

run $i$	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15
luby($i$)	1	1	2	1	1	2	4	1	1	2	1	1	2	4	8
cutoff (base 30)	30	30	60	30	30	60	120	30	30	60	30	30	60	120	240
total so far	30	60	120	150	180	240	360	390	420	480	510	540	600	720	960

How it grows: the first $2^k - 1$ terms are two copies of the first $2^{k-1} - 1$ terms, then $2^{k-1}$. Runs 1–7 repeat as runs 8–14, then 8 appears.

reading it against the histogram

On the heavy instance, half the runs finish within 31 nodes. A cutoff of 30 or 60 catches those, and twelve of the first fifteen runs have one of those cutoffs.

The rare long runs never get to run long: the biggest cutoff in 960 nodes of work is 240, where plain search once spent over 3 000.

Each cutoff size gets about equal total effort: runs of 30 use 240 nodes, runs of 60 use 240, runs of 120 use 240, and the single 240.

explore interactively

19 · restarts

What to remember across restarts

Heuristic state: dom/wdeg weights, activity, impacts. The next search branches differently because it learned.
Nogoods: a set of decisions proved to have no solution below it. Record it as a constraint and the next search never enters that subtree again.
Nogoods from restarts (Lecoutre et al. 2007): at each restart, extract the nogoods implied by the current partial tree's refuted branches.
The incumbent, for optimisation: the objective bound carries over.

nogoods, done right

A decision nogood such as "$x_1 = 3 \wedge x_4 = 1$ fails" is weak: it mentions decisions, not reasons. The strong form records why it failed, the propagators and removed values responsible. That's clause learning, the core of CDCL SAT solvers (unit 20), and lazy clause generation brings it to CP (unit 21). (How search uses reasons to skip subtrees, and why it stays complete: unit 19b.)

Lab then's 3-colouring in unit 17 was that difference: 120 334 nodes without learning, 3 106 branches with.

section 4

Large neighbourhood search

The exact engine, as a heuristic's inner loop.

19 · LNS

Destroy and repair

Start from a solution (lab: a greedy dispatch schedule).
Destroy: free a part of it. Lab: a random 30% of the jobs; the other jobs keep their order on every machine.
Repair: search the freed part with the full propagation engine and objective < best, under a small node limit.
Accept an improvement and repeat.

Large neighbourhood search: Shaw (1998) introduced it for vehicle routing with CP repair. ALNS (Ropke & Pisinger 2006) chooses among several destroy operators adaptively.

Lab then, random job-shops, 22 500 nodes each, makespan:

instance	greedy	LNS	branch-and-bound
8 × 4	54	49	57
10 × 5	72	66	77
12 × 6	117	78	84

Branch-and-bound spends its budget at the bottom of one subtree. LNS spends it around a good schedule, where the improvements are.

19 · LNS · example

One LNS step, on a 3 × 3 job-shop

job	operations (machine, duration)
J0	M1 5, then M0 2, then M2 4
J1	M2 3, then M0 2, then M1 2
J2	M0 1, then M2 4, then M1 5

destroy, then repair

Destroy: free job J1. On every machine, J0 and J2 keep their order: M0 J2 before J0, M1 J0 before J2, M2 J2 before J0.

Repair: search where J1 goes, with makespan $\le 15$. Propagation places it in the gaps: M0 at 3–5, M1 at 5–7. J2's last operation moves up to 7–12.

Makespan 16 → 12, which brute force confirms is optimal.

Outline colour is the job; J1, red, is the freed job. In the greedy schedule J1 waits for M0 until time 7, which pushes J2's M1 operation to 11–16.

explore interactively

The greedy schedule (reference greedy_schedule, round-robin over jobs): first operations J0 on M1 at 0, J1 on M2 at 0, J2 on M0 at 0; second round J0 on M0 at 5, J1 on M0 at 7 (M0 busy until 7), J2 on M2 at 3; third round J0 on M2 at 7, J1 on M1 at 9, J2 on M1 at 11 (after J1). Makespan 16.

The step is the reference lns_jobshop with seed 0, one iteration, relax 0.34: it frees exactly one job, J1, keeps the other jobs' machine orders as precedences, and asks for makespan at most 15. The repaired schedule: J1 on M2 0–3, M0 3–5, M1 5–7; J2's last operation 7–12. Kept orders check: M0 has J2 (0–1) before J0 (5–7), M1 has J0 (0–5) before J2 (7–12), M2 has J2 (3–7) before J0 (7–11). The optimum over all $3!^3 = 216$ machine orders, by brute force, is 12.

The greedy schedule's weakness is typical: dispatch commits J1's M0 operation to after J0's without looking ahead, and one freed job was enough to undo it.

19 · LNS

In the tools

CP-SAT's portfolio

Several workers in parallel: different search strategies, restarts, and many LNS workers (random, graph-based, relaxation-induced, scheduling-specific) sharing one incumbent.

Lab then, 30 × 15 job-shop (durations 1–99), 10 s:

configuration	makespan	bound
8 workers, LNS on	1837–1851	1828
8 workers, LNS off	1852	1828
1 worker	2047–2049	1828

The LNS switch is within run-to-run noise here. The portfolio (8 workers against 1) is the big effect.

OR-Tools routing

A local search engine with a CP core: a first solution, then guided local search over routing neighbourhoods.

n	unit 10 LP bound	routing, 5 s	gap
12	422.0	442	4.5%
16	478.2	483	1.0%
20	662.0	680	2.6%

The LP bound from unit 10 certifies how good the heuristic solution is.

19 · lab

Lab 19 — what you build

about 2½ hours · uv run co lab 19

Luby, randomised first-fail, and a quasigroup completion model.
dom/wdeg, through the engine's failure hook.
Restarts on a Luby schedule with a node budget.
Runtime distributions and their order statistics.
Greedy dispatch and LNS for job-shop.

uv run co test 19
uv run co then 19     # histograms · LNS vs B&B · CP-SAT portfolio · OR-Tools routing

19 · checkpoint

Done when

checkpoint

All five lab steps are ✓.
You can show the heavy-tailed runtime histogram from your own solver and demonstrate that restarts move its mean, with distributions, not one lucky run.
You can say why restarts hurt on an instance without a tail.

Reading

Gomes et al., J. Automated Reasoning 24 (2000).
Luby, Sinclair & Zuckerman, IPL 47 (1993).
Boussemart et al., ECAI 2004.
Shaw, CP 1998.

19b backjumping and why search is complete · 20 SAT and CDCL: learning from conflicts · 21 lazy clause generation · 26 local search and metaheuristics.

Search, restarts, and large neighbourhood search

The route

Heuristics

Variable and value ordering

dom/wdeg, on four variables

Heavy tails

The runtime distribution of one instance

Restarts and nogoods

Luby's universal schedule

Luby cutoffs, term by term

What to remember across restarts

Large neighbourhood search

Destroy and repair

One LNS step, on a 3 × 3 job-shop

In the tools

Lab 19 — what you build

Done when

Reading

Next