Discrete Optimization Lecture 4 M. Pawan Kumar

Change in Schedule Five lectures Friday 10 th February, :00am – 11:15am Monday 27 th February, :45am – 11:15am Monday 5 th March, :45am – 1:00pm Friday 9 th March, :00am – 9:30am Monday 12 th March, :45am – 11:15am Three lab sessions Monday 27 th February, :30am – 1:00pm Friday 9 th March, :45am – 11:15am Monday 12 th March, :30am – 1:00pm One exam Monday 19 th March, Time: TBA

Questions about Lab 1?

Finite set Σ called alphabet of size ≥ 2 –{0,1} –{a,b,c,d,e,…,x,y,z} Set Σ* of all finite length strings (called words) –0, 1, 00, 01, 10, 11, 000,…. –discrete, optimization Size of word size(w) = number of letters –size(00) = 2 –size(discrete) = 8 Decision Problem

Problem Π is a subset of Σ* –All words with the answer “yes” Informal problem –Given input word x  Σ*, does x  Π ? Polynomial-time solvable problem Π –There exists a polynomial-time algorithm for the informal problem –Polynomial in size(x) Decision Problem

P Polynomial-time solvable problem Π –There exists a polynomial-time algorithm that decides whether x  Σ* belongs to Π or not. –Polynomial in size(x) P = {all Π, Π is polynomial time solvable} For example –Shortest path with +ve lengths  P –Shortest path with no –ve length circuit  P

NP Π  NP –There exists a problem Π’  P –There exists a polynomial p –There exists an x, size(x) ≤ p(size(w)) such that –w  Π if and only if wx  Π’ Polynomial-time checkable ‘certificate’ For example –Hamiltonian graph problem  NP

Reduction Π is reducible to Λ –There exists a polynomial time algorithm –Given w, returns x –w  Π if and only if x  Λ Λ  NP-complete –All Π  NP can be reduced to Λ –“Hardest” problems in NP

Outline Reduction –“SAT” is reducible to “3-SAT” (review) –“3-SAT” is reducible to “Partition” –“Partition” is reducible to “Hamiltonian Path” NP-hard Problems NP-completeness of SAT

Alphabet Σ containing variables x 1,x 2,…,x n And special symbols (, ), ∧, ∨,~ And not containing 0 and 1 A variable is a Boolean expression Boolean Expression

Alphabet Σ containing variables x 1,x 2,…,x n And special symbols (, ), ∧, ∨,~ And not containing 0 and 1 If v and w are Boolean expressions then –(v ∧ w) is a Boolean expression Boolean Expression

Alphabet Σ containing variables x 1,x 2,…,x n And special symbols (, ), ∧, ∨,~ And not containing 0 and 1 If v and w are Boolean expressions then –(v ∨ w) is a Boolean expression Boolean Expression

Alphabet Σ containing variables x 1,x 2,…,x n And special symbols (, ), ∧, ∨,~ And not containing 0 and 1 If v and w are Boolean expressions then –~v is a Boolean expression –~w is a Boolean expression Boolean Expression

Alphabet Σ containing variables x 1,x 2,…,x n And special symbols (, ), ∧, ∨,~ And not containing 0 and 1 For example, f(x 1,x 2,…,x n ) –((x 2 ∧ x 3 ) ∨ ~(x 3 ∨ x 5 ) ∧ x 2 ) ∨ ~(x 2 ∧ x 5 ) –~(~x 2 ∨ ~x 3 ) ∧ ~x 1 Boolean Expression

f(x 1,x 2,…,x n ) is satisfiable if –there exists an assignment x 1 = α 1, …, x n = α n –where α i  {0,1} –such that f(x 1,x 2,…,x n ) = 1 The following identities hold –0 ∧ 0 = 0, 0 ∧ 1 = 0, 1 ∧ 0 = 0, 1 ∧ 1 = 1 –0 ∨ 0 = 0, 0 ∨ 1 = 1, 1 ∨ 0 = 1, 1 ∨ 1 = 1 –~0 = 1, ~1 = 0 –(0) = 0, (1) = 1 Satisfiability

Given the alphabet Σ Given the identities for 0 and 1 SAT is a subset of Σ* that is satisfiable Informal Problem –Given a Boolean expression w, is w satisfiable SAT

A special case of SAT Given variables x 1,x 2,…,x n  Σ –B 1 consists of x 1,~x 1,…,x n,~x n –B 2 consists of (w 1 ∨ … ∨ w k ), w i  B 1, 1≤k≤3 –B 3 consists of w 1 ∧ w 2 ∧ … ∧ w m, w i  B 2 –e.g., (x 1 ∨ ~x 2 ∨ x 3 ) ∧ (x 2 ∨ ~x 3 ∨ x 4 ) ∧ (~x 1 ∨ ~x 2 ) 3-SAT is the subset of B 3 that is satisfiable 3-SAT

x 1 = x 2 ∨ x 3 –(x 1 ∨ ~x 2 ) ∧ (x 1 ∨ ~x 3 ) ∧ (~x 1 ∨ x 2 ∨ x 3 ) x 1 = x 2 ∧ x 3 –(~x 1 ∨ x 2 ) ∧ (~x 1 ∨ x 3 ) ∧ (x 1 ∨ ~x 2 ∨ ~x 3 ) x 1 = ~x 2 –(x 1 ∨ x 2 ) ∧ (~x 1 ∨ ~x 2 ) SAT is reducible to 3-SAT Proved in the last lecture !!

Outline Reduction –“SAT” is reducible to “3-SAT” (review) –“3-SAT” is reducible to “Partition” –“Partition” is reducible to “Hamiltonian Path” NP-hard Problems NP-completeness of SAT

A special case of SAT Given variables x 1,x 2,…,x n  Σ –B 1 consists of x 1,~x 1,…,x n,~x n –B 2 consists of (w 1 ∨ … ∨ w k ), w i  B 1, 1≤k≤3 –B 3 consists of w 1 ∧ w 2 ∧ … ∧ w m, w i  B 2 –e.g., (x 1 ∨ ~x 2 ∨ x 3 ) ∧ (x 2 ∨ ~x 3 ∨ x 4 ) ∧ (~x 1 ∨ ~x 2 ) 3-SAT is the subset of B 3 that is satisfiable 3-SAT

A finite set X Partition of X is a collection of subsets –Mutually exclusive –Collectively exhaustive For example, X = {a,b,c,d,e,f} –{{a,b},{c},{d,e,f}} is a partition –{{a,b},{a,c},{d,e,f}} is not a partition –{{a,b},{c},{d,e}} is not a partition Partition

A finite set X Partition of X is a collection of subsets –Mutually exclusive –Collectively exhaustive Problem: Given collection of subsets C –Does C contain a partition of X? –Or not? Partition

f = w 1 ∧ w 2 ∧ … w m Bipartite undirected graph with V = V 1 U V 2 –V 1 are variables x 1,x 2,…,x n –V 2 are words w 1,w 2,…,w m Edges E = E 1 U E 2 –E 1 = {(w i,x j )}, x j  w i –E 2 = {(w i,x j )}, ~x j  w i 3-SAT is reducible to Partition

Collection C 1 of sets {w i } U E i –E i is non-empty –E i is a subset of edge set incident with w i Collection C 2 of sets {x j } U E j and {x j } U E’ j –E j is the set of all edges in E 1 incident with x j –E’ j is the set of all edges in E 2 incident with x j f is satisfiable iff C 1 U C 2 contains a partition 3-SAT is reducible to Partition Proof on board !!!

Outline Reduction –“SAT” is reducible to “3-SAT” (review) –“3-SAT” is reducible to “Partition” –“Partition” is reducible to “Hamiltonian Path” NP-hard Problems NP-completeness of SAT

A finite set X Partition of X is a collection of subsets –Mutually exclusive –Collectively exhaustive Problem: Given collection of subsets C –Does C contain a partition of X? –Or not? Partition

Digraph D = (V, A) Path P is Hamiltonian if –It traverses each vertex in V –All vertices in the path are distinct Problem: Given D –Does it contain a Hamiltonian Path? –Or not? Hamiltonian Path Connection to Shortest Path?

Partition Problem: Set X, Collection C –X = {1,2,…,k} –C = {C 1,C 2,…,C m } Let C i = {j 1,…,j t } Partition is reducible to Hamiltonian Introduce r 0 and s 0. Connect r m to s 0.

Partition Problem: Set X, Collection C –X = {1,2,…,k} –C = {C 1,C 2,…,C m } Let C i = {j 1,…,j t } Partition is reducible to Hamiltonian C has a partition iff G has Hamiltonian r 0 -s k path

Partition Problem: Set X, Collection C –X = {1,2,…,k} –C = {C 1,C 2,…,C m } Let C i = {j 1,…,j t } Partition is reducible to Hamiltonian Left as Exercise !!

Outline Reduction NP-hard Problems NP-completeness of SAT

NP-hard Need not be in NP –No polynomial-time checkable certificate –Not even a decision problem (e.g. optimization) At least as hard as NP-complete problems Λ  NP-hard –There exists Π  NP-complete –Π is reducible to Λ

NP-hard Image Courtesy of Wikipedia

NP-hard General MAP estimation problem is NP-hard How can we prove this? Left as exercise

Outline Reduction NP-hard Problems NP-completeness of SAT –Random Access Machine (review) –Cook’s Theorem

Given an array f with elements = 0,1 strings RAM executes a set of instructions Each instruction can –Read entries from prescribed positions –Perform arithmetic operation on read entries –Write answers to prescribed positions Random Access Machine

Given an array f with elements = 0,1 strings Finite set of variables z 0, z 1, …, z k Initially, z i = 0 and f contains input Instructions numbered 0, 1, …, t Variable z 0 stores the instruction to execute Random Access Machine

Given an array f with elements = 0,1 strings Finite set of variables z 0, z 1, …, z k Initially, z i = 0 and f contains input Instructions numbered 0, 1, …, t Stop if z 0 > t Random Access Machine

Given an array f with elements = 0,1 strings Finite set of variables z 0, z 1, …, z k Initially, z i = 0 and f contains input Read instruction –z i := f(z j ) Random Access Machine

Given an array f with elements = 0,1 strings Finite set of variables z 0, z 1, …, z k Initially, z i = 0 and f contains input Write instruction –f(z i ) := z j Random Access Machine

Given an array f with elements = 0,1 strings Finite set of variables z 0, z 1, …, z k Initially, z i = 0 and f contains input Add instruction –z i := z j + z k Random Access Machine

Given an array f with elements = 0,1 strings Finite set of variables z 0, z 1, …, z k Initially, z i = 0 and f contains input Subtract instruction –z i := z j - z k Random Access Machine

Given an array f with elements = 0,1 strings Finite set of variables z 0, z 1, …, z k Initially, z i = 0 and f contains input Multiply instruction –z i := z j z k Random Access Machine

Given an array f with elements = 0,1 strings Finite set of variables z 0, z 1, …, z k Initially, z i = 0 and f contains input Divide instruction –z i := z j / z k Random Access Machine

Given an array f with elements = 0,1 strings Finite set of variables z 0, z 1, …, z k Initially, z i = 0 and f contains input Increment instruction –z i := z i + 1 Random Access Machine

Given an array f with elements = 0,1 strings Finite set of variables z 0, z 1, …, z k Initially, z i = 0 and f contains input Binarize instruction –z i := 1, if z i > 0 –z i := 0, otherwise Random Access Machine

Outline Reduction NP-hard Problems NP-completeness of SAT –Random Access Machine (review) –Cook’s Theorem

Key Observation z i = f(z j ) Remember z i, z j and f(z j ) are Boolean strings –z i = a 1 a 2 a 3 a 4 …a n, a  {0,1} n –f(z j ) = b 1 b 2 b 3 b 4 …b n, b  {0,1} n “Read” as Boolean expression –a k =b k implies (a k ∧ b k ) ∨ (~a k ∧ ~b k ) = 1 –a k = b k = 1 OR a k = b k = 0 –B(a,b) = 1

Key Observation f(z i ) = z j Remember z i, z j and f(z i ) are Boolean strings –f(z i ) = a 1 a 2 a 3 a 4 …a n, a  {0,1} n –z j = b 1 b 2 b 3 b 4 …b n, b  {0,1} n “Write” as Boolean expression –a k =b k implies (a k ∧ b k ) ∨ (~a k ∧ ~b k ) = 1 –a k = b k = 1 OR a k = b k = 0 –B(a,b) = 1

Key Observation z i = z j + z k Remember z i, z j and z k are Boolean strings –z i = a 1 a 2 a 3 a 4 …a n, a  {0,1} n –z j = b 1 b 2 b 3 b 4 …b n, b  {0,1} n –z k = c 1 c 2 c 3 c 4 …c n, c  {0,1} n “Add” as Boolean expression –B(a,b,c) = 1 –Boolean algebra !!

Key Observation z i = z j - z k Remember z i, z j and z k are Boolean strings –z i = a 1 a 2 a 3 a 4 …a n, a  {0,1} n –z j = b 1 b 2 b 3 b 4 …b n, b  {0,1} n –z k = c 1 c 2 c 3 c 4 …c n, c  {0,1} n “Subtract” as Boolean expression –B(a,b,c) = 1 –Boolean algebra !!

Key Observation z i = z j z k Remember z i, z j and z k are Boolean strings –z i = a 1 a 2 a 3 a 4 …a n, a  {0,1} n –z j = b 1 b 2 b 3 b 4 …b n, b  {0,1} n –z k = c 1 c 2 c 3 c 4 …c n, c  {0,1} n “Multiply” as Boolean expression –B(a,b,c) = 1 –Boolean algebra !!

Key Observation z i = z j / z k Remember z i, z j and z k are Boolean strings –z i = a 1 a 2 a 3 a 4 …a n, a  {0,1} n –z j = b 1 b 2 b 3 b 4 …b n, b  {0,1} n –z k = c 1 c 2 c 3 c 4 …c n, c  {0,1} n “Divide” as Boolean expression –B(a,b,c) = 1 –Boolean algebra !!

Key Observation z i = z i + 1 Remember old/new z i are Boolean strings –Old z i = a 1 a 2 a 3 a 4 …a n, a  {0,1} n –New z i = b 1 b 2 b 3 b 4 …b n, b  {0,1} n “Increment” as Boolean expression –B(a,b) = 1 –Boolean algebra !!

Key Observation z i = 1 if z i > 0 and 0 otherwise Remember old/new z i are Boolean strings –Old z i = a 1 a 2 a 3 a 4 …a n, a  {0,1} n –New z i = b 1 b 2 b 3 b 4 …b n, b  {0,1} n “Binarize” as Boolean expression –B(a,b) = 1 –Boolean algebra !!

Cook’s Theorem Any problem in NP can be reduced to SAT Consider a problem Π  NP There is a corresponding problem Π’  P w  Π –There exists x, size(x) = poly(size(w)) –wx  Π’

Input for RAM of Π’ Input f(1), f(2), …, f(q) Each f(i)  {0,1} n Let w = f(1)f(2)…f(q’) Then x=f(q’+1)f(q’+2)…f(q) Size of input = q*n

Output for RAM of Π’ W.l.o.g., at termination output is written at f(0) E.g., f(0) = {1} n implies wx  Π’, otherwise not.

Variables for RAM of Π’ Variables z 0,z 1,…,z k k is polynomial in the size of input

Machine Memory The word m = z 0 z 1..z k f(0)f(1)…f(q) Size of m = (k+q+2)*n = poly(size(input)) Machine memory at iteration i = m i m i = z i 0 z i 1..z i k f i (0)f i (1)…f i (q)

History The algorithm terminates after r iterations Define history h = m 0 m 1 m 2 …m r Size of h = (r+1)*size(m) = s = poly(size(input))

Correct and Incorrect History Given h = {0,1} s, it is “correct” if there is an input wx such that the RAM has history h Given h = {0,1} s, it is “incorrect” if there is no input wx such that the RAM has history h

SAT Boolean expression g(h) –g(h) = 1 if h is “correct” –g(h) = 0 if h is “incorrect” (How to construct g?) Set f 0 (1)f 0 (2)f 0 (q’) = w Set f r (0) = {1} n w  Π if and only if g is satisfiable