1. Intro to Computational Complexity
INFO 2950
Prof. Carla Gomes
Module
Intro to Computational Complexity
Rosen, Chapter 3

2. How do computer scientists differentiate between good (efficient) and bad (not efficient) algorithms?
The yardstick is that any algorithm that runs in no more than polynomial time is an efficient algorithm; everything else is not.

3. c lg n lgc n n n2 n3 nc,c≥1 rn, r>1 Efficient algorithms
Ordered functions by their growth rates c
Order
constant 1
logarithmic 2
polylogarithmic 3
nr ,0<r<1 n
sublinear 4
linear 5
nr ,1<r<2
subquadratic 6
quadratic 7
cubic 8
nc,c≥1
rn, r>1
polynomial 9
exponential 10
lg n
lgc n n3 n2
Efficient algorithms
Not efficient algorithms

4. log scale

5. Polynomial vs. exponential growth (Harel 2000)
Binary B&B alg.
exponential
polynomial
LP’s interior point
Min. Cost Flow Algs
Transportation Alg
Assignment Alg
Dijkstra’s alg. N2

6. Theory of NP-completeness or NP-hardness Easy vs. hard problems
Problem Complexity
Theory of NP-completeness or NP-hardness
Easy vs. hard problems

7. Overview of complexity
How can we show a problem is efficiently solvable?
We can show it constructively. We provide an algorithm and show that it solves the problem efficiently. E.g.:
Shortest path problem - Dijkstra’s algorithm runs in polynomial time, O(n2). Therefore the shortest path problem can be solved efficiently.
Linear Programming – The Interior Point method has polynomial worst-case complexity. Therefore Linear programming can be solved efficiently.
(*) The simplex method has exponential worst case complexity/ However, in practice the simplex algorithm
seems to scale as m3, where m is the number of functional constraints.

8. Overview of complexity
How can we show a problem is not efficiently solvable?
How do you prove a negative? Much harder!!!
This is the aim of complexity theory.

9. Easy (efficiently solvable) problems vs Hard Problems
Easy Problems - we consider a problem X to be “easy” or efficiently solvable, if there is a polynomial time algorithm A for solving X. We denote by P the class of problems solvable in polynomial time.
Hard problems --- everything else. Any problem for which there is no polynomial time algorithm is an intractable problem. .

10. Two problems
Problem 1:
Input: A road map of cities
with distances attached to the
road map, and two designated
cities A and B
Output: A description of the
shortest path between A and B
Problem 2:
Input: A road map of cities
with distances attached to the
road map, and two designated
cities A and B
Output: A description of the
longest path between A and B
We know that it is easy – it can be solved efficiently!
Can we come up with an efficient algorithm for the longest path?

11. I’d like you to develop an effcient algorithm to find the longest path between two points in a graph.

12. Your Longest Path Algorithm between two nodes, u and v
G=(N,E) 1 2 3 4 5 6 u v
Initialization: MaxPath  none ;
MaxPathLength  0;
For each path P starting at 1:
if P is a simple path from u to v and length(P) > MaxPath
MaxPath  P;
MaxPathLength  length(P)
Return:
MaxPath; MaxPathLength;

13. I know…If each intersection
leads to two roads and we have
100 intersections that can
lead to 2100 paths …
Is that the best you can do? -- that seems to be a bad algorithm!!!

14. I cant’ find an efficient algorithm. I guess I’m too dumb.

15. I can’t find an efficient algorithm, but neither can these famous researchers.

16. In 1936, Alan Turing, a British mathematician, showed that there exists a relatively simple universal computing device that can perform any computational process.
Computers use such a universal model.
Alan Turing
Turing Machine
(abstraction)
Turing also showed the limits of computation – some problems cannot be computed even with the most powerful computer and even with unlimited amount of time – e.g., Halting problem.

17. Brilliant mathematician, synthesizer, and promoter
of the stored program concept, whose logical design
of the Institute of Advanced Studies (Princeton)
Computer became the prototype of today’s computer (*)
- the von Neumann Architecture.
John Von Neumann
(*) sequential i.e., non-parallel computers

18. Invented the theory of NP-Completeness –
Steven Cook
Invented the theory of NP-Completeness –
proved that a simple problem - Satisfiability
is NP-Complete.
Given a propositional formula, is there an assignment to its
variables (a, b, and c – True or False) making the formula true?
Richard Karp
Showed that several important problems
and applications are NP-Complete and NP-hard,
including Integer Programming.

19. Michael Garey
David Johnson

20. Invented Linear Programming Formulations: max 3 x + 5 y
s.t
x ≤ 4
y ≤ 12
3 x + 2 y ≤ 18
x,y ≥ 0
Invented Simplex Algorithm
George Dantzig

21. Satisfiability

22. Propositional Satisfiability problem
Satifiability (SAT): Given a formula in propositional calculus, is there a model
(i.e., a satisfying interpretation, an assignment to its variables) making it true?
We consider clausal form, e.g.:
( a  b   c ) AND ( b   c) AND ( a  c)
possible assignments
SAT: prototypical hard combinatorial search and reasoning problem. Problem is NP-Complete. (Cook 1971)
Surprising “power” of SAT for encoding computational problems.

23. Satisfiability as an Encoding Language

24. Encoding Latin Square Problems in Propositional Logic
Variables:
Each variables represents a color assigned to a cell.
Clauses:
Some color must be assigned to each cell
No color is repeated in the same row
No color is repeated in the same column
(clause of length n); n2
(sets of negative binary clauses); n(n-1)/2 n n
(example
Row i
Color k)
(sets of negative binary clauses);
n(n-1)/2 n n
(example
Colum j
Color k)

25. 3D Encoding or Full Encoding
This encoding is based on the cubic representation of the quasigroup: each line of the cube contains exactly one true variable;
Variables:
Same as 2D encoding.
Clauses:
Same as the 2 D encoding plus:
Each color must appear at least once in each row;
Each color must appear at least once in each column;
No two colors are assigned to the same cell;

26. Dimacs format At the top of the file is a simple header.
p cnf <variables> <clauses>
Each variable should be assigned an integer index. Start at 1, as 0 is used to indicate the end of a clause. The positive integer a positive literal, whereas a negative interger represents a negative literal.
Example
 ( x1  x7)

27. Extended Latin Square 2x2
order 2
-1 -1
p cnf 8 24
A cell gets at most a color
No repetition of color in a column
No repetition of color in a row
1/2 3/4
5/6 7/8
A cell gets a color
1 – cell 11 is red
2 – cell 11 is green
3 – cell 12 is red
4 – cell 12 is green
5 – cell 21 is red
6 – cell 21 is green
7 – cell 22 is red
8 – cell 22 is green
A given color goes in each column
A given color goes in each row

28. Significant progress in Satisfiability Methods
Software and hardware verification – complete methods are critical - e.g. for verifying the correctness of chip design, using SAT encodings
Applications:
Hardware and
Software Verification
Planning,
Protocol Design, etc.
Going from 50 variable, 200 constraints
to 1,000,000 variables and 5,000,000 constraints
in the last 10 years
Current methods can verify automatically the correctness of > 1/7 of a Pentium IV.

29. Model Checking

30. Turing Award
Source: Slashdot

31. A “real world” example

32. Bounded Model Checking instance:
i.e. ((not x1) or x7)
and ((not x1) or x6)
and … etc.

33. clauses / constraints are getting more interesting…
10 pages later: …
(x177 or x169 or x161 or x153 …
or x17 or x9 or x1 or (not x185))
clauses / constraints are getting more interesting…

34. 4000 pages later:
!!!
a 59-cnf clause… …

35. this instance in less than one minute.
Finally, 15,000 pages later:
Note that:
… !!!
MiniSAT solver solves
this instance in less than one minute.

36. Effective propositional inference

37. Effective propositional inference
Two families of algorithms for propositional inference (checking satisfiability) based on model checking (which are quite effective in practice):
Complete backtracking search algorithms
DPLL algorithm (Davis, Putnam, Logemann, Loveland)
Incomplete local search algorithms
WalkSAT algorithm

38. Theory of NP-completeness and NP-hardness Easy vs. hard problems

39. Can satisfiability or integer programming be solved in polynomial time?
FACT: every algorithm that has ever been developed for satisfiability or integer programming takes exponential time.
Hundreds of very smart researchers have tried to come up with polynomial time algorithms for satisfiability or integer programming, and failed.
It is generally believed that there is no polynomial time algorithm for satisifiability or integer programming.
Complexity theory: deals with proving that satisfiability, integer programming, and many other problems are computationally hard.

40. Decision Problems
NP-Completeness theory deals with decision problems. What is a decision problem? A problem for which there is a yes/no answer.
Examples:
Given a propositional formula, is there an assignment of values True
or False to its variables that makes the formula true?
Is there a path between two nodes in a graph shorter than k?
(decision version of shortest path problem)
Is there a path between two nodes in a graph longer than k?
(decision version of longest path problem)
Most optimization problems can be formulated as a decision problem

41. Class NP Class of Problems solvable in Nondeterministic-Polynomial Time
We say that a decision problem is solvable in Non-deterministic polynomial time if:
The solution can be verified in polynomial time. (E.g., verifying that a path has length greater than K)
If we imagine that we have an exponential number of processors, we can check all possible solutions simultaneously and therefore answer in polynomial time NP P

42. NP P NP-Complete Class NP-Complete
The first problem to be shown to be NP-Complete was Satisfiability – Cook showed that all the problems in NP could be translated (in polynomial time) as Satisfiability problems;
The word complete means that every problem in the class NP-complete can be reduced (in polynomial time) into another problem of the class NP-complete. For example all the problems in the class NP-complete can be written as Satisfiability problems.
The class of NP-Complete problems is the class of the hardest computational problems in the class NP: every NP problem can be transformed into an NP-complete problem (the reverse is not true!!!) NP P
NP-Complete

43. Is P not equal to NP? $1,000,000 question
Is that true that not all problems is NP can be solved in polynomial time?
Class of NP-Complete Problems
They all admit exponential time solutions;
Nobody has ever been able to find a polynomial time solution for any single problem in the class;
Nobody has ever been able to prove an exponential lower bound for any single problem in the class;

44. P not equal to NP? $1,000,000 question
Pictorial interpretation of this question
Is this the right picture?
There are problems in NP that are inherently intractable
and cannot be solved in polynomial time. NP P
NP-Complete

45. P not equal to NP? $1,000,000 question
Pictorial interpretation of this question
Or is this the right picture?
All the problems in NP can be solved in polynomial time.
Even though at this point we don’t know of polynomial time
algorithms to solve some problems in NP, they exist…
P = NP

46. Class of NP-Complete Problems One Million Dollar Prize
Completeness
– if someone were to find a polynomial time solution for a single problem in the class NP-complete  all the problems could be solved in polynomial time!!!
if someone were to prove an exponential lower bound for a single problem in the class NP-complete  all the problems in the class would be intractable !!!

47. P vs. NP One Million Dollar Prize

48. Class of NP-Complete Problems One Million Dollar Prize
Completeness
– if someone were to find a polynomial time solution for a single problem in the class NP-complete  all the problems could be solved in polynomial time!!!
if someone were to prove an exponential lower bound for a single problem in the class NP-complete  all the problems in the class would be intractable !!!

49. NP-Complete problems are inherently hard!
Conjecture
NP-Complete problems are inherently hard!
They are intractable ! NP P
NP-Complete

50. On Proving NP-Completeness results
Suppose that we want to prove that the a problem 
is NP-Complete. How do we do it?
Find a known NP-complete problem, NPC.
Show that NPC can be transformed (in polynomial time) into .
Show that there is a solution to NPC if and only if there there is a solution to .

51. Proof of NP-Completeness of Sudoku
A hamiltonian cycle is a cycle that passes through each node exactly once.
A hamiltonian path is a path that includes every node of G.
Suppose that we know that the problem of deciding if we can complete a Latin Square is NP-Complete – i.e., the Latin Square Completion problem is NP-Complete.
We will show that the problem of deciding if we can complete a partial Sudoku instance is also NP-Complete.

52. Sudoku
Can we complete this matrix using numbers from 1 to 9, with repeating a number in a row, column, or block?
9 55 ~ 3x possible completions

53. 1 1 3 4 5 6 7 8
Reduction
To Sudoku 1 3 4 5 6 7 8 1 1 2 6 7 8 1 3 4 5 1 7 8 6 4 5 3
Latin Square
Completion
Problem 4 5 3 1 7 8 6 4 5 3 1 7 8 6 1 2 8 6 7 5 3 4 5 3 4 1 2 8 6 7 8 6 7 5 3 4 1 2

54. If there is
a Latin
Square
Completion
There is also
a way of
Completing
the Sudoku
matrix 1 2 1 2 1 3 4 5 6 7 8 1 1 3 4 5 6 7 8 2 1 1 2 1 2 6 7 8 2 1 3 4 5 2 1 7 8 6 4 5 3
Latin Square
Completion
Problem 4 5 3 2 1 7 8 6 4 5 3 7 8 6 2 1 1 2 8 6 7 5 3 4 5 3 4 1 2 8 6 7 8 6 7 5 3 4 1 2

55. There is
also a
way of
completing
the Latin
Square
Completion
If there is
a way of
the Sudoku
matrix 2 1 1 2 2 1 3 4 5 6 7 8 1 3 4 5 6 7 8 2 1 1 2 6 7 8 2 1 3 4 5 2 1 7 8 6 4 5 3
Latin Square
Completion
Problem 4 5 3 2 1 7 8 6 4 5 3 1 7 8 6 2 1 2 8 6 7 5 3 4 5 3 4 1 2 8 6 7 8 6 7 5 3 4 1 2

56. NP-Hard Problems
The theory of NP-Completeness deals with decision problems.
What about optimization problems?
An optimization problem whose corresponding decision problem is NP-complete is said to be NP-Hard – it means that it is at least as hard as the problems in the class NP-complete.

57. On NP-hardness
Theorem. If problem X is NP-hard, and if X is a special case of Y, then Y is NP-hard.
Example integer programming is NP-hard
0-1 integer programming is a special case of integer programming. Therefore, integer programming is NP-hard.

58. Class PSPACE What if we worry about space - i.e., memory requirements?
Class PSPACE - the set of decision problems
that can be solved using a polynomial amount
of memory, and unlimited time.
Clearly P  PSPACE

59. Class PSPACE Class PSPACE - class of problems that appears
to be harder than NP and co-NP.
Why? space can be re-used while time cannot.
Examples:
Consider an algorithm that counts from 0 to 2n – 1. while this algorithm can be implemented with a simple n-bit counter, it runs for an exponential time!
We can also solve the satisfiability problem using only a polynomial amount of space, for example by trying all possible assignments using also an n-bit counter.

60. PSPACE-Complete
A decision problem is in PSPACE-complete if it is in PSAPCE, and
every problem in PSPACE can be reduced to it in polynomial time. The
problems in PSPACE-complete can be thought of as the hardest problems
in PSPACE. These problems are widely suspected to be outside of P and
NP, but that is not known.

61. Satisfiability (NP-complete):
Quantified Satisfiability (PSPACE-complete):
(The most basic PSPACE-complete problem is identical to satisfiability, except it alternates existential and universal quantifiers)
The PSPACE-complete problem resembles a game: is there some move I can
make, such that for all moves my opponent might make, there will then be some
move I can make to win? The question alternates existential and universal
quantifiers. Not surprisingly, many puzzles turn out to be NP-complete,
and many games turn out to be PSPACE-complete.

62. Checkers is PSPACE-complete when generalized so that it can be played on an
n × n board.
Generalized versions of the games Hex and Reversi and
the solitaire games such as Rush Hour, Mahjong, Atomix and Sokoban.
Note that the definition of PSPACE-complete is based on
asymptotic complexity: the time it takes to solve a
problem of size n, in the limit as n grows without bound.
That means a game like checkers (which is played on an 8
× 8 board) could never be PSPACE-complete. That is why
all the games were modified by playing them on an n × n
board instead.

63. PSPACE NP
Co-NP P

64. Some examples of NP-hard problems
Longest path
Traveling Salesman Problem
Capital Budgeting Problem (knapsack problem)
Independent Set Problem
Fire Station Problem (set covering)
0-1 Integer programming
Integer Programming
Project management with resource constraints
and thousands more

65. NO WAY!!! Okay – Should we give up?
Here is why:
The theory of NP-completeness is only a worst case result. Not all problem instances are as hard as the worst case.
Real problems tend to have sub-problems that are tractable and by exploiting the structure of such sub-problems using efficient algorithms such Unit Propagation, LP, Min Cost Flow, Transportation. Assignment and shortest path methods we can solve much larger problem instances.
In the 1970’s we could only solve Binary Integer Programming instances with  100 variables. By exploiting the structure we can now solve real world instances with over 120,000 variables and 4000 functional constraints.
In the 1990’s we could only solve Satisfiability instances with  50 variables and 200 clauses. By using randomization and learning to exploit the structure we can now solve real world instances with over 1,000,000 variables and 5,000,000 functional clauses.

66. Planning and Scheduling And Supply Chain Management
NP-Complete and
NP-Hard Problems
Experiment
Design
Goal
Start
Satisfiability
(A or B) (D or E or not A)
Planning and Scheduling
And Supply Chain Management
EXPONENTIAL
FUNCTION
POLYNOMIAL
Hard Computational
Problems
Scale Exponentially
EXPONENTIAL-TIME
ALGORITHMS
EXPLOSIVE
COMBINATORICS
Data Analysis
& Data Mining
Protein
Folding
And Medical
Applications
Capital Budgeting
And Financial Appl.
Combinatorial
Auctions
Information
Retrieval
Fiber optics routing
Software & Hardware
Verification
Many more
applications!!!
Require
powerful computational and
mathematical tools!