Problems you shouldn’t tackle

Problem Complexity

Relations between Problems, Algorithms, and Programs Problem Algorithm Program....

Algorithmic Performance Thus Far Some examples thus far: –O(1) Insert to front of linked list –O(log N) Binary Search –O(N) Simple/Linear Search –O(N Log N)Merge/Quick/Heap Sort –O(N 2 )Insertion/Bubble Sort But it could get worse: –O(N 5 ), O(N 2000 ), etc.

An O(N 5 ) Example For N = 256 N 5 = = 1,100,000,000,000 If we had a computer that could execute a million instructions per second… 1,100,000 seconds = 12.7 days to complete But it could get worse…

The Power of Exponents A rich king and a wise peasant…

The Wise Peasant’s Pay Day(N)Pieces of Grain N2N 63 9,223,000,000,000,000, ,450,000,000,000,000,000

How Bad is 2 N ? Imagine being able to grow a billion (1,000,000,000) pieces of grain a second… It would take –585 years to grow enough grain just for the 64 th day –Over a thousand years to fulfill the peasant’s request! ?

So the King cut off the peasant’s head.

The Towers of Hanoi A B C Goal: Move stack of rings to another peg –Rule 1: May move only 1 ring at a time –Rule 2: May never have larger ring on top of smaller ring

Original State Move 1 Move 2 Move 3 Move 4 Move 5 Move 6 Move 7 Towers of Hanoi: Solution

Towers of Hanoi - Complexity For 3 rings we have 7 operations. To add a 4 th ring, you have to move the upper 3 rings to one peg, move the 4 th ring, then move the upper 3 rings on top of it – doubling the amount of work. In general, the cost is 2 N – 1 = O(2 N ) This grows incredibly fast!

Towers of Hanoi (2 N ) Runtime For N = 64 2 N = 2 64 = 18,450,000,000,000,000,000 If we had a computer that could execute a million instructions per second… It would take 584,000 years to complete But it could get worse…

The Bounded Tile Problem Match up the patterns in the tiles. Can it be done, yes or no?

The Bounded Tile Problem Matching tiles

Tiling a 5x5 Area 25 available tiles remaining

Tiling a 5x5 Area 24 available tiles remaining

Tiling a 5x5 Area 23 available tiles remaining

Tiling a 5x5 Area 22 available tiles remaining

Tiling a 5x5 Area 2 available tiles remaining

Analysis of the Bounded Tiling Problem Tile a 5 by 5 area (N = 25 tiles) 1st location: 25 choices 2nd location: 24 choices And so on… Total number of arrangements: –25 * 24 * 23 * 22 * 21 *.... * 3 * 2 * 1 –25! (Factorial) = 15,500,000,000,000,000,000,000,000 Bounded Tiling Problem is O(N!)

Tiling (N!) Runtime For N = 25 25! = 15,500,000,000,000,000,000,000,000 If we could “place” a million tiles per second… It would take 470 billion years to complete Why not a faster computer?

A Faster Computer If we had a computer that could execute a trillion instructions per second (a million times faster than our MIPS computer)… 5x5 tiling problem would take 470,000 years 64-ring Tower of Hanoi problem would take 213 days Why not an even faster computer!

Where Does this Leave Us? Clearly algorithms have varying runtimes. We’d like a way to categorize them: –Reasonable, so it may be useful –Unreasonable, so why bother running

Performance Categories of Algorithms Sub-linear O(Log N) Linear O(N) Nearly linear O(N Log N) Quadratic O(N 2 ) Exponential O(2 N ) O(N!) O(N N ) Polynomial

Cost and Complexity Algorithm complexity can be expressed in Order notation, e.g. “at what rate does work grow with N?”: –O(1)Constant –O(logN)Sub-linear –O(N)Linear –O(NlogN)Nearly linear –O(N 2 )Quadratic –O(X N )Exponential But, for a given problem, how do we know if a better algorithm is possible?

The Problem of Sorting For example, in discussing the problem of sorting: Two algorithms to solve: –Bubblesort – O(N 2 ) –Mergesort – O(N Log N) Can we do better than O(N Log N)?

Algorithm vs. Problem Complexity Algorithmic complexity is defined by analysis of an algorithm Problem complexity is defined by –An upper bound – defined by an algorithm –A lower bound – defined by a proof

The Upper Bound Defined by an algorithm Defines that we know we can do at least this well Perhaps we can do better Lowered by a better algorithm –“When I was a boy I knew only about Bubblesort with O(N 2 ). Today I am a man I have learned Mergesort with O(NlogN).” John Wayne in “True Algorithms” 1954

The Lower Bound Defined by a proof Defines that we know we can do no better than this It may be worse Raised by a better proof –“For comparison sorting you can probably prove that sorting must have a O(N). I can prove that sorting has O(NlogN).”

Upper and Lower Bounds The Upper bound is the best algorithmic solution that has been found for a problem. –“What’s the best that we know we can do?” The Lower bound is the best solution that is theoretically possible. –“What cost can we prove is necessary?”

Changing the Bounds Upper bound Lowered by better algorithm Lower bound Raised by better proof

Open Problems The upper and lower bounds differ. Upper bound Lowered by better algorithm Lower bound Raised by better proof Unknown

Closed Problems The upper and lower bounds are identical. Upper bound Lower bound

Closed Problems Better algorithms are still possible Better algorithms will not provide an improvement detectable by “Big O” Better algorithms can improve the constant costs hidden in “Big O” characterizations

Tractable vs. Intractable Problems are tractable if the upper and lower bounds have only polynomial factors. –O (log N) –O (N) –O (N K ) where K is a constant Problems are intractable if the upper and lower bounds have an exponential factor. –O (N!) –O (N N ) –O (2 N )

Problems that “Cross the Line” The upper bound implies intractable The lower bound implies a tractable Could go either way…

NP-Complete Problems

Problems that Cross the Line What if a problem has: –An exponential upper bound –A polynomial lower bound We have only found exponential algorithms, so it appears to be intractable. But... we can’t prove that an exponential solution is needed, we can’t prove that a polynomial algorithm cannot be developed, so we can’t say the problem is intractable...

NP-Complete Problems The upper bound suggests the problem is intractable The lower bound suggests the problem is tractable The lower bound is linear: O(N) They are all reducible to each other –If we find a reasonable algorithm (or prove intractability) for one, then we can do it for all of them!

Example NP-Complete Problems Path-Finding (Traveling salesman) Map coloring [sub-network frequencies] Scheduling – classes, airlines Matching (bin packing) 2-D arrangement problems Planning problems (pert planning) First-order Predicate Calculus

Class Scheduling Problem With N teachers with certain hour restrictions M classes to be scheduled, can we: –Schedule all the classes –Make sure that no two teachers teach the same class at the same time –No teacher is scheduled to teach two classes at once –No rooms double booked

Determinism vs. Nondeterminism Nondeterministic algorithms produce an answer by a series of “correct guesses” Deterministic algorithms (like those that a computer executes) make decisions based on information.

NP-Complete “NP-Complete” comes from: – Nondeterministic Polynomial – Complete - “Solve one, Solve them all” There are more NP-Complete problems than provably intractable problems.

Proving NP-Completeness Show that the problem is in NP. (i.e. show that there are a finite number of steps, but you don’t know which to take when.) Assume it is not NP complete Show how to convert an existing NPC problem into the problem that we are trying to show is NP Complete (in polynomial time).

Why are we telling you all this? If you are working on a tough problem and you can prove that it’s NP-Complete you can: –a. Take the rest of the day off. –b. Luck out and find a solution –c. Use an approximation algorithm

Decidable vs. Undecidable Problems

Decidable Problems We now have three categories: –Tractable problems –NP-Complete problems –Intractable problems All of the above have algorithmic solutions, even if impractical.

Undecidable Problems No algorithmic solution exists –Regardless of cost –These problems aren’t computable –No answer can be obtained in finite amount of time

The Unbounded Tiling Problem What if we took the tiling puzzle and removed the bounds: –For a given number (T) of different kinds of tiles –Can we arrive at an arrangement that will fill any size area? By removing the bounds, this problem becomes undecidable.

The Halting Problem Given an algorithm A and an input I, will the algorithm reach a stopping place? loop exitif (x = 1) if (even(x)) then x <- x div 2 else x <- 3 * x + 1 endloop In general, we cannot solve this problem in finite time.

A Hierarchy of Problems For a given problem, is there or will there ever be an algorithmic solution? ProblemIn Theory In Application Undecidable NO NO Intractable YES NO Tractable YES YES

Questions?