eleg667/2001-f/Topic-1a 1 A Brief Review of Algorithm Design and Analysis

eleg667/2001-f/Topic-1a2 Outline  Basic concepts;  Case study;  Analyzing algorithms  Comparing algorithms  Problem complexity

eleg667/2001-f/Topic-1a3 Algorithm - Definition Sequence of steps to carry out a particular task Examples: Recipe; Assembly instructions; ……

eleg667/2001-f/Topic-1a4 Characteristics of Algorithms  Algorithms are precise. Each step has a clearly defined meaning;  Algorithms are effective. The task is always done as required;  Algorithms have a finite number of steps;  Algorithms must terminate.

eleg667/2001-f/Topic-1a5 Any algorithm can be stated using the following logic constructs  Sequence: steps must follow each other in a logical sequence;  Decision: there may be alternative steps that may be taken subject to a particular condition;  Repetition: certain steps may be repeated while, or until, a certain condition is true.

eleg667/2001-f/Topic-1a6 Why study algorithms? A motivating example…

eleg667/2001-f/Topic-1a7 Example: sorting playing cards Start with one card on the left hand and the others cards facing down on the table; while there are cards left on the table do { take one card from the table; compare it with each of the cards already in the hand, from left to right, inserting it into the right position; } Sequence Decision Repetition Insertion Sort

eleg667/2001-f/Topic-1a8 Expressing computer algorithms  It should be expressed in a language more precise, less ambiguous, and more compact than a “natural language” such as English;  Algorithms are usually written in a pseudocode language and later translated to a real programming language.

eleg667/2001-f/Topic-1a9 Insertion Sort – An Intuitive Picture j QK 5 … Q i key A A1A1 …

eleg667/2001-f/Topic-1a10 Insertion Sort in Pseudocode Insertion-Sort(A) ; A is an array of numbers for j = 2 to length(A) key = A[j] i = j - 1 while i > 0 and A[i] > key A[i+1] = A[i] i = i - 1 A[i+1] = key SequenceRepetition Decision

eleg667/2001-f/Topic-1a11 Algorithms and Machine Models  Sequential algorithms and machine models  Parallel algorithms and machine models

eleg667/2001-f/Topic-1a12 Analyzing algorithms  Estimate the running time of algorithms; = F(Problem Size) = F(Input Size) = number of primitive operations used (add, multiply, compare etc)

eleg667/2001-f/Topic-1a13 Analysis for Insertion Sort Insertion-Sort(A)CostTimes ( Iterations ) 1 for j = 2 to length(A) { c 1 2 key = A[j] c 2 3 i = j – 1 c 3 4 while i > 0 and A[i] > key c 4 5 A[i+1] = A[i] c 5 6 i = i – 1 c 6 7 A[i+1] = key c 7 } n n - 1

eleg667/2001-f/Topic-1a14 Insertion Sort Analysis (cont.) Best Case: Array already sorted, tj = 1 for all j

eleg667/2001-f/Topic-1a15 Insertion Sort Analysis (cont.) Worst Case: Array in reverse order, tj = j for all j Note that

eleg667/2001-f/Topic-1a16 Insertion Sort Analysis (cont.) Average Case: Check half of array on average, tj = j/2 for all j T(n) = a.n 2 + bn + c (quadractic in n) We are usually interested in the worst-case running time

eleg667/2001-f/Topic-1a17 Insertion Sort demo…

eleg667/2001-f/Topic-1a18 Growth rates  constant growth rate: T(n) = c  linear growth rate : T(n) = c*n  logarithmic growth rate : T(n) = c*log n  quadratic growth rate : T(n) = c*n 2  exponential growth rate : T(n) = c*2 n

eleg667/2001-f/Topic-1a19 Asymptotic Analysis  Ignoring constants in T(n)  Analyzing T(n) as n "gets large" Example: The big-oh (O) Notation

eleg667/2001-f/Topic-1a20 Formally… T(n) = O(f(n)) if there are constants c and n such that T(n) < c.f(n) when n  n 0 c.f(n) T(N) n0n0 n f(n) is an upper bound for T(n)

eleg667/2001-f/Topic-1a21 O ( big-oh)  Describes an upper bound for the running time of an algorithm Upper bounds for Insertion Sort running times: worst case: O(n 2 ) T(n) = c 1.n 2 + c 2.n + c 3 best case: O(n) T(n) = c 1.n + c 2 average case: O(n 2 ) T(n) = c 1.n 2 + c 2.n + c 3 Time Complexity

eleg667/2001-f/Topic-1a22 Some properties of the O notation  Fastest growing function dominates a sum O(f(n)+g(n)) is O(max{f(n), g(n)})  Product of upper bounds is upper bound for the product If f is O(g) and h is O(r)  then fh is O(gr)  f is O(g) is transitive If f is O(g) and g is O(h) then f is O(h)  Hierarchy of functions O(1), O(logn), O(n 1/2 ), O(nlogn), O(n 2 ), O(2 n ), O(n!)

eleg667/2001-f/Topic-1a23 Times on a 1-billion-steps-per-second computer

eleg667/2001-f/Topic-1a24  Simple statement sequence s 1 ; s 2 ; …. ; s k O(1) as long as k is constant  Simple loops for(i=0;i<n;i++) { s; } where s is O(1) Time complexity is n O(1) or O(n) Analyzing an Algorithm

eleg667/2001-f/Topic-1a25 Analyzing an Algorithm (cont.)  Nested loops for(i=0;i<n;i++) for(j=0;j<n;j++) { s; } Complexity is n O(n) or O(n 2 )  Recursion Search(n) if middle_element = key return it else if it’s greater search_left(n/2) else search_right(n/2) Solve recurrence equation: T(n) = T(1) + T(n/2) Complexity is O(log 2 n)

eleg667/2001-f/Topic-1a26 Reasonable and Unreasonable Algorithms  Polynomial Time algorithms An algorithm is said to be polynomial if it is O( n c ), c >1 Polynomial algorithms are said to be reasonable They solve problems in reasonable times!  Exponential Time algorithms An algorithm is said to be exponential if it is O( r n ), r > 1 Exponential algorithms are said to be unreasonable

eleg667/2001-f/Topic-1a27 Another example: Quick Sort  Uses recursion to repetitively divide list in 1/2 and sort the two halves  QuickSort partitions the list into a left and a right sublist, where all values in the left sublist are less than all values in the right sublist. Then the QuickSort is applied recursively to the sublists until each sublist has only one element.

eleg667/2001-f/Topic-1a28 Quick Sort (cont.) becomes becomes

eleg667/2001-f/Topic-1a29 Quick Sort (cont.) QuickSort (list(0,n)): If length (list) > 1 then partition into 2 sublists from 0..split-1 and from split + 1..n QuickSort (list (0,split-1)) QuickSort (list (split + 1, n)

eleg667/2001-f/Topic-1a30 1. Pick an array value, pivot. 2. Use two indices, i and j. a. Begin with i = left and j =right + 1 b. As long as i is less than j do 3 steps: i. Keep increasing i until we come to an element A[i] >= pivot. ii. Keep decreasing j until we come to an element A[j] <= pivot. iii. Swap A[i] and A[j]. Quick Sort – Partition phase

eleg667/2001-f/Topic-1a31 The Operation of Quicksort

eleg667/2001-f/Topic-1a32 Quick Sort – Complexity  Assume the list size (n) is a power of 2 and that pivot is always chosen such that it divides the list into two equal pieces  At each step we are cutting in half the size to be sorted  O(nlog 2 n)

eleg667/2001-f/Topic-1a33 Quick Sort demo…

eleg667/2001-f/Topic-1a34 Comparing the Sort Algorithms…

eleg667/2001-f/Topic-1a35 Problem complexity problems sort... InsertionShellsort Quicksort Algorithms:

eleg667/2001-f/Topic-1a36 Definitions  A problem’s upper bound refers to the best that we know how to do. It is determined by the best algorithmic solution that we have found for a given problem. Example: For the sorting problem O(n.logn) is the upper bound.

eleg667/2001-f/Topic-1a37 Definitions (cont.)  A problem’s lower bound refers to the best algorithmic solution that is theoretically possible. Example: It can be proved that sorting an unsorted list cannot be done in less than O(n.log n), unless we make assumptions about the particular input data. Thus O(n.log n) is the lower bound for sorting problems.

eleg667/2001-f/Topic-1a38 Definitions (cont.)  Upper bounds tell us the best we’ve been able to do so far;  Lower bounds tell us the best we can hope to do. For a given problem, if the upper and lower bounds are the same, then we refer to that problem as a closed problem.

eleg667/2001-f/Topic-1a39 Tractable and Intractable problems  Tractable problems have both upper and lower bounds polynomial – O(logn), O(n), O(n.logn), O(n k );  Intractable problems have both upper and lower bounds exponential - O(2 n ), O(n!), O(n n );

eleg667/2001-f/Topic-1a40 Problems that cross the line  We have found only exponential solutions (i.e. from the standpoint of our algorithms, it appears to be intractable);  We cannot prove the necessity of an exponential solution (i.e. from the standpoint of our proofs, we cannot say that it is intractable).

eleg667/2001-f/Topic-1a41 NP-complete problems  The upper bound suggests that the problem is intractable;  No proof exists that shows these problems are intractable;  Once a solution is found, it can be checked in polynomial time;  They are all reducible to one another;

eleg667/2001-f/Topic-1a42 Example  Traveling salesman: give a weighted graph (e.g., cities with distances between them) and some distance k (e.g., the maximum distance you want to travel), is there some tour that visits all the points (e.g., cities) and returns home such that distance  k?  …

eleg667/2001-f/Topic-1a43 P = NP ? Win the Turing Award and get a $1 million prize!