Analysis of Algorithms COME 355 Introduction

What is an algorithm? An algorithm is a sequence of unambiguous instructions for solving a problem, i.e., for obtaining a required output for any legitimate input in a finite amount of time. “computer” problem algorithm inputoutput Source: Levitin

Algorithm An algorithm is a sequence of unambiguous instructions for solving a problem, i.e., for obtaining a required output for any legitimate input in a finite amount of time.An algorithm is a sequence of unambiguous instructions for solving a problem, i.e., for obtaining a required output for any legitimate input in a finite amount of time.

1-4 What is an algorithm? Recipe, process, method, technique, procedure, routine,… with following requirements:Recipe, process, method, technique, procedure, routine,… with following requirements: 1.Finiteness terminates after a finite number of stepsterminates after a finite number of steps 2.Definiteness rigorously and unambiguously specifiedrigorously and unambiguously specified 3.Input valid inputs are clearly specifiedvalid inputs are clearly specified 4.Output can be proved to produce the correct output given a valid inputcan be proved to produce the correct output given a valid input 5.Effectiveness steps are sufficiently simple and basicsteps are sufficiently simple and basic

Historical Perspective Muhammad ibn Musa al-Khwarizmi – 9 th century mathematicianMuhammad ibn Musa al-Khwarizmi – 9 th century mathematician

1-6 Euclid’s Algorithm Problem: Find gcd(m,n), the greatest common divisor of two nonnegative, not both zero integers m and n Examples: gcd(60,24) = 12, gcd(60,0) = 60, gcd(0,0) = ? Euclid’s algorithm is based on repeated application of equality gcd(m,n) = gcd(n, m mod n) until the second number becomes 0, which makes the problem trivial. Example: gcd(60,24) = gcd(24,12) = gcd(12,0) = 12

1-7 Two descriptions of Euclid’s algorithm Step 1 If n = 0, return m and stop; otherwise go to Step 2 Step 2 Divide m by n and assign the value fo the remainder to r Step 3 Assign the value of n to m and the value of r to n. Go to Step 1. while n ≠ 0 do r ← m mod n m← n m← n n ← r n ← r return m

Other methods for computing gcd(m,n) Consecutive integer checking algorithm Step 1 Assign the value of min{m,n} to t Step 2 Divide m by t. If the remainder is 0, go to Step 3; otherwise, go to Step 4 Step 3 Divide n by t. If the remainder is 0, return t and stop; otherwise, go to Step 4 Step 4 Decrease t by 1 and go to Step 2

1-9 Other methods for gcd(m,n) [cont.] Middle-school procedure Step 1 Find the prime factorization of m Step 2 Find the prime factorization of n Step 3 Find all the common prime factors Step 4 Compute the product of all the common prime factors and return it as gcd(m,n) Is this an algorithm? Example: If M = (11) 4 and N = (11) 1 (17) 2If M = (11) 4 and N = (11) 1 (17) 2 then the greatest common divisor isthen the greatest common divisor is gcd(M,N) = 2 min(4,2) 3 min(0,1) 5 min(2, 4) 7 min(3,3) (11) min(1,4) (17) min(0,2)gcd(M,N) = 2 min(4,2) 3 min(0,1) 5 min(2, 4) 7 min(3,3) (11) min(1,4) (17) min(0,2) = (11) 1 (17) 0 = (11) 1 = (11) 1 (17) 0 = (11) 1

Sieve of Eratosthenes In mathematics, the Sieve of Eratosthenes is a simple, ancient algorithm for finding all prime numbers up to a specified integer. It works efficiently for the smaller primes (below 10 million). It was created by Eratosthenes, an ancient Greek mathematician.In mathematics, the Sieve of Eratosthenes is a simple, ancient algorithm for finding all prime numbers up to a specified integer. It works efficiently for the smaller primes (below 10 million). It was created by Eratosthenes, an ancient Greek mathematician.mathematicsalgorithmprime numbersEratosthenesancient Greekmathematicianmathematicsalgorithmprime numbersEratosthenesancient Greekmathematician A prime number is a natural number which has exactly two distinct natural number divisors: 1 and itself.A prime number is a natural number which has exactly two distinct natural number divisors: 1 and itself.prime numbernatural numberdivisors1prime numbernatural numberdivisors1 AlgorithmAlgorithm To find all the prime numbers less than or equal to a given integer n by Eratosthenes' method:To find all the prime numbers less than or equal to a given integer n by Eratosthenes' method: 1.Create a list of consecutive integers from two to n: (2, 3, 4,..., n). 2.Initially, let p equal 2, the first prime number. 3.Strike from the list all multiples of p less than or equal to n. (2p, 3p, 4p, etc.) 4.Find the first number remaining on the list after p (this number is the next prime); replace p with this number. 5.Repeat steps 3 and 4 until p 2 is greater than n. 6.All the remaining numbers in the list are prime.

1-11 Sieve of Eratosthenes Input: Integer n ≥ 2 Output: List of primes less than or equal to n for p ← 2 to n do A[p] ← p for p ← 2 to  n  do if A[p]  0 //p hasn’t been previously eliminated from the list j ← p * p if A[p]  0 //p hasn’t been previously eliminated from the list j ← p * p while j ≤ n do while j ≤ n do A[j] ← 0 //mark element as eliminated A[j] ← 0 //mark element as eliminated j ← j + p j ← j + p

Example To find all the prime numbers less than or equal to 30, proceed as follows:To find all the prime numbers less than or equal to 30, proceed as follows: First generate a list of integers from 2 to 30:First generate a list of integers from 2 to 30: Strike (sift out) the multiples of 2 resulting in: Strike (sift out) the multiples of 2 resulting in: The first number in the list after 2 is 3; strike the multiples of 3 from the list to get: The first number in the list after 2 is 3; strike the multiples of 3 from the list to get: The first number in the list after 3 is 5; strike the remaining multiples of 5 from the list: The first number in the list after 3 is 5; strike the remaining multiples of 5 from the list: The first number in the list after 5 is 7, but 7 squared is 49 which is greater than 30 so the process is finished. The final list consists of all the prime numbers less than or equal to The first number in the list after 5 is 7, but 7 squared is 49 which is greater than 30 so the process is finished. The final list consists of all the prime numbers less than or equal to 30.

1-13 Example of computational problem: sorting Statement of problem:Statement of problem: Input: A sequence of n numbers Input: A sequence of n numbers Output: A reordering of the input sequence so that a ´ i ≤ a ´ j whenever i so that a ´ i ≤ a ´ j whenever i < j Instance: The sequence Instance: The sequence Algorithms:Algorithms: Selection sortSelection sort Insertion sortInsertion sort Merge sortMerge sort (many others)(many others)

1-14 Selection Sort Input: array a[1],…,a[n]Input: array a[1],…,a[n] Output: array a sorted in non-decreasing orderOutput: array a sorted in non-decreasing order Algorithm:Algorithm: for i =1 to n swap a[i] with smallest of a[i],…a[n]

1-15 Why study algorithms? Theoretical importanceTheoretical importance the core of computer sciencethe core of computer science Practical importancePractical importance A practitioner’s toolkit of known algorithmsA practitioner’s toolkit of known algorithms Framework for designing and analyzing algorithms for new problemsFramework for designing and analyzing algorithms for new problems

1-16 Basic Issues Related to Algorithms How to design algorithmsHow to design algorithms How to express algorithmsHow to express algorithms Proving correctnessProving correctness Efficiency (How to analyze algorithm efficiency?)Efficiency (How to analyze algorithm efficiency?) Theoretical analysisTheoretical analysis Empirical analysisEmpirical analysis OptimalityOptimality

1-17 Algorithm design techniques/strategies Brute forceBrute force Divide and conquerDivide and conquer Decrease and conquerDecrease and conquer Transform and conquerTransform and conquer Space and time tradeoffsSpace and time tradeoffs Greedy approach Dynamic programming Iterative improvement Backtracking Branch and bound

1-18 Analysis of Algorithms How good is the algorithm?How good is the algorithm? CorrectnessCorrectness Time efficiencyTime efficiency Space efficiencySpace efficiency Does there exist a better algorithm?Does there exist a better algorithm? Lower boundsLower bounds OptimalityOptimality

1-19 Fundamental data structures list list arrayarray linked listlinked list stringstring stack stack queue queue priority queue priority queue graph tree set and dictionary

1-20 Some Typical Problems SortingSorting SearchingSearching Shortest paths in a graphShortest paths in a graph Minimum spanning treeMinimum spanning tree Primality testingPrimality testing Traveling salesman problemTraveling salesman problem Knapsack problemKnapsack problem ChessChess Towers of HanoiTowers of Hanoi Program terminationProgram termination

Some Typical Problems Fourier Transform Input: A sequence of n real or complex values h_i, 0 <= i <= n-1, sampled at uniform intervals from a function h. Problem: Compute the discrete Fourier transform H of h Nearest Neighbor Input: A set S of n points in d dimensions; a query point q. Input: A set S of n points in d dimensions; a query point q. Problem: Which point in S is closest to q? Problem: Which point in S is closest to q? SOURCE: Steve Skiena’s Algorithm Design Manual + Prof. K. Daniels (for problem descriptions, see graphics gallery at ) (for problem descriptions, see graphics gallery at Shortest Path Input: Edge-weighted graph G, with start vertex s and end vertex t Problem: Find the shortest path from s to t in G Bin Packing Input: A set of n items with sizes d_1,...,d_n. A set of m bins with capacity c_1,...,c_m. Problem: How do you store the set of items using the fewest number of bins?

Some Typical Problems Transitive Closure Input: A directed graph G=(V,E). Problem: Construct a graph G'=(V,E') with edge (i,j) in E' iff there is a directed path from i to j in G. For transitive reduction, construct a small graph G'=(V,E') with a directed path from i to j in G' iff (i,j) in E. Convex Hull Input: A set S of n points in d- dimensional space. Problem: Find the smallest convex polygon containing all the points of S. Problem: Find the smallest convex polygon containing all the points of S. Eulerian Cycle Input: A graph G=(V,E). Problem: Find the shortest tour of G visiting each edge at least once. Edge Coloring Input: A graph G=(V,E). Problem: What is the smallest set of colors needed to color the edges of E such that no two edges with the same color share a vertex in common?

Some Typical Problems Hamiltonian Cycle Input: A graph G=(V,E). Problem: Find an ordering of the vertices such that each vertex is visited exactly once. Clique Input: A graph G=(V,E). Problem: What is the largest S that is a subset of V such that for all x,y in S, (x,y) in E?

Tools of the Trade: Core Material Algorithm Design Patterns Algorithm Design Patterns dynamic programming, linear programming, greedy algorithms, approximation algorithms, randomized algorithms, sweep algorithms, (parallel algorithms) dynamic programming, linear programming, greedy algorithms, approximation algorithms, randomized algorithms, sweep algorithms, (parallel algorithms) Advanced Analysis Techniques Advanced Analysis Techniques amortized analysis, probabilistic analysis amortized analysis, probabilistic analysis Theoretical Computer Science principles Theoretical Computer Science principles NP-completeness, NP-hardness NP-completeness, NP-hardness Asymptotic Growth of Functions Summations Recurrences Sets Probability MATH Proofs Calculus Combinations Logarithms Number Theory Geometry Trigonometry Complex Numbers Permutations Linear Algebra Polynomials

Textbook Required: Introduction to AlgorithmsIntroduction to Algorithms by T.H. Corman, C.E. Leiserson, R.L. Rivestby T.H. Corman, C.E. Leiserson, R.L. Rivest McGraw-HillMcGraw-Hill ISBN ISBN nd Edition