Analysis of Algorithms: Methods and Examples CSE 2320 – Algorithms and Data Structures Alexandra Stefan Based on presentations by Vassilis Athitsos and Bob Weems University of Texas at Arlington Updated 2/3/2016 1

Reading CLRS: Chapter 3 (all) – strongly encourraged Sedgewick: Chapter 2 (except for 2.5 – Basic recurrences) 2

Counting instructions Count in detail the total number of instructions executed by each of the following pieces of code: // Example A. Notice the ; at the end of the for loop. for (i = 0; i <N; i++) ; // Example B (source: Dr. Bob Weems) for (i=0; i<N; i++) for (j=0; j<N; j++) { c[i][j] = 0; for (k=0; k<N; k++) c[i][j] += a[i][k]*b[k][j]; }

Counting instructions – continued // Example C (source: Dr. Bob Weems) T(N) = … for (i = 0; i<N; i++) for(j=1; j<N; j = j+j) printf("A"); // Example D. T(N) = …. for (i = 0; i<N; i++) for(j=1; j<N; j = j*2) printf("A"); // Example E. T(N) = …. for (i = 0; i<N; i++) for(j=N; j>=1; j = j/2) printf("A"); // Example F. T(N) = …. for (i = 0; i<N; i++) for(j=1; j<N; j = j+2) printf("A"); 4

Estimate runtime Problem: The total number of instructions in a program (or a piece of a program) is and the computer it runs on executes 10 9 instructions per second. How long will it take to run this program? Give the answer in seconds. If it is very large, transform it in larger units (hours, days, years). Summary: – Total instructions: – Speed: 10 9 instructions/second Answer: – Time = (total instructions)/speed = (10 12 instructions) / (10 9 instr/sec) = 10 3 seconds ~ 15 minutes Note that this computation is similar to computing the time it takes to travel a certain distance ( e.g. 120miles) given the speed (e.g. 60 miles/hour). 5

Estimate runtime A slightly different way to formulate the same problem: – total number of instructions in a program (or a piece of a program) is and – the computer it runs on executes one instruction in one nanosecond (10 -9 seconds) – How long will it take to run this program? Give the answer in seconds. If it is very large, transform it in larger units (hours, days, years) Summary: – total instructions – seconds per instruction Answer: – Time = (total instructions) * (seconds per instruction) = (10 12 instructions)* (10 -9 sec/instr) = 10 3 seconds ~ 15 minutes 6

Motivation for Big-Oh Notation Scenario: a client requests an application for sorting his records. Which one of the 3 sorting algorithms that we discussed will you implement? – Selection sort – Insertion sort – Merge sort 7

Comparing algorithms Comparing linear, N lg N, and quadratic time. Quadratic time algorithms become impractical (too slow) much faster than linear and N lg N time algorithms. Of course, what we consider "impractical" depends on the application. – Some applications are more tolerant of longer running times. 8 NN log NN2N (1 million)about 20 million10 12 (one trillion) 10 9 (1 billion)about 30 billion10 18 (one quintillion) (1 trillion)about 40 trillion10 24 (one septillion)

Motivation for Big-Oh Notation Given an algorithm, we want to find a function that describes the time performance of the algorithm. Computing the number of instructions in detail is NOT desired: – It is complicated and the details are not relevant – The number of machine instructions and runtime depend on factors other than the algorithm: Programming language Compiler optimizations Performance of the computer it runs on (CPU, memory) There are some details that we would actually NOT want this function to include, because they can make a function unnecessarily complicated. When comparing two algorithms we want to see which one is better for very large data – asymptotic behavior – It is not very important what happens for small size data. The Big-Oh notation describes the asymptotic behavior and greatly simplifies algorithmic analysis. 9

Asymptotic Notation Goal: we want to be able to say things like: – Selection sort will take time proportional to N 2 Θ(N 2 ) – Insertion sort will take time at most proportional N 2 O(N 2 ) proportional to N when the data is already sorted – Any sorting algorithm will take time at least proportional to N Ω(N) Math functions that are: – Θ(N 2 ) : – O(N 2 ) : – Ω(N 2 ) : 10

11

Big-Oh 12

Asymptotic Bounds and Notation (CLRS chapter 3) 13

Abuse of notation 14 Instead of : We may use:

Theta vs Big-Oh The Theta notation is stricter than the Big-Oh notation: – We can say that N 2 = O(N 100 ). – We cannot say that N 2 = Θ(N 100 ). 15

Simplifying Big-Oh Notation Suppose that we are given this running time: f(N) = 35N N + lg(N) How can we express f(N) in Big-Oh notation? 16

Simplifying Big-Oh Notation Suppose that we are given this running time: f(N) = 35N N + lg(N) How can we express f(N) in Big-Oh notation? Typically we say that f(N) = O(N 2 ). The following are also correct, but unnecessarily complicated, and thus less useful, and rarely used. – f(N) = O(N 2 ) + O(N). – f(N) = O(N 2 ) + O(N) + O(lgN) + O(1). – f(N) = O(35N N + lg(N) ). 17

Simplifying Big-Oh Notation Suppose that we are given this running time: f(N) = 35N N + lg(N) We say that g(N) = O(N 2 ). Why is this mathematically correct? – Why can we ignore the non-quadratic terms? Ans 1: Using the Big-Oh definition: we can find an N 0 and c such that, for all N ≥ N 0 : f(N) ≤ cN 2. – If you don't believe this, do the calculations for practice. Use: c = 38, N 0 = 41 ( or N 0 = 1536) f(N) = 35N N + lg(N) ≤ 38N 2, for all N ≥ 41 18

Polynomial functions If f(N) is a polynomial function, then it is Θ of the dominant term. 19

Properties of O, Ω and Θ 20

Using Limits 21

Using Limits 22

Using Limits: An Example 23

Using Limits: An Example 24

Using Limits: An Example 25

Big-Oh Hierarchy 26

Big-Oh Transitivity 27

Big-Oh Transitivity 28

Using Substitutions 29

Using Substitutions 30

Example Problem 1 31

Example Problem 2 32

Asymptotic notation for two parameters (CLRS) f(N,M) is O(g(N,M)) if there exist constants c 0, N 0 and M 0 such that: f(N,M) ≤ c 0 g(N,M) for all pairs (N,M) s.t. either N ≥ N 0 or M ≥ M 0 33

Useful logarithm properties 34

Summary 35