1. CIS265/506 Cleveland State University – Prof. Victor Matos
Chapter Developing Efficient Algorithms
CIS265/506 Cleveland State University – Prof. Victor Matos
Adapted from: Introduction to Java Programming: Comprehensive Version, Eighth Edition by Y. Daniel Liang

2. Executing Time Suppose two algorithms perform the same task such as
search (linear search vs. binary search) and
sorting (selection sort vs. insertion sort).
Which one is better?
One possible approach to answer this question is to implement these algorithms in Java and run the programs to get execution time. But there are two problems for this benchmarking approach:
First, there are many tasks running concurrently on a computer. The execution time of a particular program is dependent on the system load.
Second, the execution time is dependent on specific input. Consider linear search and binary search for example. If an element to be searched happens to be the first in the list, linear search will find the element quicker than binary search. 2

3. Growth Rate
It is very difficult / inaccurate to compare algorithms by measuring their execution time.
To overcome these problems, a theoretical approach was developed to analyze algorithms independent of computers and specific input.
This approach provides an approximated description of how a change on the size of the input affects the number of statements to be executed.
In this way, you can see how fast an algorithm’s execution time increases as the input size increases, so you can compare two algorithms by examining their growth rates. 3

4. Analysis of Algorithms
Google search on ‘Donald Knut’ processed on April 2, 2013 4

5. F (N : input size) ⟶ ( N: number of executed statements )
Growth Rate
Goal :
Estimate the execution ‘time’ of an algorithm in relation to the input size.
F (N : input size) ⟶ ( N: number of executed statements )
Algorithm_Performance = O( f(n) )
Note
Computer scientists use the Big O notation to abbreviate for “order of magnitude.” 5

6. Big O Notation
Consider linear search. The linear search algorithm compares the key with the elements in the array sequentially until the key is found or the array is exhausted.
If the key is not in the array, it requires n comparisons for an array of size n.
If the key is in the array, it requires n/2 comparisons on average.
The algorithm’s execution time is proportional to the size of the array.
If you double the size of the array, you will expect the number of comparisons to double.
The algorithm grows at a linear rate. 6

7. Big O Notation Observations
The growth rate of the Linear Search algorithm has an order of magnitude of n.
Using this notation, the complexity of the linear search algorithm is O(n), pronounced as “order of n” or “linear on n” 7

8. Best, Worst, and Average Cases
For the same input size, an algorithm’s execution time may vary, depending on the input.
An input that results in the shortest execution time is called the best-case input and
An input that results in the longest execution time is called the worst-case input.
Worst-case analysis is very useful. You can show that the algorithm will never be slower than the worst-case.
The analysis is generally conducted for the worst-case. 8

9. Ignoring Multiplicative Constants
The linear search algorithm requires n comparisons in the worst-case and n/2 comparisons in the average-case. Both cases require O(n) time. The multiplicative constants have no impact on growth rates. The growth rate for n/2 or 100n is the same as n, i.e., O(n) = O(n/2) = O(100n).
f(n) n
n / 2
100 n
100 50
10000
200
20000 2
f( xi+1 ) / f( xi ) ⟶
f(200) / f(100)
f(100) / f(50)
f(20000) / f(10000)

10. Ignoring Non-Dominating Terms
Consider the algorithm for finding the maximum number in an array of n elements.
The Big O notation allows you to ignore the non-dominating parts of the function and highlight its most important component.
For example consider the case: f(n) = 4n + 2;
You should discard the additive and multiplicative factors so f(n)≈n
Therefore, the complexity of this algorithm is O(n). 10

11. Ignoring Non-Dominating Terms
Consider the algorithm for finding the maximum number in an array of n elements.
int largest = a[0];
int i = 1;
while ( i < a.length ) {
if ( a[i] > largest ) {
largest = a[i]; }
i++;
Max number of statements executed in this fragment is:
* N ⟶ f(n) = 4n ⟶ O(n)
Therefore if the array’s length is n
n = 1000 f(n) = 4002
n = 2000 f(n) = 8002
n = 3000 f(n) = 12002
n = 4000 f(n) = 16002
...
Complexity of this algorithm is O(n). 11

12. Useful Mathematical Summations (to be used later)12

13. Examples: Determining Big-O
OPTIONS: (1) Time Complexity, find T(n) estimating number of instructions
(2) Space Complexity, find S(n) estimating space needed to work.
Time Complexity is the most commonly used expression of effectiveness.
Patterns to look for
Repetition
Sequence
Selection
Logarithm 13

14. Examples: Determining Big-O
Timing: Relating instructions to CPU clock-cycles on venerable Intel-8080 14

15. Repetition: Simple Loops
for (i = 1; i <= n; i++) {
k = k + 5; }
executed
n times
constant time
Time Complexity
T(n) = (a constant c) * n = cn ⟶ O(n)
Ignore multiplicative constants (e.g., “c”). 15

16. Repetition: Nested Loops
for (i = 1; i <= n; i++) {
for (j = 1; j <= n; j++) {
k = k + i + j; }
executed
n times
inner loop
executed
n times
constant time
Time Complexity
T(n) = (a constant c) * n * n = cn2 ⟶ O(n2)
Ignore multiplicative constants (e.g., “c”). 16

17. Repetition: Nested Loops
for (i = 1; i <= n; i++) {
for (j = 1; j <= i; j++) {
k = k + i + j; }
executed
n times
inner loop
executed
i times
constant time
Time Complexity
T(n) = c + 2c + 3c + 4c + … + nc = c n(n+1)/2
=(c/2)n2 + (c/2)n ⟶ O(n2) quadratic algorithm
Ignore non-dominating terms
Ignore multiplicative constants 17

18. Repetition: Nested Loops
for (i = 1; i <= n; i++) {
for (j = 1; j <= 20; j++) {
k = k + i + j; }
executed
n times
inner loop
executed
20 times
constant time
Time Complexity
T(n) = 20 * c * n ⟶ O(n)
Ignore multiplicative constants (e.g., 20*c) 18

19. Sequence executed 10 times executed inner loop n times executed
for (j = 1; j <= 10; j++) {
k = k + 4; }
executed
10 times
for (i = 1; i <= n; i++) {
for (j = 1; j <= 20; j++) {
k = k + i + j; }
executed
n times
inner loop
executed
20 times
Time Complexity
T(n) = c * * c * n ⟶ O(n) 19

20. Selection O(n) Time Complexity
if (list.contains(e)) {
System.out.println(e); }
else
for (Object t: list) {
System.out.println(t);
Constant c
Let n be
list.size().
Loop : O(n)
Time Complexity
T(n) = test time (condition) + worst-case (then/ else)
= O(n) + O(n)
⟶ O(n) 20

21. Constant Time
The Big O notation estimates the execution time of an algorithm in relation to the input size.
If the time is not related to the input size, the algorithm is said to take constant time with the notation O(1).
Example:
A method that retrieves an element at a given index in an array takes constant time, because it does not grow as the size of the array increases. 1 … i
n-1
Data [i] ⟵ Extract From ( Array_Memory_Loc + (i-1)*cellSize ) 21

22. Logarithm: Analyzing Binary Search
Each iteration in the Binary Search algorithm contains a fixed number of operations, denoted by c (see next listing)
Continue on Lower Half or Continue on Upper Half 1 2 3 …
n/2
n-2
n-1 n
low middle high 22

23. Logarithm: Analyzing Binary Search
Let T(n) denote the time complexity for a binary search on a list of n elements.
Without loss of generality, assume n is
a power of 2 ( n = 2k ) and k=log(n).
Since binary search eliminates half
of the input after two comparisons we have… 23

24. Logarithm: Analyzing Binary Search
public class BinarySearch { /** Use binary search to find a key in the list */ public static int binarySearch(int[] list, int key) { int low = 0; int high = list.length - 1; while (high >= low) { int mid = (low + high) / 2; if (key < list[mid]) high = mid - 1; else if (key == list[mid]) return mid; else low = mid + 1; } return -low - 1; // Now high < low 24

25. Logarithmic Time
Ignoring constants and smaller terms, the complexity of the binary search algorithm is O( log(n) ).
It is called a logarithmic algorithm.
Note
The base of the log is 2, but the base does not affect a logarithmic growth rate, so it can be omitted.
The logarithmic algorithm grows slowly as the problem size increases. If you square the input size, you only double the time for the algorithm. 25

26. President Obama & the Bubble Sort

27. Analyzing Selection Sort
SelectionSort (see next listing), finds the largest number in the list and places it last. It then finds the largest number remaining and places it next to last, and so on until the list contains only a single number.
4. Exclude last, continue exploration on remaining elements 1 2 3 …
n-2
n-1 n
1. Find Largest element Place largest
element here
3. Exchange if necessary
temp 27

28. Analyzing Selection Sort
The number of comparisons is n-1 for the first iteration, n-2 for the second iteration, and so on.
Let T(n) denote the complexity for selection sort and c denote the total number of other operations such as assignments and additional comparisons in each iteration.
Ignoring constants and smaller terms, the complexity of the selection sort algorithm is O(n2). 28

29. Analyzing Selection Sort
public class SelectionSort { public static void selectionSort(double[] list) { for (int i = 0; i < list.length - 1; i++) { // Find the minimum in the list[i..list.length-1] double currentMin = list[i]; int currentMinIndex = i; for (int j = i + 1; j < list.length; j++) { if (currentMin > list[j]) { currentMin = list[j]; currentMinIndex = j; } // Swap list[i] with list[currentMinIndex] if necessary; if (currentMinIndex != i) { list[currentMinIndex] = list[i]; list[i] = currentMin;

30. Quadratic Time
An algorithm with the O(n2) time complexity is called a quadratic algorithm.
A quadratic algorithm grows quickly as the problem size increases. If you double the input size, the time for the algorithm is quadrupled.
Algorithms with a double-nested loop are often quadratic.
n2 Quadratic
n Linear 30

31. Analyzing Insertion Sort
The insertion sort algorithm (see next listing) sorts a list of values by repeatedly inserting a new element into a sorted partial array until the whole array is sorted.
Add 10 10
Add 30 30
Add 25 25
Add 7 7
2 times each?. At step k you ask k-1 questions and make k-1 shifts therefore 2(k-1) expensive ops are performed plus additional c work. 31

32. Analyzing Insertion Sort
At the kth iteration, to insert an element into an array of size k, it may take k comparisons to find the insertion position, and k moves to insert the element.
Let T(n) denote the complexity for insertion sort and c denote the total number of other operations such as assignments and additional comparisons in each iteration. So,
2 times each?. At step k you ask k-1 questions and make k-1 shifts therefore 2(k-1) expensive ops are performed plus additional c work. 32

33. Analyzing Insertion Sort
public static void insertionSort(double[] list) { for (int i = 1; i < list.length; i++) { // insert list[i] into a sorted sublist list[0..i-1] // so that list[0..i] is sorted. double currentElement = list[i]; int k; for (k = i - 1; k >= 0 && list[k] > currentElement; k--) { list[k + 1] = list[k]; } // Insert the current element into list[k+1] list[k + 1] = currentElement; 33

34. Analyzing Towers of Hanoi
The Towers of Hanoi (Listing 19.7) moves n disks from tower A to tower B with the assistance of tower C recursively as follows:
Move the first n – 1 disks from A to C with the assistance of tower B.
Move disk n from A to B.
Move n - 1 disks from C to B with the assistance of tower A.
Let T(n) denote the complexity for the algorithm that moves disks and c denote the constant time to move one disk, i.e., T(1) is c. So, 34

35. Analyzing Towers of Hanoi35

36. Analyzing Towers of Hanoi
Exponential
Performance 2n 36

37. Comparing Common Growth Functions
Constant time
Logarithmic time
Linear time
Log-linear time
Quadratic time
Cubic time
Exponential time 37

38. Case Study: Fibonacci Numbers
public static long fib(long index) { if (index == 0) // Base case return 0; else if (index == 1) // Base case return 1; else // Reduction and recursive calls return fib(index - 1) + fib(index - 2); }
Finonacci series: …
indices:
fib(0) = 0;
fib(n) fib(1) = 1;
fib(n) = fib(n -1) + fib(n -2); n >=2 38

39. Complexity for Recursive Fibonacci Numbers
Since
and
Therefore, the recursive Fibonacci method takes 39

40. Case Study: Non-recursive version of Fibonacci Numbers
public static long fib(long n) { long f0 = 0; // For fib(0) long f1 = 1; // For fib(1) long f2 = 1; // For fib(2) if (n == 0) return f0; else if (n == 1) return f1; else if (n == 2) return f2; for (int i = 3; i <= n; i++) { f0 = f1; f1 = f2; f2 = f0 + f1; }
Obviously, the complexity of this new algorithm is O(n)
This is a tremendous improvement over the recursive algorithm. 40

41. Case Study: GCD Algorithms Version 1
public static int gcd(int m, int n) { int gcd = 1; for (int k = 2; k <= m && k <= n; k++) { if (m % k == 0 && n % k == 0) gcd = k; } return gcd;
The complexity of this algorithm is O(n)
Assume m ≥ n 41

42. Case Study: GCD Algorithms Version 2A
for (int k = n; k >= 1; k--) { if (m % k == 0 && n % k == 0) { gcd = k; break; }
The worst-case time complexity of this algorithm is still O(n).
Assume m ≥ n 42

43. Case Study: GCD Algorithms Version 2B (Improved variation)
public static int gcd(int m, int n) { int gcdResult = 1; if (m == n) return m; for (int k = m/2; k >= 1; k--) { if (m % k == 0 && n % k == 0) { gcdResult = k; break; } return gcdResult;
The worst-case time complexity of this algorithm is still O(n).
Assume m ≥ n 43

44. Euclid’s Algorithm The time complexity of this algorithm is O(log(n)).
public static int gcd(int m, int n) { if (m % n == 0) return n; else return gcd(n, m % n); }
The time complexity of this algorithm is O(log(n)). 44

45. Euclid’s Algorithm Worst Case Analysis. Assume m ≥n
If n ≤ m/2 then m%n < m/2,
since the remainder of m divided by n is always less than n
If n > m/2 then m%n = m-n < m/2. Therefore m%n ≤ m/2
The argument n passed to the gcd method is reduced by half every two interactions.
After invoking gcd two times, second param is less than n/2
After invoking gcd four times, second param is less than n/22
After invoking gcd six times, second param is less than n/23
After invoking gcd k times, second param is less than n/2(k/2)

46. Euclid’s Algorithm Observation
The worst case scenario occurs when the two numbers m and n produce the most divisions
It turns out that two consecutive Fibonacci numbers will result in the most divisions (0, 1, 1, 2, 3, 5, 8, 13, 21,…)
Example. Observe that k ≤ 2log(13)=2*3.7 ⋍ 7 (# of invocations)
gcd(21, 13)
= gcd(13, 8)
= gcd(8, 5)
= gcd(5, 3)
= gcd(3, 2)
= gcd(2, 1) =1 46

47. Practical Considerations
The big O notation provides a good theoretical estimate of algorithm efficiency.
However, two algorithms of the same time complexity are not necessarily equally efficient.
As shown in the preceding example, both gcd algorithms Version_1 and Version_2B have the same complexity,
but the one in Version_2B is obviously better practically. 47

48. Prime Numbers Version2 – A Better Algorithm O(√n) Proof.
We can prove that if n is not a prime, n must have a factor that is greater than 1 and less than or equal to √ n.
Proof.
Since n is not a prime, there exist two numbers p and q such that n = pq with 1 < p ≤ q. Note that n = √ n √ n. Therefore p must be less than or equal to √ n. Hence, you need to check only whether n is divisible by 2, 3, 4, 5, … or √ n. If not, n is prime.
This significantly reduces the time complexity of the algorithm to O(√n).
Modify previous code as follows: 48

49. Prime Numbers Listing 23.5 O(√ n) 49 public class DriverPrime1 {
public static void main(String[] args) {
final int NUMBER_OF_PRIMES = 50; // Number of primes to display
final int NUMBER_OF_PRIMES_PER_LINE = 10; // Display 10 per line
int count = 0; // Count the number of prime numbers
int number = 2; // A number to be tested for primeness
System.out.println("The first 50 prime numbers are \n");
while (count < NUMBER_OF_PRIMES) {
boolean isPrime = true;
for (int divisor = 2; divisor <= Math.sqrt(number); divisor++) {
if (number % divisor == 0) { // If true, number is not prime
isPrime = false; // Set isPrime to false
break; // Exit the for loop }
if (isPrime) {
count++; // Increase the count
if (count % NUMBER_OF_PRIMES_PER_LINE == 0) {
System.out.println(number);
else
System.out.printf("%6d \t", number );
number++;
O(√ n) 49

50. Prime Numbers Version3 – Sieve of Eratosthenes – O(n √n / log (n) )
This algorithm uses an array named primes of n Boolean values.
Initially, all elements in primes are set true.
Since the multiples of 2 are not prime, set primes[2 *i ] to false.
Since the multiples of 3 are not prime, set primes[3 * i ] to false.
Since the multiples of 5 are not prime, set primes[5 * i ] to false.
You only need to consider the multiples of a prime number k = 2, 3, 5, 7, 11, … and set the corresponding element in primes to false.
Afterward, if primes[i ] is still true, then i is a prime number. 50

51. Prime Numbers Version3 – Sieve of Eratosthenes - O(n √n / log (n) )1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 26 27
Initial T
K=2 F
K=3
K=5
Skip sequences starting with a value marked ‘F’ (non-prime):
4, 6, 8, 9, 10, 12 51

52. Prime Numbers – Sieve of Eratosthenes
public class DriverPrime3 {
// PrimeNumber - Listing Sieve of Eratosthenes O( n*sqrt(n) / log(n)
public static void main(String[] args) {
final int N = 50; // Number of primes to display
boolean[] primes = new boolean[N + 1];
System.out.printf("The first %d prime numbers are \n", N);
for (int i = 0; i < N; i++)
primes[i] = true;
showPrimes(primes);
for (int k = 2; k < N / k; k++) {
if (primes[k]) {
for (int i = k; i < N / k; i++) {
primes[k * i] = false; }
System.out.println("k=" + k);
}// main
private static void showPrimes(boolean[] primes) {
for (int i = 1; i < primes.length; i++) {
System.out.printf("\t%6d: %6s, ", i, primes[i]);
if (i % 10 == 0)
System.out.println();