1. CS-2852 Data Structures LECTURE 3A Andrew J. Wozniewicz Image copyright © 2010 andyjphoto.com

2. CS-2852 Data Structures, Andrew J. Wozniewicz Agenda Algorithm Efficiency – Big-O Notation – Big-Omega Notation – Polynomial versus Exponential Algorithms – Tractable and Intractable Problems

3. CS-2852 Data Structures, Andrew J. Wozniewicz Perpetuum Mobile

4. CS-2852 Data Structures, Andrew J. Wozniewicz Perpetuum Mobile Perpetual motion machine. Impossible, due to the 1 st and 2 nd laws of thermodynamics. People are still trying to construct one. Perpetuum Mobile of Villard de Honnecourt (about 1230). Source: Wikipedia.

5. CS-2852 Data Structures, Andrew J. Wozniewicz Performance Measurement Measuring the time it takes to run a modern application is notoriously difficult.

6. CS-2852 Data Structures, Andrew J. Wozniewicz Performance Measurement Event-driven applications. Tools/instrumentation must be used. Be careful what you are measuring. Maintain healthy skepticism about the results. Measuring the time it takes to run a modern application is notoriously difficult.

7. CS-2852 Data Structures, Andrew J. Wozniewicz Machine Independent Analysis Assume that every basic operation takes constant time Basic operations: – Addition, Subtraction – Multiplication, Division – Memory Access Efficiency of an algorithm is the number of basic operations it performs. We do not distinguish between basic operations.

8. CS-2852 Data Structures, Andrew J. Wozniewicz Order of Increase n log n exp (n) n n²

9. CS-2852 Data Structures, Andrew J. Wozniewicz Linear Search public int find(int[] data, int key) { for (int i = 0; i < data.length; ++i) { if (data[i] > key) return -1; else if (data[i] == key) return i; } return -1; }

10. CS-2852 Data Structures, Andrew J. Wozniewicz Linear Search public int find(int[] data, int key) { int n = data.length; for (int i = 0; i < n; i++) { } return -1; }

11. CS-2852 Data Structures, Andrew J. Wozniewicz Linear Search public int find(int[] data, int key) { int n = data.length; for (int i = 0; i < n; i++) { } return -1; }

12. CS-2852 Data Structures, Andrew J. Wozniewicz Linear Search public int find(int[] data, int key) { int n = data.length; for (int i = 0; i < n; i++) { } return -1; } WORST-CASE SCENARIO

13. CS-2852 Data Structures, Andrew J. Wozniewicz Linear Search public int find(int[] data, int key) { int n = data.length; for (int i = 0; i < n; i++) { } return -1; } O(n)

14. CS-2852 Data Structures, Andrew J. Wozniewicz Common Elements public bool haveNoCommonElts(int[] x, int[] y) { int n = x.length; for (int j = 0; j < n; j++) { if (find(y, x[j]) >= 0) return false; } return true; }

15. CS-2852 Data Structures, Andrew J. Wozniewicz Common Elements public bool haveNoCommonElts(int[] x, int[] y) { int n = x.length; for (int j = 0; j < n; j++) { int m = y.length; for (int i = 0; i < m; i++) { } return true; }

16. CS-2852 Data Structures, Andrew J. Wozniewicz Common Elements public bool haveNoCommonElts(int[] x, int[] y) { int n = x.length; for (int j = 0; j < n; j++) { int m = y.length; for (int i = 0; i < m; i++) { } return true; } WORST-CASE SCENARIO

17. CS-2852 Data Structures, Andrew J. Wozniewicz Common Elements public bool haveNoCommonElts(int[] x, int[] y) { int n = x.length; for (int j = 0; j < n; j++) { int m = y.length; for (int i = 0; i < m; i++) { } return true; } WORST-CASE SCENARIO

18. CS-2852 Data Structures, Andrew J. Wozniewicz Common Elements public bool haveNoCommonElts(int[] x, int[] y) { int n = x.length; for (int j = 0; j < n; j++) { int m = y.length; for (int i = 0; i < m; i++) { } return true; } WORST-CASE SCENARIO

19. CS-2852 Data Structures, Andrew J. Wozniewicz Common Elements public bool haveNoCommonElts(int[] x, int[] y) { int n = x.length; for (int j = 0; j < n; j++) { int m = y.length; for (int i = 0; i < m; i++) { } return true; } O(n²)

20. CS-2852 Data Structures, Andrew J. Wozniewicz Binary Search public static int binarySearch(int[] sorted, int first, int upto, int key) { while (first < upto) { int mid = (first + upto) / 2; // Compute mid point. if (key < sorted[mid]) { upto = mid; // repeat search in bottom half. } else if (key > sorted[mid]) { first = mid + 1; // Repeat search in top half. } else { return mid; // Found it. return position } return - (first + 1); // Failed to find key }

21. CS-2852 Data Structures, Andrew J. Wozniewicz Binary Search Each iteration of the loop considers half of the elements from the previous iteration. An example of Divide-and-Conquer algorithm.

22. CS-2852 Data Structures, Andrew J. Wozniewicz Binary Search Each iteration of the loop considers half of the elements from the previous iteration. An example of Divide-and-Conquer algorithm. O(ln n)

23. CS-2852 Data Structures, Andrew J. Wozniewicz Function Orders A function f(n) is O(g(n)) if ``increase’’ of f(n) is not faster than that of g(n). A function f(n) is O(g(n)) if there exists a number n 0 and a nonnegative c such that for all n  n 0, 0  f(n)  cg(n). If lim n  f(n)/g(n) exists and is finite, then f(n) is O(g(n))

24. CS-2852 Data Structures, Andrew J. Wozniewicz Example Functions sqrt(n), n, 2n, ln n, exp(n), n + sqrt(n), n + n 2 lim n  sqrt(n) /n = 0sqrt(n) is O(n) n is not O(sqrt(n)) lim n  n /2n = 1/2n is O(2n) lim n  2n /n = 22n is O(n) lim n  ln(n) /n = 0ln(n) is O(n) n is not O(ln(n)) exp(n) is not O(n) lim n  n/exp(n) = 0n is O(exp(n)) lim n  (n+sqrt(n)) /n = 1n + sqrt(n) is O(n) lim n  n/(sqrt(n)+n) = 1n is O(n+sqrt(n)) n + n 2 is not O(n) lim n  n/(n + n 2 ) = 0n is O(n + n 2 ) Examples of Big-O Analysis

25. CS-2852 Data Structures, Andrew J. Wozniewicz Implication of Big-Oh Notation Suppose we know that our algorithm uses at most O(f(n)) basic steps for any n inputs, and n is sufficiently large. Then we know that our algorithm will terminate after executing at most constant times f(n) basic steps. We know that a basic step takes a constant time in a machine. Hence, our algorithm will terminate in a constant times f(n) units of time, for all large n.

26. CS-2852 Data Structures, Andrew J. Wozniewicz Other Complexity Notation Asymptotically less than or equal to O Asymptotically greater than or equal to  Asymptotically equal to  Asymptotically strictly less o

27. CS-2852 Data Structures, Andrew J. Wozniewicz Big-Omega Notation A function f(n) is  (g(n)) if there exists a number n 0 and a nonnegative c such that for all n  n 0, cg(n)  f(n). If lim n  f(n)/g(n) > 0, then f(n) is  (g(n)). A lower bound for f(n) A function f(n) is  (g(n)) if g of f(n) is not faster than that of g(n).

28. CS-2852 Data Structures, Andrew J. Wozniewicz Example Functions sqrt(n), n, 2n, ln n, exp(n), n + sqrt(n), n + n 2 n is  (sqrt(n)) n is  (ln(n)) exp(n) is  (n) lim n  (n+sqrt(n)) /n = 1 n + sqrt(n) is  (n) lim n  n/(sqrt(n)+n) = 1 n is  (n+sqrt(n)) n + n 2 is  (n) Examples of Big-Omega Analysis

29. CS-2852 Data Structures, Andrew J. Wozniewicz Implication of the  Notation Suppose, an algorithm has complexity  (f(n)). This means that there exists a positive constant c such that for all sufficiently large n, there exists at least one input for which the algorithm consumes at least cf(n) steps. We know that a basic step takes a constant time in a machine. Hence, our algorithm will terminate in at least a constant times f(n) units of time, for some large n.

30. CS-2852 Data Structures, Andrew J. Wozniewicz Intractable Problems Some computational problems have "an inherent complexity that cannot be circumvented by clever programming." Only polynomial-time algorithms are useful -> tractable problems Exponential algorithms are not useful -> intractable problems – Travelling Salesman Problem

31. CS-2852 Data Structures, Andrew J. Wozniewicz Travelling Salesman Problem (TSP) Given a list of n cities and their pairwise distances, the task is to find a shortest possible tour that visits each city exactly once.

32. CS-2852 Data Structures, Andrew J. Wozniewicz Travelling Salesman Problem (TSP) A brute-force algorithm for the TSP might take n! steps. Even if each electron in the universe (10 79 ) had the power of today's fastest supercomputer (10 12 instructions per second), and each worked for the life of the universe (10 17 seconds) on solving the problem, it would barely make a dent in solving a problem with N = 1,000 1000! >> 10 1000 >> 10 79 * 10 12 * 10 17

33. CS-2852 Data Structures, Andrew J. Wozniewicz Summary Time Complexity Analysis of Algorithms – Big-Oh Notation – O(f(n)) is based on worst case analysis – Asymptotic – O(1) < O(n) < O(ln n) < O(n ln n) < O(n²) < O(n 3 ) < O(e n ) < O(n!) < O(n n ) < … – Tractability Polynomial versus Exponential Algorithms

34. Questions? Image copyright © 2010 andyjphoto.com