1. Introduction to Algorithms: Verification, Complexity, and Searching (2) Andy Wang Data Structures, Algorithms, and Generic Programming

2. Lecture Overview Review: Binary search algorithms More on computational complexity

3. Binary Search Goal Find a value in a collection of values Idea Divide and conquer

4. Binary Search (2) Requirements Collection must be “array”-like Can use an index to jump to any array element Collection must be sorted Efficiency Very fast No extra space required

5. Binary Search—the idea 01234567 1123354753607282 Search range: 0 – 7 Search target: 35 35 < 47

6. Binary Search—the idea 01234567 1123354753607282 Search range: 0 – 3 Search target: 35 23 < 35

7. Binary Search—the idea 01234567 1123354753607282 Search range: 2 - 3 Search target: 35 35 == 35

8. Binary Search Algorithm Three versions Binary_search Lower_bound Upper_bound Assumptions Collection L of data type of T with size sz L is sorted Element t of type T

9. Binary Search Algorithm (2) Outcomes Binary_search: true if t in L; false, otherwise Lower_bound: smallest j, where t <= L[j] Upper_bound: smallest j, where t < L[j] 01234567 1123354753607282 Lower boundUpper bound

10. If there are duplicate entries 01234567 1123353553607282 Search range: 0 – 7 Search target: 35 Lower bound Smallest j, where t <= L[j] Upper bound Smallest j, where t < L[j]

11. If t is not in L… 01234567 1123242553607282 Search range: 0 – 7 Search target: 35 Lower bound Upper bound

12. Correctness and Loop Invariants Correctness Loop termination State when entering the loop State when exiting the loop Loop invariants Conditions that remain true for each iteration Mathematical induction

13. Invariants—Binary Search unsigned int lower_bound(T* L, unsigned max, T t) { unsigned int low = 0, mid, high = max; while (low < high) { // (1) low < high // (2) L[low - 1] < t <= L[high] (if index is valid) mid = (low + high) / 2; if (L[mid] < t) { low = mid + 1; } else { high = mid; } // (3) low <= high // (4) high – low has decreased // (5) L[low - 1] < t <= L[high] (if index is valid) } return low; }

14. Invariants—Binary Search unsigned int lower_bound(T* L, unsigned max, T t) { unsigned int low = 0, mid, high = max; while (low < high) { // (1) low < high // (2) L[low - 1] < t <= L[high] (if index is valid) mid = (low + high) / 2; if (L[mid] < t) { low = mid + 1; } else { high = mid; } // (3) low <= high // (4) high – low has decreased // (5) L[low - 1] < t <= L[high] (if index is valid) } return low; } t does not have to be in L

15. Invariants—Binary Search // (1) low < high // (2) L[low - 1] < t <= L[high] mid = (low + high) / 2; if (L[mid] < t) { low = mid + 1; } else { high = mid; } // (3) low <= high low = mid + 1 = (old_low + high)/2 + 1 low <= (old_low + high)/2 + 1 low < (high + high)/2 + 1 low < high + 1 low <= high

16. Invariants—Binary Search // (1) low < high // (2) L[low - 1] < t <= L[high] mid = (low + high) / 2; if (L[mid] < t) { low = mid + 1; } else { high = mid; } // (3) low <= high high = mid = (low + old_high)/2 high > (low + old_high)/2 - 1 high > (low + low)/2 - 1 high > low – 1 high >= low

17. Invariants—Binary Search unsigned int lower_bound(T* L, unsigned max, T t) { unsigned int low = 0, mid, high = max; while (low < high) { // (1) low < high // (2) L[low - 1] < t <= L[high] (if index is valid) mid = (low + high) / 2; if (L[mid] < t) { low = mid + 1; } else { high = mid; } // (3) low <= high // (4) high – low has decreased // (5) L[low - 1] < t <= L[high] (if index is valid) } return low; } Termination: (3) shows that the loop can terminate (4) shows progress

18. Invariants—Binary Search unsigned int lower_bound(T* L, unsigned max, T t) { unsigned int low = 0, mid, high = max; while (low < high) { // (1) low < high // (2) L[low - 1] < t <= L[high] (if index is valid) mid = (low + high) / 2; if (L[mid] < t) { low = mid + 1; } else { high = mid; } // (3) low <= high // (4) high – low has decreased // (5) L[low - 1] < t <= L[high] (if index is valid) } return low; } Return value: (5) smallest t <= L[j], since L[j < low] != t, and t <= L[high]

19. Computational Complexity Compares growth of two functions Independent of constant multipliers and lower-order effects Metrics “Big O” Notation “Big Omega” Notation “Big Theta” Notation

20. Big “O” Notation f(n) is O(g(n)) iff  c, n 0 > 0 | 0 = n 0 f(n) = n cg(n) = cn 2 n0n0

21. Examples F(n) is O(1) F(n) = 1 F(n) = 2 F(n) = c (constant) F(n) is O(log(n)) F(n) = 1 F(n) = 2log(n) F(n) = 3log 2 (4n 5 ) + 1 F(n) = c 1 log c2 (c 3 n c4 ) + O(log(n)) + O(1)

22. Examples F(n) = O(n) F(n) = 2log(n) F(n) = n F(n) = 3n + 1 F(n) = c 1 n + O(n) + O(log(n)) F(n) = O(nlog(n)) F(n) = 3n + 2 F(n) = nlog(n) F(n) = 3nlog 4 (5n 7 ) + 2n F(n) = c 1 nlog c2 (c 3 n c4 ) + O(nlog(n)) + O(n)

23. Examples F(n) = O(n 2 ) F(n) = 3nlog(n) + 2n F(n) = n 2 F(n) = 3n 2 + 2n + 1 F(n) = c 1 n 2 + O(n 2 ) + O(nlog(n))

24. Big “Theta” Notation f(n) is  (g(n)) iff  c 1, c 2, n 0 > 0 | 0 = n 0 f(n) = n n0n0 1/2g(n) = 1/2n 2g(n) = 2n

25. Examples F(n) is  (1) F(n) = 1 F(n) = 2 F(n) = c (constant) F(n) is  (log(n)) F(n) = 1 F(n) = 2log(n) F(n) = 3log 2 (4n 5 ) + 1 F(n) = c 1 log c2 (c 3 n c4 ) + O(log(n)) + O(1)

26. Examples F(n) =  (n) F(n) = 2log(n) F(n) = n F(n) = 3n + 1 F(n) = c 1 n + O(n) + O(log(n)) F(n) =  (nlog(n)) F(n) = 3n + 2 F(n) = nlog(n) F(n) = 3nlog 4 (5n 7 ) + 2n F(n) = c 1 nlog c2 (c 3 n c4 ) + O(nlog(n)) + O(n)

27. Examples F(n) =  (n 2 ) F(n) = 3nlog(n) + 2n F(n) = n 2 F(n) = 3n 2 + 2n + 1 F(n) = c 1 n 2 + O(n 2 ) + O(nlog(n))

28. Big “Omega” Notation f(n) is  (g(n)) iff  c, n 0 > 0 | 0 = n 0 f(n) = n n0n0 cg(n) = c

29. Examples F(n) is  (1) F(n) = 1 F(n) = 2n F(n) = n 2 F(n) is  (log(n)) F(n) = 1 F(n) = 2log(n) F(n) = nlogn + n + 1 F(n) = n 3

30. Examples F(n) =  (n) F(n) = 2log(n) F(n) = n F(n) = 3n 2 + 1 F(n) = nlogn + 3n 2 + 1 F(n) =  (nlog(n)) F(n) = 3n + 2 F(n) = nlog(n) F(n) = 3n 2 F(n) = n 3 + 2n 2

31. Examples F(n) =  (n 2 ) F(n) = 3nlog(n) + 2n F(n) = n 2 F(n) = 3n 3 + 2n + 1

32. Order the following functions… n 1/50, log(n 2 ), (log(n)) 2, 5n 2, 100 log(n), 1.1 n

33. Order the following functions… n 1/50, log(n 2 ), (log(n)) 2, 5n 2, 100 log(n), 1.1 n log(n 2 ) = 2log(n) (log(n)) 2 = log(n)log(n) 100 log(n) = n log(100) = n 2 log(n 2 ) < (log(n)) 2 < n 1/50 < 100 log(n) < 5n 2 < 1.1 n

34. Complexity Analysis Steps Find n = size of input Find an atomic activity to count Find f(n) = the number of atomic activities done by an input size of n Complexity of an algorithm = complexity of f(n)

35. Algorithm Complexity—Loops for (j = 0; j < n; ++j) { // 3 atomics } Complexity =  (3n) =  (n)

36. Loops with Break for (j = 0; j < n; ++j) { // 3 atomics if (condition) break; } Upper bound =  (3n) =  (n) Lower bound =  (3) =  (1) Complexity = O(n)

37. Loops in Sequence for (j = 0; j < n; ++j) { // 3 atomics } for (j = 0; j < n; ++j) { // 5 atomics } Complexity =  (3n + 5n) =  (n)

38. Nested Loops for (j = 0; j < n; ++j) { // 2 atomics for (k = 0; k < n; ++k) { // 3 atomics } Complexity =  ((2 + 3n)n) =  (n 2 )

39. Sequential Search for (T item = begin(L); item != end(L); item = next(L)) { if (t == item) return true; } if (t == item) return true; return false; Input size: n Atomic computation: comparison Complexity = O(n)

40. Binary Search unsigned int lower_bound(T* L, unsigned max, T t) { unsigned int low = 0, mid, high = max; while (low < high) { mid = (low + high) / 2; if (L[mid] < t) { low = mid + 1; } else { high = mid; } return low; } Input size: n Atomic computation: comparison Complexity = k iterations x 1 comparison/loop

41. Iteration Count for Binary Search unsigned int lower_bound(T* L, unsigned max, T t) { unsigned int low = 0, mid, high = max; while (low < high) { mid = (low + high) / 2; if (L[mid] < t) { low = mid + 1; } else { high = mid; } return low; } Iter # search space 1n 2n/2 3n/4 kn/2 (k-1) n/2 (k-1) = 1 n = 2 (k-1) log 2 (n) = k - 1

42. Iteration Count for Binary Search unsigned int lower_bound(T* L, unsigned max, T t) { unsigned int low = 0, mid, high = max; while (low < high) { mid = (low + high) / 2; if (L[mid] < t) { low = mid + 1; } else { high = mid; } return low; } n/2 (k-1) = 1 n = 2 (k-1) log 2 (n) = k - 1 log 2 (n) + 1 = k Complexity function f(n) = log(n) iterations x 1 comparison/loop =  (log(n))

43. Announcement Exam 1 (9/29) String class BitVector and Bit operations Hash functions/hash tables Templates Vectors Algorithm verification, complexity analysis, and searching