1. Introduction to Algorithm Analysis Concepts 15-211 Fundamental Data Structures and Algorithms Aleks Nanevski and Margaret Reid-Miller January 15, 2003 based on the lecture by Peter Lee

2. Plan  Today  Introduction to some basic concepts in the design of data structures  lists, related algorithms and analysis  Reading:  For today: Chapter 5 and 7.1-7.3  For next time: Chapter 18 and 19

3. Homework 1 is available!  See the Blackboard  Due Monday, Jan.19, 11:59pm

4. A First Data Structure

5. Lists of integers Operations:  create a new empty list  return the length of the list  add an integer to the end of the list  … 32714

6. Implementing lists  How shall we implement this?  What design process could we use?  One answer:  Think mathematically  Think inductively

7. Induction  Recall proofs by induction:  If trying to prove that a property P(n) holds for 0, 1, 2, …, then  Prove the base case of P(0)  For n>0, assume P(n-1), show that P(n) holds

8. Inductive definitions  A great deal of computer science can and must be defined inductively  Example: factorial fact(n) = 1 * 2 *... * n  Inductive definition:  fact(0) = 1  fact(n) = n * fact(n-1), for n>0 Base case Inductive case

9. Implementing lists  How shall we implement this?  What design process could we use?  One answer:  Think mathematically  Think inductively

10. Inductive definitions  An integer list is either  an empty list, or  an integer paired with an integer list Base case Inductive case

11. Integer lists in Java  An integer list is either  an empty list, or  an integer paired with an integer list use null define a new IntList class The inductive definition gives us guidance on ways to implement integer lists in Java One possibility (not really the best):

12. Integer lists in Java public class IntList { int head; IntList tail; public IntList(int n, IntList l) { head = n; tail = l; } …  An integer list is either  an empty list, or  an integer paired with an integer list

13. Integer lists in Java  Example:  L.head == 3  How to access 7? L = 37109 head tail head tail

14. Integer lists in Java  Example:  L.head == 3  How to access 7?  L.tail.head == 7  L.tail.tail.tail.head == ? L = 37109 head tail head tail

15. Integer lists in Java  Example:  L.head == 3  How to access 7?  L.tail.head == 7  L.tail.tail.tail.head == 0 L = 37109 head tail head tail

16. How about the length operation?

17. Another inductive definition  The length of a list L is  0, if L is the empty list  1 + length of the tail of L, otherwise

18. Implementing length() public class ListOps { public static int length(IntList l) { if (l == null) return 0; else return 1 + length(l.tail); } …

19. The add operation  Attach n onto the end of list L  Inductive definition:  the singleton list containing n, if L is the empty list L = 37109 n

20. The add operation  Attach n onto the end of list L  Inductive definition:  the singleton list containing n, if L is the empty list  otherwise, a list whose head is ? tail is ? L = 37109 n

21. The add operation  Attach n onto the end of list L  Inductive definition:  the singleton list containing n, if L is the empty list  otherwise, a list whose head is the head of L, and tail is M  where M is the result of appending n onto the end of the tail of L L = 37109 n

22. Implementing add() public class ListOps { … public static IntList add(int n, IntList l){ if (l == null) return new IntList(n, null); else return new IntList(l.head,add(n,l.tail)); } …

23. Running time  How much time does it take to compute length() and add() ?  No, not “wall-clock” time.

24. The “step”  In order to abstract from a particular piece of hardware, operating system, and language, we will focus on counting the number of steps of an algorithm  A “step” should execute in constant time  That is, its execution time should not vary much when the size of the input varies

25. Constant-time operations public static int length(IntList l) { if (l == null) return 0; else return 1 + length(l.tail); } This is the only operation in length() that does not run in a constant amount of time. Hence, we want to know how many times this operation is invoked.

26. Constant-time operations public static int length(IntList l) { if (l == null) return 0; else return 1 + length(l.tail); } Each call to length() requires at most a constant amount of time plus the time for a recursive call on the tail So, the “steps” we want are the number of recursive calls

27. length()  How many steps for length()?  Inductive reasoning:  0 steps (i.e. recursive calls), if l == null  1 + #steps for l.tail, otherwise  For a list with N elements, length() requires N steps  We say that length() is linear time.

28. Why do we care about “steps”? n 100n sec7n 2 sec2 n sec 1 100 s7 s2 s 5.5 ms 175 s32 s 101 ms.7 ms1 ms 454.5 ms14 ms1 year 100100 ms7 sec10 16 year 1,0001 sec12 min-- 10,00010 sec20 hr-- 1,000,0001.6 min.22 year--

29. Our goal  Our goal is to compare algorithms against each other  Not compute the “wall-clock” time  We will also want to know if an algorithm is “fast”, “slow”, or maybe so slow as to be impractical

30. What about add()? public static IntList add(int n, IntList l) { if (l == null) return new IntList(n, null); else return new IntList(l.head, add(n, l.tail)); }

31. What about add()? public static IntList add(int n, IntList l) { if (l == null) return new IntList(n, null); else return new IntList(l.head, add(n, l.tail)); } Answer: also linear time.

32. Let’s Try Something a Bit Harder…

33. Reverse  The reversal of a list L is:  L, if L is empty  otherwise, the head of L added to the end of M  where M is the reversal of the tail of L

34. Implementing reverse() public static IntList reverse(IntList l) { if (l == null) return null; else { IntList r = reverse (l.tail); return add (l.head, r); }

35. How many steps?  How many “steps” does reverse take?  Think back to the inductive definition:  The reversal of a list L is:  L, if L is empty  otherwise, the head of L added to M where M is the reversal of the tail of L

36. Running time for reverse The running time is given by the following recurrence equation: t(n) = 1, if n=0 t(n) = n + t(n-1) time required to reverse the tail time required to add head to the end Solving for t would tell us how many steps it takes to reverse a list of length n

37. Reverse t(0) = 1 t(n) = n + t(n-1) public static IntList reverse(IntList l) { if (l == null) return null; else { IntList r = reverse(l.tail); return add (l.head, r); }

38. Solving recurrence equations  A common first step is to use repeated substitution:  t(n) = n + t(n-1)  = n + (n-1) + t(n-2)  = n + (n-1) + (n-2) + t(n-3)  and so on…  = n + (n-1) + (n-2) + (n-3) + … + 1

39. Gauss says that this is easy… t(n) = n + (n-1) + (n-2) + … +1 = n(n+1)/2 But how on earth did he come up with this beautiful little closed-form solution?

40. Incrementing series  By the way, this is an arithmetic series that comes up over and over again in computer science, because it characterizes many nested loops: for (i=1; i<n; i++) { for (j=1; j<i; j++) { f(); }

41. Mathematical handbooks  For really common series like this one, standard textbooks and mathematical handbooks will usually provide closed-form solutions.  So, one way is simply to look up the answer.  Another way is to try to think visually…

42. Visualizing it n n 0123… 1 2 3 … Area: n 2 /2 Area of the leftovers: n/2 So: n 2 /2 + n/2 = (n 2 +n)/2 = n(n+1)/2

43. Proving it  Yet another approach is to start with an answer or a guess, and then verify it by induction.  t(1) = 1(1+1)/2 = 1  Inductive case:  for n>1, assume t(n-1) = (n-1)(n- 1+1)/2 = (n 2 - n)/2  then t(n) = n + (n 2 - n)/2  = (n 2 + n)/2  = n(n+1)/2

44. Gauss says that this is easy... Fold the sequence in half 1 n sums up to: n+1 2 n-1 sums up to: n+1 3 n-2 sums up to: n+1 4 n-3 sums up to: n+1 5 n-4... 6 n-5...... total: (n+1)*n/2

45. Summations  Arithmetic and geometric series come up everywhere in analysis of algorithms.  Some series come up so frequently that every computer scientist should know them by heart.

46. Quadratic time  Very roughly speaking,  f(n) = n(n+1)/2  grows at about the same rate as  g(n) = n 2  In such cases, we say that reverse() runs in quadratic time  (we’ll be more precise about this later in the course)

47. How about Sorting? Everybody knows how to sort an array, but we have singly linked lists. As always, think inductively: sort(nil) = nil sort(L) = insert the head into the right place in sort(tail(L))

48. Ordered Insert Need to insert element in order, in an already sorted lists. 25102050 122512102050

49. Code for ordered insert The running time depends on the position of x in the new list. But in the worst case this could take n steps. public IntList order_insert(int x,IntList l) { if (x <= l.head) return new IntList(x, l); else { IntList t = order_insert(x, l.tail); return new IntList(l.head, t); }

50. Analysis of sort() sort(nil) = nil sort(L) = insert the head into the right place in sort(tail(L)) t(0) = 1 t(n) = n + t(n-1) which we already know to be “very roughly” n 2, or quadratic time.

51. Insertion sort for i = 2 to n do insert a[i] in the proper place in a[1:i-1] This is yet another example of a doubly- nested loop… for i=2, this sorts a[1:2] for i=3, this sorts a[1:3]... for i=n, this sorts a[1:n]

52. How fast is insertion sort? We’ve essentially counted the number of computation steps in the worst case. But what happens if the elements are nearly sorted to begin with?

53. A preview of some questions  Question: Insertion sort takes n 2 steps in the worst case, and n steps in the best case. What do we expect in the average case? What is meant by “average”?  Question: What is the fastest average time that we could ever hope to sort in? How could we prove our answer?

54. Worst-case analysis  We’ll have much more to say, later in the course, about “worst-case” vs “average-case” vs “expected case” performance.

55. Better sorting  The sorting algorithm we have just shown is called insertion sort.  It is OK for very small data sets, but otherwise is slow.  Later we will look at several sorting algorithms that run in many fewer steps.

56. Quiz Break

57. Doubling summation Like the incrementing summation, sums of powers of 2 are also encountered frequently in computer science. What is the closed-form solution for this sum? Prove your answer by induction.

58. Hint 1: Visualizing it Imagine filling a glass by halves… 2 n-1 2 n-2 2 n-3 2 n-4 2 n-5

59. Hint 2: Visualizing it  A somewhat geekier hint: term in binary 2 0 1 2 1 10 2 2 100 2 3 1000 2 4 10000 … What is the sum of this column of binary numbers?

60. Proving it  Base case:  When n=1, then 2 0 = 2 1 -1  Induction step, when n>1.  Assume true for n’<n, consider n  By the IH, then

61. How does this series appear in programming? Think of its recurrence equation: Can you think of an algorithm whose running time is characterized by this series? t(0) = 1 t(n) = 2 * t(n-1) + 1

62. Summary  Counting constant-time “steps” of computation  Linear time and quadratic time  Recurrence equations  Sums of geometric series  Simple list algorithms  Next time: Programming tips