1. Introduction to Algorithm Analysis Concepts 15-211 Fundamental Data Structures and Algorithms Peter Lee January 21, 2003

2. Plan  Today  Introduction to some basic concepts in the analysis of algorithms  Reading:  For today: Chapter 5 and 7.1-7.3  For next time: Chapter 18 and 19

3. Homework 1 is available!  See the Blackboard Assignments area  Due on Monday, Jan.27, 11:59pm  Get started today!

4. Induction  Recall proofs by induction:  If trying to prove that a property P(n) holds for all natural numbers 0, 1, 2, …, then  Prove the base case of P(0).  Assume P(n), show that P(n+1) holds.

5. Inductive definitions  A great deal of computer science can be defined inductively.  For example, we can think of the factorial function defined as follows:  fact(0) = 1  fact(n) = n * fact(n-1), for n>0

6. Inductive definition of lists  From last time, we also saw that induction can be used to define some data structures:  An integer list is either  empty, or  an integer paired with an integer list Base case Inductive case

7. Last week’s implementation public interface IntListInterface {…} public class IntList implements IntListInterface { public IntListInterface append(int n) { next = next.append(n); return this; } } public class EmptyList implements IntListInterface { public IntListInterface append(int n) { return new IntList(n); } }

8. Which … … prompts another question: What's the running time? Not “wall-clock” time, but the number of “steps”? Answer: The append is handed down to the end of the list, where it eventually causes a miraculous transformation. This could be as many as n “steps”, where n is the length of the list. A bit later, we will make some mathematical definitions to allow us to say that this is O(n), or linear time.

9. Our goal  Our goal is to compare algorithms against each other  Not compute the “wall-clock” time  We will also want to know if an algorithm is “fast”, “slow”, or maybe so slow as to be impractical

10. Input size  Generally, we care about this most when the amount of time required increases as the size or amount of input increases

11. The “step”  In order to abstract from a particular piece of hardware, operating system, and language, we will focus on counting the number of steps of an algorithm  A “step” should execute in constant time  That is, it’s execution time should not vary much when the size of the input varies

12. Why do we care about “steps”? n 100n sec7n 2 sec2 n sec 1 100 s7 s2 s 5.5 ms 175 s32 s 101 ms.7 ms1 ms 454.5 ms14 ms1 year 100100 ms7 sec10 16 year 1,0001 sec12 min-- 10,00010 sec20 hr-- 1,000,0001.6 min.22 year--

13. Counting steps can be hard to do  Programs loop and make recursive calls  And their execution is dependent on (unknown) inputs  We’ll need to use math to help us!

14. List reversal public class IntList implements IntListInterface { public IntListInterface reverse() { IntListInterface t = next.reverse(); if (t == null) return this; return t.append(element); } public class EmptyList implements IntListInterface { public IntListInterface reverse() { return this; } }

15. Example: list reversal  The reversal of a list L is:  L, if L is empty  M.append(n), otherwise  where n is the first element of L, and  M is the reversal of the tail of L

16. Example: list reversal  The reversal of a list L is:  L, if L is empty  M.append(n), otherwise  where n is the first element of L, and  M is the reversal of the tail of L some constant- time steps

17. First Running Time For EmptyList clearly just one “step”. This is called constant time, or O(1) For IntList: - the #steps for the tail, plus - the #steps for append Ignoring constants: t(0) = 1 t(n) = n + t(n-1) Solving for t would tell us how many “steps” it takes to reverse a list

18. Klaus said that this is easy… t(n) = n + (n-1) + (n-2) + … 1 = n(n-1)/2 But how on earth did he come up with this beautiful little closed-form solution?

19. Solving recurrence equations  Klaus has already shown us one very common first step:  repeated substitution  t(n) = n + t(n-1)  = n + (n-1) + t(n-2)  = n + (n-1) + (n-2) + t(n-3)  and so on…  = n + (n-1) + (n-2) + (n-3) + … + 1

20. Incrementing series  By the way, this is an arithmetic seires that comes up over and over again, because characterizes many nested loops: for (i=1; i<n; i++) { for (j=1; j<i; j++) { f(); }

21. Mathematical handbooks  For really common series like this one, standard textbooks and mathematical handbooks will usually provide closed-form solutions.  So, one way is simply to look up the answer.  Another way is to try to think visually…

22. Visualizing it n n 0123… 1 2 3 … Area: n 2 /2 Area of the leftovers: n/2 So: n 2 /2 + n/2 = (n 2 +n)/2 = n(n+1)/2

23. Proving it  Yet another approach is to start with an answer or a guess, and then verify it by induction.  t(1) = 1(1+1)/2 = 1  Inductive case:  for n>1, assume t(n-1) = (n-1)(n-1+1)/2 = (n-1) 2 /2 = n 2 /2 – n + 1/2  then t(n) = n + n 2 /2 – n + 1/2  = n 2 /2 + 1/2  = n(n+1)/2

24. Summations  Arithmetic and geometric series come up everywhere in analysis of algorithms.  Some series come up so frequently that every computer scientist should know them by heart.

25. How to Sort a List? Sorting an array is easy and we know many algorithms. But how do we sort a singly linked list? As always, think inductively: sort(nil) = nil sort(L) = insert head(L) in “order” to sort(tail(L))

26. Insertion sort 1054713993022247105139930222134710599302221347991053022213304799105222 10547139930222

27. Code for ordered insert public IntList(int x, IntList head) { num = x; next = head; } public IntList order_insert(int x) { if( x <= num ) return new IntList(x,this); next = next.order_insert(x); return this; } tail takes care of placement new constructor method

28. Analysis of ordered insert This creates only one new object (the node holding x). The running time depends on the position of x in the new list. But in the worst case this could take n steps.

29. Analysis of sort() sort(nil) = nil sort(L) = insert the head into the right place in sort(tail(L)) t(0) = 1 t(n) = n + t(n-1) and we already know the answer to this…

30. Insertion sort for i = 2 to n do insert a[i] in the proper place in a[1:i-1] This is yet another example of a doubly- nested loop…

31. How fast is insertion sort? We’ve essentially counted the number of computation steps in the worst case. But what happens if the elements are nearly sorted to begin with?

32. Worst-case analysis  We’ll have much more to say, later in the course, about “worst-case” vs “average-case” vs “expected case” performance.

33. How long does it take to sort?  Can we do better than n(n+1)/2?  In the worst case?  In the average case?  We’ll find out later…

34. Quiz Break

35. Doubling summation Like the incrementing summation, sums of powers of 2 are also encountered frequently in computer science. What is the closed-form solution for this sum? Prove your answer by induction.

36. Hint 1: Visualizing it Imagine filling a glass by halves… 2 n-1 2 n-2 2 n-3 2 n-4 2 n-5

37. Hint 2: Visualizing it  A somewhat geekier hint: term in binary 2 0 1 2 1 10 2 2 100 2 3 1000 2 4 10000 … What is the sum of this column of binary numbers?

38. Proving it  Base case:  When n=1, then 2 0 = 2 1 -1  Induction step, when n>1.  Assume true for n’<n, consider n  By the IH, then

39. Stacks and Queues, Revisited

40. A Stack interface public interface Stack { public void push(Object x); public void pop(); public Object top(); public boolean isEmpty(); public void makeEmpty(); }

41. Stacks are LIFO a b c d e Push operations:

42. Stacks are LIFO a b c d e Pop operation: Last element that was pushed is the first to be popped.

43. A Queue interface public interface Queue { public void enqueue(Object x); public Object dequeue(); public boolean isEmpty(); public void makeEmpty(); }

44. Queues are FIFO front back krqcm

45. Queues are FIFO front back krqcm y Enqueue operation:

46. Queues are FIFO frontback krqcmy Enqueue operation:

47. Queues are FIFO frontback krqcmy Dequeue operation:

48. Implementing stacks, 1 abc Linked representation. All operations constant time, or O(1).

49. Implementing stacks, 2  An alternative is to use an array-based representation.  What are some advantages and disadvantages of an array-based representation? abc top

50. Array representation of stacks  But what to do when the array overflows?  Can we still get O(1) operations?  We could have a linked list of arrays.  Is there another way?

51. An idea  Let’s try doubling the size of the array every time it overflows.  In more detail:  Start off with an array of size n.  When the array overflows, allocate a new array of size 2n, copy all of the previous n elements to it.  Set n=2n, and then continue by using the new array.

52. A Big 15-211 Hint  Whenever you see something doubling, immediately think of powers of 2:  2 0, 2 1, 2 2, 2 3, 2 4, … Opportunity for induction…

53. Question 1  In the worst case, what is the running time of the push operation?  I.e., how many copying operations might be needed?  Answer: O(n).  If a push() operation causes an overflow, then potentially all n elements on the stack must be copied.

54. Question 2  In the worst case, when starting with an empty stack, what is the average running time for all of the push operations in any given sequence of legal stack operations?

55. What contributes  What contributes to the running time?  The push itself  Possibility of creating a new array  Copying all of the elements to a new array Constant-time overhead This is the overhead that we have to worry about

56. Informal analysis  Let’s try counting a sequence of push operations, starting with array of size 0.  push: 0 copies (create new array of size 2 0 )  push: 1 copy (into a new array of size 2 1 )  push: 2 copies (into a new array of size 2 2 )  push push: 4 copies (into a new array of size 2 3 )  push push push push: 8 copies (into a new array of size 2 4 )

57. Informal analysis, cont’d copy operations are needed. So, on average, one copy is needed for every push operation. 2 n-1 +2 n-2 +2 n-3 …+2 0 =  2 n For 2 n push operations, a grand total of

58. Question 2, again  In the worst case, when starting with an empty stack, what is the average running time for all of the push operations in any given sequence of legal stack operations?  Answer: O(1)  Each push might require the creation of a new array plus, on average, a copy.

59. Amortized running time  We say that our array-based implementation of stacks runs in amortized constant (O(1)) time.  The J2SE java.util.ArrayList class is based on this representation.java.util.ArrayList

60. A question to ponder…  Question: How would you use an implementation of the Stack interface to implement the Queue interface?  What would be the running time of the operations?

61. A queue from two stacks f g h i j Enqueue: e d c b a Dequeue: What happens when the stack on the right becomes empty?