1. Deriving Algorithms Ran Libeskind-Hadas Department of Computer Science Harvey Mudd College

2. Assumptions… Course emphasis is on preparing students to… – Design their own algorithms – Analyze and prove the correctness of those algorithms As opposed to what? – Emphasis on learning how a number of “classical” algorithms work – Practice implementing existing algorithms

3. Assumptions… Course emphasis is on preparing students to… – Design their own algorithms – Analyze and prove the correctness of those algorithms As opposed to what? – Emphasis on learning how a number of “classical” algorithms work – Practice implementing existing algorithms

4. if emphasis on design then Use a paradigmatic approach – Divide and conquer – Dynamic programming – Greed (and it’s dangers) – Amortized analyses – Integration of algorithms and data structures

5. But even in this approach… It’s easy to show representative algorithms rather than derive them

6. Why Derive? It’s intellectually satisfying Provides a deeper sense of how algorithms are designed Helps students remember algorithms Provides students with confidence and skills to design their own algorithms

7. A Derivation-First Approach A paradigmatic approach to algorithms is not necessarily one that emphasizes derivations We are using a derivation-based approach to teaching the undergraduate algorithms course at Harvey Mudd Currently about 20% of the material is being taught this way; we hope to do more in the future!

8. The Setting… Sophomore/Junior level algorithms course 75-100 students Two 75 minute lectures per week – Repeat about four cycles of: 15 minutes of lecture 5 minutes of pair work

9. Examples Deriving sorting algorithms Deriving Dijkstra’s shortest path algorithm Deriving Kruskal’s MST algorithm Paper describes other examples Disclaimer: Many of these ideas are known to computer scientists and some appear in books or scholarly articles. But algorithms courses don’t appear to be taught in this way.

10. Sorting Observation: Selection sort and mergesort are “natural” but heapsort seems like magic Approach: Derive heapsort from selection sort and mergesort Assumptions: – Students have seen selection sort and mergesort before (e.g., CS 1) – Students have seen heaps before (e.g., CS 2) – They might even have seen heapsort, but if so it usually hasn’t stuck!

11. Recall selection sort and mergesort… 42 23 5 5 16 47 3 3 8 8 12

12. Mergesort and it’s analysis 42 23 5 5 16 47 3 3 8 8 12 T(n) = 2 T(n/2) + n T(1) = c 5 5 16 23 42 3 3 8 8 12 47 merge  (n log n)

13. If dividing in two is good… Let’s try dividing into k pieces! What’s the recurrence relation now?

14. If dividing in two is good… Let’s try dividing into k pieces! What’s the recurrence relation now? T(n) = k T(n/k) + (k-1)n T(n) = c Now we have k “fingers” at the front of k lists! Now solve this!

15. If dividing in two is good… T(n) = k T(n/k) + (k-1)n T(n) = c T(n) = (k-1) n log k n Now we have k “fingers” at the front of k lists! What do you expect it to be when k = 2? Will it ever be asymptototically better than n log n? What do you expect it to be when k > 2? What happens when k = n?

16. When k = n T(n) = (k-1) n log k n = = (n-1) n which is  (n 2 ) Have you seen this algorithm before? Now we have n “fingers” at the front of n lists! …

17. When k = n This is Selection sort! But it seems a bit crazy to do all of this work to repeatedly find the minimum! Now we have n “fingers” at the front of n lists! …

18. When k = n … Insert all elements into a heap Repeatedly remove elements from heap

19. When k = n … Insert all elements into a heap Repeatedly remove elements from heap Set up and solve recurrence O(n log n)

20. Take-Away Mergesort generalizes naturally from 2-way to k-way splitting When k = n, mergesort “degenerates” to selection sort But, merging more cleverly results in heapsort Important connections established Heapsort is derived rather than pulled from a hat!

21. Dijkstra Observation: Dijkstra’s algorithm is often presented and proved correct, but little intuition is offered and it’s hard to remember how it works. Approach: Motivate Dijkstra from BFS Assumptions: Students have seen… – BFS (and the “wave” metaphor) – Heaps

22. Dijkstra 1000 3 5 6 7 10 9 12

23. Dijkstra 1000 3 5 6 7 10 9 12

24. Dijkstra 1000 3 5 6 7 10 9 12 1000 3

25. Dijkstra 1000 3 5 6 7 10 9 12 1000 3 3 8

26. What’s required to keep track of the wavefront? Deleting the item with minimum value Updating values when they get better “offers”

27. Keeping track of the wave! 1000 3 8 0 0 0 3 3 3 8 8

28. Take-Away BFS is “natural” but Dijkstra is much less obvious But Dijkstra can be derived as an efficient implementation of BFS for weighted graphs (assuming integer weights) Now it makes sense to prove that Dijkstra also works for arbitrary non-negative real-valued edge weights

29. Deriving Kruskal’s MST Algorithm Observation: Minimum spanning tree algorithms are typically presented and then proved correct. Approach: Derive intuitively reasonable MST algorithms and then show that they are correct Assumptions: Students have seen some elementary graph theory and it is reviewed briefly here (e.g., trees and their properties, graph cuts).

30. Minimum Spanning Trees Start with motivation of the problem Begin by assuming all edge weights are distinct Give one result: The cheapest edge crossing a cut must be in every MST 5 1

31. How would you go about choosing edges for a MST 5 1

32. 5 1

33. 5 1 What next?

34. How would you go about choosing edges for a MST 5 1

35. 5 1 What next?

36. How would you go about choosing edges for a MST 5 1

37. 5 1

38. Finally… What is the algorithm in general? How do you prove that it’s correct?

39. Other derived algorithms Other shortest path algorithms Other minimum spanning tree algorithms Network flow algorithms

40. Outcomes “Standard” approach taught in 2011 “Derivation-first” approach taught in 2012 – Only 5 of the 30 lectures changed Student course evals on 1-7 Likert Scale 20112012 “Course promoted learning objectives”6.526.80 “Stimulated interest”6.416.75 “Learned a great deal”6.596.82

41. Anecdotally A derivation-based approach appears to help students… – Feel intellectually engaged with the material – Feel a strong sense that “I could have done that!” – Remember how algorithms work and be able to explain them to others – Derive their own algorithms!

42. Future work We are collecting data more methodically this year to better understand the impact of the derivation-based approach Developing a website where derivation “modules” can be contributed and borrowed

43. Comments and Questions