1. Computing Science 1P Lecture 19: Friday 9 th March Simon Gay Department of Computing Science University of Glasgow 2006/07

2. Computing Science 1P Lecture 19 - Simon Gay2 What's coming up? Fri 9 th March (today):lecture as normal Mon 12 th – Wed 14 th March:labs: FPP Wed 14 th March:lecture / tutorial as normal Fri 16 th March:NO LECTURE EASTER BREAK: Mon 19 th March – Fri 6 th April Tue 10 th – Wed 11 th April:Monday is a holiday Drop-in labs / FPP Wed 11 th April:lecture / tutorial as normal NORMAL SCHEDULE RESUMES

3. 2006/07Computing Science 1P Lecture 19 - Simon Gay3 Free Programming Project 2 We feel that the FPP in semester 1 was very beneficial for those of you who did it, and there is some evidence that you enjoyed it too. So, there will be another FPP now, and the handout describes it. As an added incentive, there will be prizes for the best projects.

4. 2006/07Computing Science 1P Lecture 19 - Simon Gay4 FPP Timetable Fri 9 th March:Unit 16 (FPP2) handed out. Start thinking about what you want to do. Mon 12 th – Wed 14 th March:Discuss your idea with your tutor; write a clear specification, work on a plan. Easter Break:Further work on your project, if you wish. Tue 10 th – Wed 11 th April:Further work and advice from tutors. Mon 16 th – Wed 18 th April:Demonstration to your tutor; submission (there will also be another Unit this week) Tutors will nominate the best projects from each group; the lecturers will select the winners; winners will also be asked to explain their programs.

5. 2006/07Computing Science 1P Lecture 19 - Simon Gay5 More on function parameters We are very familiar with the idea of defining a function with parameters: def test(x,y,z): and then calling the function with the correct number of parameters in the correct order: f(1,"hello",1.2) So far, this is the norm in most programming languages. Python is unusually flexible in providing extra features.

6. 2006/07Computing Science 1P Lecture 19 - Simon Gay6 Naming the parameters when calling a function Optionally we can give the name of the parameter when we call the function: f(x=1,y="hello",z=1.2) Why would we do this? If the parameters have informative names, then the function call (as well as the function definition) becomes more readable: def lookup(phonebook,name): number = lookup(phonebook = myBook, name = "John")

7. 2006/07Computing Science 1P Lecture 19 - Simon Gay7 More on naming parameters If we name the parameters when calling a function, then we don't have to put them in the correct order: number = lookup(phonebook = myBook, name = "John") number = lookup(name = "John", phonebook = myBook) are both correct.

8. 2006/07Computing Science 1P Lecture 19 - Simon Gay8 Default values of parameters We can specify a default value for a parameter of a function. Giving a value to that parameter when calling the function then becomes optional. def lookup(phonebook,name,errorvalue="") Example: then number = lookup(myBook, "John") is equivalent to number = lookup(myBook, "John", "")

9. 2006/07Computing Science 1P Lecture 19 - Simon Gay9 Default values of parameters We can specify a default value for a parameter of a function. Giving a value to that parameter when calling the function then becomes optional. def lookup(phonebook,name,errorvalue="") Example: number = lookup(myBook, "John", "Error") If we want to we can write

10. 2006/07Computing Science 1P Lecture 19 - Simon Gay10 Algorithms We're going to spend a little time discussing algorithms, a central aspect of computing science and programming. An algorithm is a systematic method or procedure for solving a problem. Every computer program is based on one or more algorithms: sometimes simple, sometimes very complex.

11. 2006/07Computing Science 1P Lecture 19 - Simon Gay11 Quoted from Wikipedia: The word algorithm comes from the name of the 9th century Persian mathematician Abu Abdullah Muhammad ibn Musa al-Khwarizmi whose works introduced Arabic numerals and algebraic concepts. He worked in Baghdad at the time when it was the centre of scientific studies and trade. The word algorism originally referred only to the rules of performing arithmetic using Arabic numerals but evolved via European Latin translation of al-Khwarizmi's name into algorithm by the 18th century. The word evolved to include all definite procedures for solving problems or performing tasks.

12. 2006/07Computing Science 1P Lecture 19 - Simon Gay12 Algorithms For a given problem there may be several algorithms which will give the solution. We are often interested in the most efficient algorithm; usually this means the fastest. A fundamental discovery of computing science is the existence of so-called NP-complete problems. These are problems which, as far as we know, cannot be solved efficiently; however, an efficient algorithm for any one of them would mean that we could solve all of them efficiently. We'll say a little more about this later, but first let's see how different algorithms can be more or less efficient.

13. 2006/07Computing Science 1P Lecture 19 - Simon Gay13 Sorting Sorting means putting data into order: numerical, alphabetical, whatever. As you know, it is a fundamental operation provided by databases; data is often stored in a sorted form to make searching easier. (E.g. telephone directories) Python lists have a built-in sort method. We can happily use it, but as computing scientists we would also like to know how it works. Let's start by thinking about possible algorithms for sorting.

14. 2006/07Computing Science 1P Lecture 19 - Simon Gay14 How do we put things in order? Think specifically about a list of numbers; we want to put them into increasing order. How do we do it? Obvious idea: Find the smallest number (we know how to do that!). Remove it and put it into the first position of a new list. Now find the smallest of the remaining numbers; it should become the second item of the new list. And so on.

15. 2006/07Computing Science 1P Lecture 19 - Simon Gay15 Selection Sort 53182764original data

16. 2006/07Computing Science 1P Lecture 19 - Simon Gay16 Selection Sort 53182764original data find smallest by looking along the list from the beginning

17. 2006/07Computing Science 1P Lecture 19 - Simon Gay17 1 Selection Sort 5382764original data find smallest by looking along the list from the beginning

18. 2006/07Computing Science 1P Lecture 19 - Simon Gay18 Selection Sort 5382764original data start a new list with the smallest item 1 sorted data

19. 2006/07Computing Science 1P Lecture 19 - Simon Gay19 Selection Sort 5382764original data 1 sorted data find smallest by looking along the list from the beginning

20. 2006/07Computing Science 1P Lecture 19 - Simon Gay20 Selection Sort 538764original data 1 sorted data put the smallest item into the new list 2

21. 2006/07Computing Science 1P Lecture 19 - Simon Gay21 Selection Sort 538764original data 1 sorted data put the smallest item into the new list 2 and so on, until the original list is empty

22. 2006/07Computing Science 1P Lecture 19 - Simon Gay22 Selection Sort: Alternative It is possible to reformulate the algorithm so that instead of removing items from the original list and putting them in a new list, we modify the original list by moving items within it. (In fact this is the more usual way to present it).

23. 2006/07Computing Science 1P Lecture 19 - Simon Gay23 Selection Sort: Alternative 53182764 find smallest item

24. 2006/07Computing Science 1P Lecture 19 - Simon Gay24 Selection Sort: Alternative 53182764 swap it with the first item

25. 2006/07Computing Science 1P Lecture 19 - Simon Gay25 Selection Sort: Alternative 13582764 swap it with the first item

26. 2006/07Computing Science 1P Lecture 19 - Simon Gay26 Selection Sort: Alternative 13582764 the yellow part is now sorted

27. 2006/07Computing Science 1P Lecture 19 - Simon Gay27 Selection Sort: Alternative 13582764 find smallest item in the non-yellow part

28. 2006/07Computing Science 1P Lecture 19 - Simon Gay28 Selection Sort: Alternative 13582764 swap it with the first item in the non-yellow part

29. 2006/07Computing Science 1P Lecture 19 - Simon Gay29 Selection Sort: Alternative 12583764 swap it with the first item in the non-yellow part

30. 2006/07Computing Science 1P Lecture 19 - Simon Gay30 Selection Sort: Alternative 12583764 and now the sorted (yellow) part of the list is bigger

31. 2006/07Computing Science 1P Lecture 19 - Simon Gay31 Selection Sort: Alternative 12583764 continue…

32. 2006/07Computing Science 1P Lecture 19 - Simon Gay32 Selection Sort: Alternative 12385764 continue…

33. 2006/07Computing Science 1P Lecture 19 - Simon Gay33 Selection Sort: Alternative 12385764 continue…

34. 2006/07Computing Science 1P Lecture 19 - Simon Gay34 Selection Sort: Alternative 12345768 continue…

35. 2006/07Computing Science 1P Lecture 19 - Simon Gay35 Selection Sort: Alternative 12345768 continue… 5 is in place already

36. 2006/07Computing Science 1P Lecture 19 - Simon Gay36 Selection Sort: Alternative 12345768 continue…

37. 2006/07Computing Science 1P Lecture 19 - Simon Gay37 Selection Sort: Alternative 12345768 continue…

38. 2006/07Computing Science 1P Lecture 19 - Simon Gay38 Selection Sort: Alternative 12345678 continue…

39. 2006/07Computing Science 1P Lecture 19 - Simon Gay39 Selection Sort: Alternative 12345678 continue… 7 is in place already

40. 2006/07Computing Science 1P Lecture 19 - Simon Gay40 Selection Sort: Alternative 12345678 continue… the last item is guaranteed to be in place

41. 2006/07Computing Science 1P Lecture 19 - Simon Gay41 Selection Sort: Alternative 12345678 finished

42. 2006/07Computing Science 1P Lecture 19 - Simon Gay42 Selection Sort in Python The first version, which builds a new list: def sort(x): s = [] while len(x) > 0: p = 0 # position of minimum so far i = 1 while i < len(x): # loop over the rest of x if x[i] < x[p]: # smaller item found p = i # update position i = i + 1 s = s + [x[p]] # put smallest in the new list del x[p] # and remove from x return s

43. 2006/07Computing Science 1P Lecture 19 - Simon Gay43 Selection Sort in Python The second version, which modifies the original list: def sort(x): i = 0 while i < len(x): p = i # position of minimum so far j = i+1 while j < len(x): # loop over the rest of x if x[j] < x[p]: # smaller item found p = j # update position j = j + 1 temp = x[i] # move smallest into position i, x[i] = x[p] # extending the sorted region x[p] = temp # of x i = i + 1

44. 2006/07Computing Science 1P Lecture 19 - Simon Gay44 Analyzing Selection Sort How can we begin to analyze the efficiency (meaning speed) of selection sort? Of course we could try it on various data sets and measure the time taken, but because different computers have different processing speeds in general, the time taken to sort 1000 numbers on my computer does not tell you much about how long it would take on your computer. Also, as computing scientists, we would like to understand something more fundamental than empirical measurements.

45. 2006/07Computing Science 1P Lecture 19 - Simon Gay45 Counting Comparisons The first idea is to analyze the algorithm and work out how many computational steps are needed to solve a problem of a given size. For sorting algorithms it turns out that the most relevant kind of computational step is the comparison of two items in the list. If the items are large pieces of data, e.g. long strings, then comparing them can be slow, and all of the other steps in the algorithm are relatively quick. For sorting algorithms we are interested in the number of comparisons needed to sort n items, expressed in terms of n.

46. 2006/07Computing Science 1P Lecture 19 - Simon Gay46 Analyzing Selection Sort Assume that we start with a list of length n. To find the smallest item, we go round a loop n-1 times, doing a comparison each time (items 2 … n are each compared with the smallest item found so far). Then we find the smallest of n-1 items, then the smallest of n-2, and so on. The total number of comparisons is (n-1) + (n-2) + (n-3) + … + 2 + 1

47. 2006/07Computing Science 1P Lecture 19 - Simon Gay47 Analyzing Selection Sort If you are taking Maths, you know that (n-1) + (n-2) + (n-3) + … + 2 + 1 = n(n-1)/2 which can easily be proved by induction. Or: n n-1

48. 2006/07Computing Science 1P Lecture 19 - Simon Gay48 Analyzing Selection Sort Selection sort needs n(n-1)/2 comparisons to sort n items. As n becomes large, the dominant term is n²/2 and we say that selection sort is an order n² algorithm. This tells us something useful, independently of the speed of a particular computer. If it takes a certain time to sort a certain data set, then to sort 10 times more data will take 100 times as long. To sort 1000 times more data will take 1 000 000 times as long. And so on.

49. 2006/07Computing Science 1P Lecture 19 - Simon Gay49 Analyzing Selection Sort nn²n²time 10100 10 000 1 0001 million1 sec 10 000100 million100 sec 100 00010 billion3 hours 1 000 0001 trillion4 months 10 000 000100 trillion3 million yrs

50. 2006/07Computing Science 1P Lecture 19 - Simon Gay50 Can we do better? There are several fairly obvious sorting algorithms which are all order n². You can look them up: e.g. insertion sort, bubble sort. They may run at different speeds for particular data sets, but they all have the feature that the running time is proportional to the square of the size of the data set. It turns out that there are more efficient sorting algorithms. The simplest to describe is merge sort, so we'll look at that.

51. 2006/07Computing Science 1P Lecture 19 - Simon Gay51 Merge Sort First we need the idea of merging two sorted lists to form a new list which is also sorted. 13582467 1 smallest

52. 2006/07Computing Science 1P Lecture 19 - Simon Gay52 Merge Sort First we need the idea of merging two sorted lists to form a new list which is also sorted. 13582467 1 smallest 2

53. 2006/07Computing Science 1P Lecture 19 - Simon Gay53 Merge Sort First we need the idea of merging two sorted lists to form a new list which is also sorted. 13582467 1 smallest 23

54. 2006/07Computing Science 1P Lecture 19 - Simon Gay54 Merge Sort First we need the idea of merging two sorted lists to form a new list which is also sorted. 13582467 1 smallest 234

55. 2006/07Computing Science 1P Lecture 19 - Simon Gay55 Merge Sort First we need the idea of merging two sorted lists to form a new list which is also sorted. 13582467 1 smallest 2345

56. 2006/07Computing Science 1P Lecture 19 - Simon Gay56 Merge Sort First we need the idea of merging two sorted lists to form a new list which is also sorted. 13582467 1 smallest 23456

57. 2006/07Computing Science 1P Lecture 19 - Simon Gay57 Merge Sort First we need the idea of merging two sorted lists to form a new list which is also sorted. 13582467 1 smallest 234567

58. 2006/07Computing Science 1P Lecture 19 - Simon Gay58 Merge Sort First we need the idea of merging two sorted lists to form a new list which is also sorted. 13582467 1 only thing left 2345678

59. 2006/07Computing Science 1P Lecture 19 - Simon Gay59 Merge Sort 53182764 Given some original data to sort: split it into two halves: 5318 2764 sort each half: (how? using merge sort!) 1358 2467 and merge: 12345678

60. 2006/07Computing Science 1P Lecture 19 - Simon Gay60 Merge in Python def merge(x,y): i = 0 # position in x j = 0 # position in y z = [] # new list while i < len(x) and j < len(y): if x[i] < y[j]: # next item comes from x z = z + [x[i]] i = i + 1 else: # next item comes from y z = z + [y[j]] j = j + 1 if i < len(x): # unmerged items remain in x z = z + x[i:] else: # unmerged items remain in y z = z + y[j:] return z

61. 2006/07Computing Science 1P Lecture 19 - Simon Gay61 Merge Sort in Python def sort(x): if len(x) <= 1: return x else: d = len(x)/2 return merge(sort(x[:d]),sort(x[d:]))

62. 2006/07Computing Science 1P Lecture 19 - Simon Gay62 Analyzing Merge Sort The algorithm repeatedly splits lists in half, sorts them, then merges the results. All the comparisons are in the merging. Think of it like this: merge length n length n/2 length n/4 length 1 …

63. 2006/07Computing Science 1P Lecture 19 - Simon Gay63 Analyzing Merge Sort Merging to produce a list of length n requires n-1 comparisons. The important thing is that this is order n. Each round of merging requires n comparisons in total (not exactly, but we only care about the fact that it is n not n² or something else). How many rounds of merging are there? Easiest to see if n is a power of 2: n = 8, 3 rounds n = 16, 4 rounds n = 32, 5 rounds and so on the number of rounds is log n (meaning logarithm to base 2)

64. 2006/07Computing Science 1P Lecture 19 - Simon Gay64 Analyzing Merge Sort There are log n rounds of merging, each requiring n comparisons. We say that merge sort has order n log n.

65. 2006/07Computing Science 1P Lecture 19 - Simon Gay65 Comparing Selection Sort and Merge Sort We now know that selection sort has order n² and merge sort has order n log n. nn log ntimen²n² 1033100 66410 000 1 00099660.01 sec1 million1 sec 10 000132 8770.1 sec100 million100 sec 100 0001.6 million1.6 sec10 billion3 hours 1 000 00020 million20 sec1 trillion4 months 10 000 000230 million4 min100 trillion3 million yrs

66. 2006/07Computing Science 1P Lecture 19 - Simon Gay66 Conclusion There are usually many algorithms for a given problem; some are more efficient than others; the difference can have huge practical significance. The subject of algorithm analysis is a large area of CS. It will come back later in the degree, especially in Levels 3 and 4. Even for the problem of sorting, there is much more to say than the fact that merge sort is better than selection sort. It is possible to prove that we can't do better than n log n, unless the data has special properties.