1. Saurav Karmakar Spring 2007
Chapter 9 Sorting
Saurav Karmakar
Spring 2007

2. Sorting
The need to sort numbers, strings, and other records arises frequently in computer applications.
The entries in any modern phone book were sorted by a computer.
Databases have features that sort the records returned by a query, ordered according to any field the user desires.
Google sorts your query results by their "relevance".
We’ve already seen how searching becomes easier for sorted data.
Sorting is perhaps the simplest fundamental problem that offers a large variety of algorithms, each with its own inherent advantages and disadvantages.

3. Insertion Sort
The list is assumed to be broken into a sorted portion and an unsorted portion
Keys will be inserted from the unsorted portion into the sorted portion.
Sorted
Unsorted

4. Insertion Sort For each new key, search backward through sorted keys
Move keys until proper position is found
Place key in proper position
Moved

5. Insertion Sort: Code Fixed n-1 iterations Worst case j-1 comparisons
template <class Comparable> void insertionSort( vector<Comparable> & a ) {
for( int p = 1; p < a.size( ); p++ )
Comparable tmp = a[ p ];
int j;
for( j = p; j > 0 && tmp < a[ j - 1 ]; j-- )
a[ j ] = a[ j - 1 ];
a[ j ] = tmp; }
Fixed n-1 iterations
Worst case j-1 comparisons
Searching for the proper
position for the new key
Move current key to right
Insert the new key to its proper position
Moved

6. Insertion Sort: Analysis
Worst Case: Keys are in reverse order
Do i-1 comparisons for each new key, where i runs from 2 to n.
Total Comparisons: … + n-1
Comparison

7. Optimality Analysis I
To discover an optimal algorithm we need to find an upper and lower asymptotic bound for a problem.
An algorithm gives us an upper bound. The worst case for sorting cannot exceed (n2) because we have Insertion Sort that runs that fast.
Lower bounds require mathematical arguments.

8. Optimality Analysis II
Making mathematical arguments usually involves assumptions about how the problem will be solved.
Invalidating the assumptions invalidates the lower bound.
Sorting an array of numbers requires at least (n) time, because it would take that much time to rearrange a list that was rotated one element out of position.

9. Rotating One Element Assumptions: Keys must be moved one at a time
All key movements take
the same amount of time
The amount of time
needed to move one key
is not dependent on n.
2nd
1st
n keys must
be moved
3rd
2nd
4th
3rd
(n) time
nth
n-1st
1st
nth

10. Other Assumptions
The only operation used for sorting the list is swapping two keys.
Only adjacent keys can be swapped.
This is true for Insertion Sort and Bubble Sort.

11. Inversions Suppose we are given a list of elements L, of size n.
Let i, and j be chosen so 1i<jn.
If L[i]>L[j] then the pair (i,j) is an inversion. 1 6 2 3 4 5 7 8 9 10
Inversion
Not an Inversion

12. Maximum Inversions The total number of pairs is:
This is the maximum number of inversions in any list.
Exchanging adjacent pairs of keys removes at most one inversion.

13. Swapping Adjacent Pairs
The only inversion
that could be removed
is the (possible) one
between the red and
green keys.
Swap Red and Green
The relative position of the Red and blue areas has not changed. No inversions between the red key and the blue area have been removed. The same is true for the red key and the orange area. The same analysis can be done for the green key.

14. Lower Bound Argument A sorted list has no inversions.
A reverse-order list has the maximum number of inversions, (n2) inversions.
A sorting algorithm must exchange (n2) adjacent pairs to sort a list.
A sort algorithm that operates by exchanging adjacent pairs of keys must have a time bound of at least (n2).

15. Lower Bound For Average I
There are n! ways to rearrange a list of n elements.
Recall that a rearrangement is called a permutation.
If we reverse a rearranged list, every pair that used to be an inversion will no longer be an inversion.
By the same token, all non-inversions become inversions.

16. Lower Bound For Average II
There are n(n-1)/2 inversions in a permutation and its reverse.
Assuming that all n! permutations are equally likely, there are n(n-1)/4 inversions in a permutation, on the average.
The average performance of a “swap-adjacent-pairs” sorting algorithm will be (n2).

17. Selection Sort
Selection sort is equally simple, and also runs in quadratic time.
We walk through unsorted list of items I and pick out the smallest item, which we append to the end of sorted list S.
Algorithm :
Start with an empty list S and the unsorted list I of n input items.
for (i = 0; i < n; i++) {
Let x be the item in I having smallest key.
Remove x from I.
Append x to the end of S. }
Whether S is an array or linked list, finding the smallest item takes Theta(n) time, so selection sort takes Theta(n2) time, even in the best case! Hence, it's even worse than insertion sort.

18. Shell Sort
With insertion sort, each time we insert an element, other elements get nudged one step closer to where they ought to be
What if we could move elements a much longer distance each time?
We could move each element:
A long distance
A somewhat shorter distance
A more shorter distance next time ……
This approach is what makes shellsort so much faster than insertion sort

19. Sorting nonconsecutive subarrays
• Here is an array to be sorted (numbers aren’t important)
Consider just the red locations
Suppose we do an insertion sort on just these numbers, as if they were the only ones in the array?
Now consider just the yellow locations
We do an insertion sort on just these numbers
Now do the same for each additional group of numbers
The resultant array is sorted within groups, but not overall

20. Doing the 1-sort
In the previous slide, we compared numbers that were spaced every 5 locations
This is a 5-sort
Ordinary insertion sort is just like this, only the numbers are spaced 1 apart
We can think of this as a 1-sort
Suppose, after doing the 5-sort, we do a 1-sort?
In general, we would expect that each insertion would involve moving fewer numbers out of the way
The array would end up completely sorted

21. Example of shell sort 5-sort 3-sort 1-sort original 81 94 11 96 12 3517 95 28 58 41 75 15
5-sort
3-sort
1-sort

22. Diminishing gaps
For a large array, we don’t want to do a 5-sort; we want to do an N-sort, where N depends on the size of the array
N is called the gap size, or interval size
We may want to do several stages, reducing the gap size each time
For example, on a 1000-element array, we may want to do a 364-sort, then a 121-sort, then a 40-sort, then a 13-sort, then a 4-sort, then a 1-sort
Why these numbers?

23. Increment sequence
No one knows the optimal sequence of diminishing gaps
This sequence is attributed to Donald E. Knuth:
Start with h = 1
Repeatedly compute h = 3*h + 1
1, 4, 13, 40, 121, 364, 1093
This sequence seems to work very well
Another increment sequence mentioned in the textbook is based on the following formula:
start with h = the half of the container’s size
hi = floor (hi-1 / 2.2)
It turns out that just cutting the array size in half each time does not work out as well

24. Analysis What is the real running time of shellsort? Nobody knows!
Experiments suggest something like O(n3/2) or O(n7/6)
Analysis isn’t always easy!

25. Merge Sort If List has only one Element, do nothing
Otherwise, Split List in Half
Recursively Sort Both Lists
Merge Sorted Lists
Mergesort(A, l, r)
if l < r
then q = floor((l+r)/2)
mergesort(A, l, q)
mergesort(A, q+1, r)
merge(A, l, q, r)

26. The Merge Algorithm Assume we are merging lists A and B into list C.
Ax := 1; Bx := 1; Cx := 1;
while Ax  n and Bx  n do
if A[Ax] < B[Bx] then
C[Cx] := A[Ax];
Ax := Ax + 1;
else
C[Cx] := B[Bx];
Bx := Bx + 1;
endif
Cx := Cx + 1;
endwhile
while only Ax  n do
C[Cx] := A[Ax];
Ax := Ax + 1;
Cx := Cx + 1;
endwhile
while only Bx  n do
C[Cx] := B[Bx];
Bx := Bx + 1;

27. Merging Merge. Keep track of smallest element in each sorted half.
Insert smallest of two elements into auxiliary array.
Repeat until done.
smallest
smallest A G L O R H I M S T A
auxiliary array

28. Merging Merge. Keep track of smallest element in each sorted half.
Insert smallest of two elements into auxiliary array.
Repeat until done.
smallest
smallest A G L O R H I M S T A G
auxiliary array

29. Merging Merge. Keep track of smallest element in each sorted half.
Insert smallest of two elements into auxiliary array.
Repeat until done.
smallest
smallest A G L O R H I M S T A G H
auxiliary array

30. Merging Merge. Keep track of smallest element in each sorted half.
Insert smallest of two elements into auxiliary array.
Repeat until done.
smallest
smallest A G L O R H I M S T A G H I
auxiliary array

31. Merging Merge. Keep track of smallest element in each sorted half.
Insert smallest of two elements into auxiliary array.
Repeat until done.
smallest
smallest A G L O R H I M S T A G H I L
auxiliary array

32. Merging Merge. Keep track of smallest element in each sorted half.
Insert smallest of two elements into auxiliary array.
Repeat until done.
smallest
smallest A G L O R H I M S T A G H I L M
auxiliary array

33. Merging Merge. Keep track of smallest element in each sorted half.
Insert smallest of two elements into auxiliary array.
Repeat until done.
smallest
smallest A G L O R H I M S T A G H I L M O
auxiliary array

34. Merging Merge. Keep track of smallest element in each sorted half.
Insert smallest of two elements into auxiliary array.
Repeat until done.
smallest
smallest A G L O R H I M S T A G H I L M O R
auxiliary array

35. Merging Merge. Keep track of smallest element in each sorted half.
Insert smallest of two elements into auxiliary array.
Repeat until done.
first half exhausted
smallest A G L O R H I M S T A G H I L M O R S
auxiliary array

36. Merging Merge. Keep track of smallest element in each sorted half.
Insert smallest of two elements into auxiliary array.
Repeat until done.
first half exhausted
smallest A G L O R H I M S T A G H I L M O R S T
auxiliary array

37. Merging Merge. Keep track of smallest element in each sorted half.
Insert smallest of two elements into auxiliary array.
Repeat until done.
first half exhausted
second half exhausted A G L O R H I M S T A G H I L M O R S T
auxiliary array

38. Merging numerical example1 2 24 27 38 13 15 26 50 52

39. Merge Sort: Analysis Sorting requires no comparisons
Merging requires n-1 comparisons in the worst case, where n is the total size of both lists (n key movements are required in all cases)
Recurrence relation:

40. Merge Sort: Space Merging cannot be done in the same place
In the simplest case, a separate list of size n is required for merging
It is possible to reduce the size of the extra space, but it will still be (n)

41. Quick Sort Little Big Pivot Point Pick a pivot x in the set.
Split List into “Big” (bigger than x) and “Little” (smaller than x).
Put the Little group first, Big group second.
Recursively sort the Big and Little groups.
Quicksort( A, l, r)
if l < r
then q = partition(A, l, r)
quicksort(A, l, q-1)
quicksort(A, q+1, r)
Little
Big
Pivot Point

42. Quick Sort Big is defined as “bigger than the pivot point”.
Little is defined as “smaller than the pivot point”.
The pivot point is chosen “at random” (But an educated guess following some strategy works better). In the following example, we pick up the middle element as the pivot.

43. Partitioning
Pick pivot == 37

44. Partitioning
Step 1: Move pivot to end of array

45. Partitioning i j
Step 2: set i == 0 and j == array.length - 1

46. Partitioning i j
Step 3: move i right until value larger than the pivot is found

47. Partitioning i j
Step 4: move j left until value less than the pivot is found

48. Partitioning i j
Step 5: swap elements at positions i and j

49. Partitioning i j
Step 6: move i right until value larger than the pivot is found

50. Partitioning i j
Step 7: move j left until value less than the pivot is found

51. Partitioning i j
Step 8: swap elements at positions i and j

52. Partitioning i j
Step 9: move i left until it hits j

53. Partitioning i j
Step 10: put pivot in correct spot

54. Quicksort :Best Case Pivot point may not be the exact median
Finding the precise median is hard
If we “get lucky”, the following recurrence applies (n/2 is approximate)

55. Quicksort :Worst Case
If the keys are in order, “Big” portion will have n-1 keys, “Small” portion will be empty.
T(N) = T(N-1) + O(N) = O(N2)
N-1 comparisons are done for first key
N-2 comparisons for second key, etc.
Result:

56. Choosing Pivot Wrong Way :: Choosing extreme elements
(does not work if elements are presorted partly, not total random)
Safe Way :: Find middle element.
Better than Average Choice ::
Three Median Partitioning.

57. A Better Lower Bound
The (n2) time bound does not apply to Quicksort, Mergesort , on average.
A better assumption is that keys can be moved an arbitrary distance.
However, we can still assume that the number of key-to-key comparisons is proportional to the run time of the algorithm.

58. Sorting in Linear Time : Counting Sort
Counting sort assumes that each of the elements is an integer in the range 1 to k, for some integer k, When k = O(n)
The Counting-sort runs in O(n) time.
The basic idea of Counting sort is to determine, for each input elements x, the number of elements less than x.
This information can be used to place elements directly into its correct position.
For example, if there 17 elements less than x, than x belongs in output position 18.

59. Counting Sort In the code for Counting sort, we are given array
A[1 . . n] of length n.
We required two more arrays, the array
B[1 . . n] holds the sorted output and
the array c[1 . . k] provides temporary working storage.
COUNTING_SORT (A, B, k)
for i ← 1 to k do
    c[i] ← 0
for j ← 1 to n do
    c[A[j]] ← c[A[j]] + 1
//c[i] now contains the number of elements equal to i
for i ← 2 to k do
    c[i] ← c[i] + c[i-1]
// c[i] now contains the number of elements ≤ i
for j ← n downto 1 do
    B[c[A[i]]] ← A[j]
    c[A[i]] ← c[A[j]] - 1

60. Counting Sort A 2 5 3 C 2 3 1

61. Counting Sort C 2 3 1 C 2 4 7 8

62. Counting Sort A 2 5 3 B C 2 4 7 8

63. Counting Sort A 2 5 3 B 3 C 2 4 6 7 8

64. Counting Sort A 2 5 3 B 3 C 1 2 4 6 7 8

65. Counting Sort A 2 5 3 B 3 C 1 2 4 5 7 8

66. Counting Sort A 2 5 3 B 2 3 C 1 2 3 5 7 8

67. Counting Sort A 2 5 3 B 2 3 C 2 3 5 7 8

68. Counting Sort A 2 5 3 B 2 3 C 2 3 4 7 8

69. Counting Sort A 2 5 3 B 2 3 5 C 2 3 4 7

70. Counting Sort A 2 5 3 B 2 3 5 C 2 4 7

71. Analysis : Counting Sort
The loop of lines 1-2   takes O(k) time
The loop of lines 3-4   takes O(n) time
The loop of lines 6-7   takes O(k) time
The loop of lines 9-11 takes O(n) time
Therefore, the overall time of the counting sort is O(k) + O(n) + O(k) + O(n) = O(k + n)
In practice, we usually use counting sort algorithm when have k = O(n), in which case running time is O(n).
COUNTING_SORT (A, B, k)
for i ← 1 to k do
  c[i] ← 0
for j ← 1 to n do
  c[A[j]] ← c[A[j]] + 1
//c[i] now contains the number of elements equal to i
for i ← 2 to k do
c[i] ← c[i] + c[i-1]
// c[i] now contains the number of elements ≤ i
for j ← n downto 1 do
B[c[A[i]]] ← A[j]
c[A[i]] ← c[A[j]] - 1

72. RADIX SORT
Radix sort is a small method that many people intuitively use when alphabetizing a large list of names. (Here Radix is 26, 26 letters of alphabet). Specifically, the list of names is first sorted according to the first letter of each names, that is, the names are arranged in 26 classes.
Intuitively, one might want to sort numbers on their most significant digit. But Radix sort do counter-intuitively by sorting on the least significant digits first.
On the first pass entire numbers sort on the least significant digit and combine in a array. Then on the second pass, the entire numbers are sorted again on the second least-significant digits and combine in a array and so on.

73. Radix Sort Example INPUT 1st pass 2nd pass 3rd pass 329 720 720 329

74. Analysis Pseudo code : RADIX_SORT (A, d)
The running time depends on the stable used as an intermediate sorting algorithm.
When each digits is in the range 1 to k, and k is not too large, so COUNTING_SORT  is the obvious choice.
In case of counting sort, each pass over n d-digit numbers takes O(n + k) time.
There are d passes, so the total time for for each digit sort is O(n+k) time.
There are d passes, so the total time for Radix sort is O(dn+kd). When d is constant and k = O(n), the Radix sort runs in linear time.
Pseudo code :
RADIX_SORT (A, d)
for i ← 1 to d do     use a stable sort to sort A on digit i     // counting sort will do the job
[Consideration : n-element array A has d digits, where digit 1 is the lowest-order digit and d is the highest-order digit ]