1. CS 615: Design & Analysis of Algorithms
Chapter 4: Sorting
Mark Allen Weiss: Chapter 7
Page

2. CS 615 Design & Analysis of Algorithms
Course Content
Introduction, Algorithmic Notation and Flowcharts (Brassard & Bratley Chp: Chapter 3)
Efficiency of Algorithms (Brassard & Bratley Chp: Chapter 2)
Basic Data Structures (Brassard & Bratley Chp: Chapter 5)
Sorting (Weiss Chp: 7)
Searching (Brassard & Bratley Chp: Chapter 9)
Graph Algorithms (Weiss Chp: 9)
Randomized Algorithms (Weiss Chp: 10)
String Searching (Sedgewick Chap. 19)
NP Completeness (Sedgewick Chap. 40)
15 May 2019
CS 615 Design & Analysis of Algorithms

3. CS 615 Design & Analysis of Algorithms
Lecture Content
Sorting by Selection
Sorting by Insertion
Insertion Sort
Shellsort
Heapsort
Mergesort
Quicksort
15 May 2019
CS 615 Design & Analysis of Algorithms

4. CS 615 Design & Analysis of Algorithms
Sorting by Selection
Algorithm:
Select the smallest number within the array
Bring it to the front of the array
Get the next smallest element
Insert it into the next location in the array
Until all elements are sorted
When this algorithm is programmed
Required time to sort the array
Vary no more than %15
Whatever the initial order of the elements to be sorted
select(T) is quadratic
procedure select(T[1..n])
for i=1 to n-1 do
minj=i
minx=T[i]
for j=i+1 to n do
if T[j]<minx
then
minj=j
minx=T[j]
T[minj]=T[i]
T[i]=minx
15 May 2019
CS 615 Design & Analysis of Algorithms

5. CS 615 Design & Analysis of Algorithms
Sorting by Insertion
Algorithm:
Set the first element as the minimum
For all other elements
Check the element if less than current minimum
If so compare the item with previous elements in array
Exchange the elements until the correct place is found
Apply the same operation to all other elements of the array
When this algorithm is programmed
Variation between best and worst case is very big
Required time depends on the number of elements in the array
insert(T) is quadratic
procedure insert(T[1..n])
for i=2 to n do
x=T[i];
j=i-1
while j>0 and x<T[j] do
T[j+1]=T[j]
j=j-1
T[j+1]=x
15 May 2019
CS 615 Design & Analysis of Algorithms

6. CS 615 Design & Analysis of Algorithms
Insertion Sort
One of the simplest algorithms
Consist of N-1 passes
In pass P
The element in position P is saved in Tmp
All larger elements prior to position P
Are moved one spot to the right
Then Tmp is placed in correct spot
InsertionSort
Original 34 8 64 51 32 21
Position Moved
After p=1 1
After p=2
After p=3
After p=4 3
After p=5 4
15 May 2019
CS 615 Design & Analysis of Algorithms

7. Pseudocode/Insertion Sort
void InsertionSort( ElementType A[ ], int N ) {
int j, P;
ElementType Tmp;
for( P = 1; P < N; P++ )
Tmp = A[ P ];
for( j = P; j > 0 && A[ j - 1 ] > Tmp; j-- )
A[ j ] = A[ j - 1 ];
A[ j ] = Tmp; }
15 May 2019
CS 615 Design & Analysis of Algorithms

8. Analysis of Insertion Sort
Because of nested loops
each step will take N iterations
Order of algorithm is O(n2)
If the data is in reverse order:
If the data is pre-sorted
Order of algorithm is O(n)
The inner loop always fail immediately
15 May 2019
CS 615 Design & Analysis of Algorithms

9. CS 615 Design & Analysis of Algorithms
Shell Sort
Invented by Donald Shell
Was the first algorithm to break the quadratic time barrier
Uses a sequence, h1,h2,...,ht, called “increment sequence”
At each sequence:
Compare h-distant element
Exchange them if the first element is greater than the second one
ShellSort
Original 81 94 11 96 12 35 17 95 28 58 41 75 15
After 5-sort
After 3-sort
After 2-sort
After 1-sort
15 May 2019
CS 615 Design & Analysis of Algorithms

10. Pseudocode/Shell Sort
/* START: fig7_4.txt */
void Shellsort( ElementType A[ ], int N ) {
int i, j, Increment;
ElementType Tmp;
for( Increment = N / 2; Increment > 0; Increment /= 2 )
for( i = Increment; i < N; i++ )
Tmp = A[ i ];
for( j = i; j >= Increment; j -= Increment )
if( Tmp < A[ j - Increment ] )
A[ j ] = A[ j - Increment ];
else
break;
A[ j ] = Tmp; }
15 May 2019
CS 615 Design & Analysis of Algorithms

11. CS 615 Design & Analysis of Algorithms
Analysis of Shellsort
The worst case is O(n2)
Hibbard’s increment
The worst case is O(n3/2)
The average case is O(n5/4)
Not proven !
The performance is quite acceptable
Widely used in practice
A good choice for sorting up large input
15 May 2019
CS 615 Design & Analysis of Algorithms

12. CS 615 Design & Analysis of Algorithms
Heapsort 10
Executes in order of O(NlogN)
The best of sorting techniques
Only Shellsort uses Sadgewick’s increment sequence is better
Heap order property
(max)Heap:
The key values of children are less than the parent
The maximum can always be found at the root
(min)Heap:
The key values of children are greater than the parent
The minimum can always be found at the root 7
.... 4 5 4 5
.... 7 10
15 May 2019
CS 615 Design & Analysis of Algorithms

13. CS 615 Design & Analysis of Algorithms
Basic Heap Operations 13
Insert
Create a hole in the first available location
If the new element X does not violate the heap property
it is done
otherwise slide the hole’s parent to the new hole
If the new element can be inserted in the parent’s node
insert it and job is finished
Else execute the same operations
until the correct place is found
insert 21 16 24 31 19 68 32 h 65 26
insert 14 13 21 16 24 31 19 68 65 26 32 14 1 2 3 4 5 6 7 8 9 10 11
15 May 2019
CS 615 Design & Analysis of Algorithms

14. Basic Heap Operations: DeleteMin13
Create a hole in the root
Percolate down:
Place the smallest child
as the parent
Place child’s smallest child
in the hole of the child
Continue the operation
until no placement remains 14 16 19 21 19 68 32 31 65 26 13 14 16 19 21 19 68 65 26 32 31
DeleteMin 1 2 3 4 5 6 7 8 9 10 11
15 May 2019
CS 615 Design & Analysis of Algorithms

15. After Deletion, The New Heap14 19 16 14 19 16 26 21 19 68 65 31 32 26 21 19 68 1 2 3 4 5 6 7 8 9 10 65 31 32
15 May 2019
CS 615 Design & Analysis of Algorithms

16. CS 615 Design & Analysis of Algorithms
Heap Sorting
Build a heap of N elements
N succesive insert operations
order is
Average: O(N)
Worst Case: O(NlogN)
Perform N DeleteMin operations
Each delete will take order of O(logN)
Record deleted elments in another array
or use the shrinked heap array
put the element in emptied location
The elements will be sorted in decreasing order
Use max(heap)
to sort the elements in increasing order
The total running time is
O(N) + O(NlogN) = O(NlogN)
Start
Build
Heap
DeleteMin
End
15 May 2019
CS 615 Design & Analysis of Algorithms

17. CS 615 Design & Analysis of Algorithms
Example: HeapSort
(Max) heap 97
DeleteMax 59 53 59 53 58 26 41 58 31 26 41 31 97 97 53 59 26 41 58 31 59 53 58 26 41 31 97 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 10
15 May 2019
CS 615 Design & Analysis of Algorithms

18. CS 615 Design & Analysis of Algorithms
Example Continues 58
DeleteMax 53 53 31 41 31 26 41 59 97 26 58 59 97 58 53 31 26 41 59 97 53 41 31 26 58 59 97 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10
15 May 2019
CS 615 Design & Analysis of Algorithms

19. CS 615 Design & Analysis of Algorithms
Example Continues 31 41
DeleteMax 26 41 26 31 53 58 59 97 53 58 59 97 31 26 41 53 58 59 97 41 26 31 53 58 59 97 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10
15 May 2019
CS 615 Design & Analysis of Algorithms

20. Example: After HeapSort26 31 41 53 58 59 97 26 31 41 53 58 59 97 1 2 3 4 5 6 7 8 9 10
15 May 2019
CS 615 Design & Analysis of Algorithms

21. CS 615 Design & Analysis of Algorithms
Merge Sort
Aptr 1 13 24 26
Runs in O(NlogN) worst-case
The fundamental operation:
merging two sorted lists
Takes
two already sorted input arrays:
A, B
One output array: C
Three counters
Aptr,Bptr, and Cptr
Algorithm
Take the smaller of A[Aptr] and B[Bptr]
Put it into C[Cptr]
Advance the counters
When any of input lists exhausted
Copy the next array to C
This step can be done in one pass
O(N) 2 15 27 38
Bptr
Cptr
Aptr
Merge 1 13 24 26 2 15 27 38
Bptr 1
Cptr
15 May 2019
CS 615 Design & Analysis of Algorithms

22. CS 615 Design & Analysis of Algorithms
Merge Sort Example
Aptr
Aptr
Aptr 1 13 24 26 1 13 24 26 1 13 24 26 2 15 27 38 2 15 27 38 2 15 27 38
Bptr
Bptr
Bptr 1 2 1 2 13 1 2 13 15
Cptr
Cptr
Cptr
Aptr
Aptr 1 13 24 26 1 13 24 26 2 15 27 38 2 15 27 38
Bptr
Bptr 1 2 13 15 24 1 2 13 15 24 26 27 38
Cptr
Cptr
15 May 2019
CS 615 Design & Analysis of Algorithms

23. CS 615 Design & Analysis of Algorithms
Divide and Conquer
Divide the unordered list into two halves
Sort the first half
Sort the second half
Merge two halves
Execute the same operations recursively
until all of the array is sorted
At each step
A temporary C array is needed
At any point there could be
NlogN temporary arrays in memory
If memory is low, not a good algorithm
Allocate dynamic memory, and use input arrays
for temporary arrays
MergeSort
15 May 2019
CS 615 Design & Analysis of Algorithms

24. Example: Full Merge Sort
First Half
First Half
First Half
Aptr
Aptr
Aptr 24 13 26 1 2 27 38 15 24 13 26 1 2 27 38 15
Bptr
Second Half
Bptr
Second Half
Bptr
Second Half
First Half
First Half 1 2 13 15 26
Aptr
Aptr
Cptr 13 24 1 26 2 27 15 38
Bptr
Second Half
Bptr
Second Half
First Half
Aptr 1 13 24 26 2 15 27 38 1 2 13 15 24 26 27 38
Bptr
Second Half
15 May 2019
CS 615 Design & Analysis of Algorithms

25. CS 615 Design & Analysis of Algorithms
Analysis of Mergesort
The order of total algorithm is
O(NlogN)
Is hardly used in main memory sorts
Needs extra memory to sort two lists
Slows down the algorithm
due to copying done between the lists
Used mostly for external sorting
Using disk spaces to sort very large lists
Recursion adds extra time
Recursive call is sometimes
more than 100 times slower
than the iteration
15 May 2019
CS 615 Design & Analysis of Algorithms

26. CS 615 Design & Analysis of Algorithms
Quicksort
Like MergeSort QuickSort is also a
divide and conquer
and recursive algorithm
Average running time
O(NLogN)
Worst-case
O(N2)
Good algorithm in theory
But in practice not easy to code correctly
Algorithm: (For array S)
If S has only 0 or 1 elements return
Pick any element v in S. This is called the pivot
Partition into S1=S-{v} and S2=S-{v} two subsets
Quicksort S1 and S2.
QuickSort
15 May 2019
CS 615 Design & Analysis of Algorithms

27. CS 615 Design & Analysis of Algorithms
QuickSort Example -5 21 97 61
-31 24 49 18 52 -5 21
-31 18 24 97 61 49 52
Pivot
Pivot
Pivot -5
-31 21 18 49 52 97 61 18 21
-31 -5 61 97 49 52 61 97
-31 -5 18 21
-31 -5 18 21 24 49 52 61 97
15 May 2019
CS 615 Design & Analysis of Algorithms

28. CS 615 Design & Analysis of Algorithms
Selecting the Pivot
Do not select the first or the second element as pivot
If the array sorted in reverse order
Pivot provides a poor partition
Alternatives
Choose the pivot randomly
Random number generation is expensive
Not all systems provide real random numbers
Choose the median
if the array is reverse ordered
Eliminates the bad case for Quicksort case
Hard to calculate
Some other methods are also available
Median3
15 May 2019
CS 615 Design & Analysis of Algorithms

29. CS 615 Design & Analysis of Algorithms
Analysis of Quicksort
Pivot selection takes a constant time
Worst-Case
O(N2)
Best Case
O(NlogN)
Average Case
15 May 2019
CS 615 Design & Analysis of Algorithms

30. CS 615 Design & Analysis of Algorithms
End of Chapter 4 Sorting
15 May 2019
CS 615 Design & Analysis of Algorithms