www.hndit.com Sorting Algorithms IT12112 Lecture 07

What is Sorting? www.hndit.com Sorting: an operation that segregates items into groups according to specified criterion. A = { 3 1 6 2 1 3 4 5 9 0 } A = { 0 1 1 2 3 3 4 5 6 9 } Arranging things into ascending or descending order is called sorting. i.e. Sorting is the process of arranging items in order.

Why Sort and Examples Consider: Sorting Books in Library www.hndit.com Why Sort and Examples Consider: Sorting Books in Library Sorting Individuals by Height (Feet and Inches) Sorting Movies (Alphabetical) Sorting Numbers (Sequential)

Types of Sorting Algorithms www.hndit.com Types of Sorting Algorithms There are many, many different types of sorting algorithms, but the primary ones are: Bubble Sort Selection Sort Insertion Sort Merge Sort Shell Sort Heap Sort Quick Sort Radix Sort Swap Sort

www.hndit.com Review of Complexity Most of the primary sorting algorithms run on different space and time complexity. Time Complexity is defined to be the time the computer takes to run a program (or algorithm in our case). Space complexity is defined to be the amount of memory the computer needs to run a program.

www.hndit.com Complexity (cont.) Complexity in general, measures the algorithms efficiency in internal factors such as the time needed to run an algorithm. External Factors (not related to complexity): Size of the input of the algorithm Speed of the Computer Quality of the Compiler

Some important factors that must be considered are: www.hndit.com Some important factors that must be considered are: Speed : the simplest algorithms are O(N2) while more advanced ones are O(N log N). No algorithm can make less than O(N log N) comparisons between keys. Storage : algorithms that sort in place are the best, needing memory O(N). Those are using linked list representation need extra N words of memory for references and those that work on a copy of the file needed O(2N).

www.hndit.com Simplicity : the simpler algorithms are often easiest to implement and outer perform more sophisticated once for small problem.

Sorting Techniques There are three major sorting techniques. Such as www.hndit.com Sorting Techniques There are three major sorting techniques. Such as Sorting by selection Sorting by insertion Sorting by exchange

Bubble Sort www.hndit.com Compares adjacent array elements and exchanges their values if they are out of order Smaller values bubble up to the top of the array and larger values sink to the bottom

Analysis of Bubble Sort www.hndit.com Provides excellent performance in some cases and very poor performances in other cases Works best when array is nearly sorted to begin with Worst case number of comparisons is O(n*n) Worst case number of exchanges is O(n*n) Best case occurs when the array is already sorted O(n) comparisons O(1) exchanges

Items of Interest www.hndit.com Notice that only the largest value is correctly placed All other values are still out of order So we need to repeat this process 1 2 3 4 5 6 101 42 35 12 77 5 Largest value correctly placed

BubbleSort Most frequently used sorting algorithm Algorithm: www.hndit.com Most frequently used sorting algorithm Algorithm: for j=n-1 to 1 …. O(n) for i=0 to j ….. O(j) if A[i] and A[i+1] are out of order, swap them (that’s the bubble) …. O(1) Analysis Bubblesort is O(n^2) Appropriate for small arrays Appropriate for nearly sorted arrays

“Bubbling” All the Elements www.hndit.com “Bubbling” All the Elements 77 12 35 42 5 1 2 3 4 5 6 101 N - 1 5 42 12 35 77 1 2 3 4 5 6 101 42 5 35 12 77 1 2 3 4 5 6 101 42 35 5 12 77 1 2 3 4 5 6 101 42 35 12 5 77 1 2 3 4 5 6 101

Reducing the Number of Comparisons www.hndit.com Reducing the Number of Comparisons 12 35 42 77 101 1 2 3 4 5 6 5 77 12 35 42 5 1 2 3 4 5 6 101 5 42 12 35 77 1 2 3 4 5 6 101 42 5 35 12 77 1 2 3 4 5 6 101 42 35 5 12 77 1 2 3 4 5 6 101

Already Sorted Collections? www.hndit.com Already Sorted Collections? What if the collection was already sorted? What if only a few elements were out of place and after a couple of “bubble ups,” the collection was sorted? We want to be able to detect this and “stop early”! 42 35 12 5 77 1 2 3 4 5 6 101

An Animated Example www.hndit.com N 8 did_swap true to_do 7 index 98 23 45 14 6 67 33 42 1 2 3 4 5 6 7 8

An Animated Example www.hndit.com N 8 did_swap false to_do 7 index 1 98 23 45 14 6 67 33 42 1 2 3 4 5 6 7 8

An Animated Example www.hndit.com N 8 did_swap false to_do 7 index 1 98 23 45 14 6 67 33 42 1 2 3 4 5 6 7 8

An Animated Example www.hndit.com N 8 did_swap true to_do 7 index 1 23 98 45 14 6 67 33 42 1 2 3 4 5 6 7 8

An Animated Example www.hndit.com N 8 did_swap true to_do 7 index 2 23 98 45 14 6 67 33 42 1 2 3 4 5 6 7 8

An Animated Example www.hndit.com N 8 did_swap true to_do 7 index 2 23 98 45 14 6 67 33 42 1 2 3 4 5 6 7 8

An Animated Example www.hndit.com N 8 did_swap true to_do 7 index 2 23 45 98 14 6 67 33 42 1 2 3 4 5 6 7 8

An Animated Example www.hndit.com N 8 did_swap true to_do 7 index 3 23 45 98 14 6 67 33 42 1 2 3 4 5 6 7 8

An Animated Example www.hndit.com N 8 did_swap true to_do 7 index 3 23 45 98 14 6 67 33 42 1 2 3 4 5 6 7 8

An Animated Example www.hndit.com N 8 did_swap true to_do 7 index 3 23 45 14 98 6 67 33 42 1 2 3 4 5 6 7 8

An Animated Example www.hndit.com N 8 did_swap true to_do 7 index 4 23 45 14 98 6 67 33 42 1 2 3 4 5 6 7 8

An Animated Example www.hndit.com N 8 did_swap true to_do 7 index 4 23 45 14 98 6 67 33 42 1 2 3 4 5 6 7 8

An Animated Example www.hndit.com N 8 did_swap true to_do 7 index 4 23 45 14 6 98 67 33 42 1 2 3 4 5 6 7 8

An Animated Example www.hndit.com N 8 did_swap true to_do 7 index 5 23 45 14 6 98 67 33 42 1 2 3 4 5 6 7 8

An Animated Example www.hndit.com N 8 did_swap true to_do 7 index 5 23 45 14 6 98 67 33 42 1 2 3 4 5 6 7 8

An Animated Example www.hndit.com N 8 did_swap true to_do 7 index 5 23 45 14 6 67 98 33 42 1 2 3 4 5 6 7 8

An Animated Example www.hndit.com N 8 did_swap true to_do 7 index 6 23 45 14 6 67 98 33 42 1 2 3 4 5 6 7 8

An Animated Example www.hndit.com N 8 did_swap true to_do 7 index 6 23 45 14 6 67 98 33 42 1 2 3 4 5 6 7 8

An Animated Example www.hndit.com N 8 did_swap true to_do 7 index 6 23 45 14 6 67 33 98 42 1 2 3 4 5 6 7 8

An Animated Example www.hndit.com N 8 did_swap true to_do 7 index 7 23 45 14 6 67 33 98 42 1 2 3 4 5 6 7 8

An Animated Example www.hndit.com N 8 did_swap true to_do 7 index 7 23 45 14 6 67 33 98 42 1 2 3 4 5 6 7 8

An Animated Example www.hndit.com N 8 did_swap true to_do 7 index 7 23 45 14 6 67 33 42 98 1 2 3 4 5 6 7 8

After First Pass of Outer Loop www.hndit.com After First Pass of Outer Loop N 8 did_swap true to_do 7 Finished first “Bubble Up” index 8 23 45 14 6 67 33 42 98 1 2 3 4 5 6 7 8

The Second “Bubble Up” www.hndit.com N 8 did_swap false to_do 6 index 1 23 45 14 6 67 33 42 98 1 2 3 4 5 6 7 8

The Second “Bubble Up” www.hndit.com N 8 did_swap false to_do 6 index 1 No Swap 23 45 14 6 67 33 42 98 1 2 3 4 5 6 7 8

The Second “Bubble Up” www.hndit.com N 8 did_swap false to_do 6 index 2 23 45 14 6 67 33 42 98 1 2 3 4 5 6 7 8

The Second “Bubble Up” www.hndit.com N 8 did_swap false to_do 6 index 2 Swap 23 45 14 6 67 33 42 98 1 2 3 4 5 6 7 8

The Second “Bubble Up” www.hndit.com N 8 did_swap true to_do 6 index 2 23 14 45 6 67 33 42 98 1 2 3 4 5 6 7 8

The Second “Bubble Up” www.hndit.com N 8 did_swap true to_do 6 index 3 23 14 45 6 67 33 42 98 1 2 3 4 5 6 7 8

The Second “Bubble Up” www.hndit.com N 8 did_swap true to_do 6 index 3 23 14 45 6 67 33 42 98 1 2 3 4 5 6 7 8

The Second “Bubble Up” www.hndit.com N 8 did_swap true to_do 6 index 3 23 14 6 45 67 33 42 98 1 2 3 4 5 6 7 8

The Second “Bubble Up” www.hndit.com N 8 did_swap true to_do 6 index 4 23 14 6 45 67 33 42 98 1 2 3 4 5 6 7 8

The Second “Bubble Up” www.hndit.com N 8 did_swap true to_do 6 index 4 No Swap 23 14 6 45 67 33 42 98 1 2 3 4 5 6 7 8

The Second “Bubble Up” www.hndit.com N 8 did_swap true to_do 6 index 5 23 14 6 45 67 33 42 98 1 2 3 4 5 6 7 8

The Second “Bubble Up” www.hndit.com N 8 did_swap true to_do 6 index 5 23 14 6 45 67 33 42 98 1 2 3 4 5 6 7 8

The Second “Bubble Up” www.hndit.com N 8 did_swap true to_do 6 index 5 23 14 6 45 33 67 42 98 1 2 3 4 5 6 7 8

The Second “Bubble Up” www.hndit.com N 8 did_swap true to_do 6 index 6 23 14 6 45 33 67 42 98 1 2 3 4 5 6 7 8

The Second “Bubble Up” www.hndit.com N 8 did_swap true to_do 6 index 6 23 14 6 45 33 67 42 98 1 2 3 4 5 6 7 8

The Second “Bubble Up” www.hndit.com N 8 did_swap true to_do 6 index 6 23 14 6 45 33 42 67 98 1 2 3 4 5 6 7 8

After Second Pass of Outer Loop www.hndit.com After Second Pass of Outer Loop N 8 did_swap true to_do 6 Finished second “Bubble Up” index 7 23 14 6 45 33 42 67 98 1 2 3 4 5 6 7 8

The Third “Bubble Up” www.hndit.com N 8 did_swap false to_do 5 index 1 23 14 6 45 33 42 67 98 1 2 3 4 5 6 7 8

The Third “Bubble Up” www.hndit.com N 8 did_swap false to_do 5 index 1 23 14 6 45 33 42 67 98 1 2 3 4 5 6 7 8

The Third “Bubble Up” www.hndit.com N 8 did_swap true to_do 5 index 1 14 23 6 45 33 42 67 98 1 2 3 4 5 6 7 8

The Third “Bubble Up” www.hndit.com N 8 did_swap true to_do 5 index 2 14 23 6 45 33 42 67 98 1 2 3 4 5 6 7 8

The Third “Bubble Up” www.hndit.com N 8 did_swap true to_do 5 index 2 14 23 6 45 33 42 67 98 1 2 3 4 5 6 7 8

The Third “Bubble Up” www.hndit.com N 8 did_swap true to_do 5 index 2 14 6 23 45 33 42 67 98 1 2 3 4 5 6 7 8

The Third “Bubble Up” www.hndit.com N 8 did_swap true to_do 5 index 3 14 6 23 45 33 42 67 98 1 2 3 4 5 6 7 8

The Third “Bubble Up” www.hndit.com N 8 did_swap true to_do 5 index 3 No Swap 14 6 23 45 33 42 67 98 1 2 3 4 5 6 7 8

The Third “Bubble Up” www.hndit.com N 8 did_swap true to_do 5 index 4 14 6 23 45 33 42 67 98 1 2 3 4 5 6 7 8

The Third “Bubble Up” www.hndit.com N 8 did_swap true to_do 5 index 4 14 6 23 45 33 42 67 98 1 2 3 4 5 6 7 8

The Third “Bubble Up” www.hndit.com N 8 did_swap true to_do 5 index 4 14 6 23 33 45 42 67 98 1 2 3 4 5 6 7 8

The Third “Bubble Up” www.hndit.com N 8 did_swap true to_do 5 index 5 14 6 23 33 45 42 67 98 1 2 3 4 5 6 7 8

The Third “Bubble Up” www.hndit.com N 8 did_swap true to_do 5 index 5 14 6 23 33 45 42 67 98 1 2 3 4 5 6 7 8

The Third “Bubble Up” www.hndit.com N 8 did_swap true to_do 5 index 5 14 6 23 33 42 45 67 98 1 2 3 4 5 6 7 8

After Third Pass of Outer Loop www.hndit.com After Third Pass of Outer Loop N 8 did_swap true to_do 5 Finished third “Bubble Up” index 6 14 6 23 33 42 45 67 98 1 2 3 4 5 6 7 8

The Fourth “Bubble Up” www.hndit.com N 8 did_swap false to_do 4 index 1 14 6 23 33 42 45 67 98 1 2 3 4 5 6 7 8

The Fourth “Bubble Up” www.hndit.com N 8 did_swap false to_do 4 index 1 Swap 14 6 23 33 42 45 67 98 1 2 3 4 5 6 7 8

The Fourth “Bubble Up” www.hndit.com N 8 did_swap true to_do 4 index 1 6 14 23 33 42 45 67 98 1 2 3 4 5 6 7 8

The Fourth “Bubble Up” www.hndit.com N 8 did_swap true to_do 4 index 2 6 14 23 33 42 45 67 98 1 2 3 4 5 6 7 8

The Fourth “Bubble Up” www.hndit.com N 8 did_swap true to_do 4 index 2 No Swap 6 14 23 33 42 45 67 98 1 2 3 4 5 6 7 8

The Fourth “Bubble Up” www.hndit.com N 8 did_swap true to_do 4 index 3 6 14 23 33 42 45 67 98 1 2 3 4 5 6 7 8

The Fourth “Bubble Up” www.hndit.com N 8 did_swap true to_do 4 index 3 No Swap 6 14 23 33 42 45 67 98 1 2 3 4 5 6 7 8

The Fourth “Bubble Up” www.hndit.com N 8 did_swap true to_do 4 index 4 6 14 23 33 42 45 67 98 1 2 3 4 5 6 7 8

The Fourth “Bubble Up” www.hndit.com N 8 did_swap true to_do 4 index 4 No Swap 6 14 23 33 42 45 67 98 1 2 3 4 5 6 7 8

After Fourth Pass of Outer Loop www.hndit.com After Fourth Pass of Outer Loop N 8 did_swap true to_do 4 Finished fourth “Bubble Up” index 5 6 14 23 33 42 45 67 98 1 2 3 4 5 6 7 8

The Fifth “Bubble Up” www.hndit.com N 8 did_swap false to_do 3 index 1 6 14 23 33 42 45 67 98 1 2 3 4 5 6 7 8

The Fifth “Bubble Up” www.hndit.com N 8 did_swap false to_do 3 index 1 No Swap 6 14 23 33 42 45 67 98 1 2 3 4 5 6 7 8

The Fifth “Bubble Up” www.hndit.com N 8 did_swap false to_do 3 index 2 6 14 23 33 42 45 67 98 1 2 3 4 5 6 7 8

The Fifth “Bubble Up” www.hndit.com N 8 did_swap false to_do 3 index 2 No Swap 6 14 23 33 42 45 67 98 1 2 3 4 5 6 7 8

The Fifth “Bubble Up” www.hndit.com N 8 did_swap false to_do 3 index 3 6 14 23 33 42 45 67 98 1 2 3 4 5 6 7 8

The Fifth “Bubble Up” www.hndit.com N 8 did_swap false to_do 3 index 3 No Swap 6 14 23 33 42 45 67 98 1 2 3 4 5 6 7 8

After Fifth Pass of Outer Loop www.hndit.com After Fifth Pass of Outer Loop N 8 did_swap false to_do 3 Finished fifth “Bubble Up” index 4 6 14 23 33 42 45 67 98 1 2 3 4 5 6 7 8

Finished “Early” www.hndit.com N 8 did_swap false to_do 3 We didn’t do any swapping, so all of the other elements must be correctly placed. We can “skip” the last two passes of the outer loop. index 4 6 14 23 33 42 45 67 98 1 2 3 4 5 6 7 8

www.hndit.com void bubbleSort( int a[ ] , int nElems) { int out, in; for(out=nElems-1; out>1; out--) { // outer loop (backward) for(in=0; in<out; in++) { // inner loop (forward) if( a[in] > a[in+1] ) // out of order? swap(in, in+1); } // swap them } } // end bubbleSort() void swap(int one, int two) long temp = a[one]; a[one] = a[two]; a[two] = temp;

void bubbleSort(int a[ ] , int nElems) { int out, in; www.hndit.com void bubbleSort(int a[ ] , int nElems) { int out, in; for(out=nElems-1; out>1; out--) // outer loop (backward) for(in=0; in<out; in++) // inner loop (forward) if( a[in] > a[in+1] ) // out of order? long temp = a[in]; // swap them a[in] = a[in+1]; a[in+1] = temp; } } // end bubbleSort()

www.hndit.com Summary “Bubble Up” algorithm will move largest value to its correct location (to the right) Repeat “Bubble Up” until all elements are correctly placed: Maximum of N-1 times Can finish early if no swapping occurs We reduce the number of elements we compare each time one is correctly placed

www.hndit.com Selection Sort Selection sort is a relatively easy to understand algorithm Sorts an array by making several passes through the array, selecting the next smallest item in the array each time and placing it where it belongs in the array Efficiency is O(n*n)

Selection Sort (continued) www.hndit.com Selection sort is called a quadratic sort Number of comparisons is O(n*n) Number of exchanges is O(n) The total number of comparisons is always (n-1) + (n-2) + ... + 1 = n(n-1) / 2 No average, best, or worst case — always the same. An O(n2) algorithm.

www.hndit.com Selection Sort In terms of an array A, the selection sort finds the smallest element in the array and exchanges it with A[0]. Then, ignoring A[0], the sort finds the next smallest and swaps it with A[1] and so on.

Selection Sorting more specifically: www.hndit.com more specifically: find the smallest value in the list switch it with the value in the first position find the next smallest value in the list switch it with the value in the second position repeat until all values are in their proper places 20 8 5 10 7 5 8 20 10 7 5 7 20 10 8 5 7 8 10 20 5 7 8 10 20

Selection Sort An example of selection sort unsorted 15 6 10 5 3 8 3 6 www.hndit.com An example of selection sort unsorted 15 6 10 5 3 8 3 6 10 5 15 8 3 5 10 6 15 8 3 5 6 10 15 8 3 5 6 8 15 10 sorted 3 5 6 8 10 15 97

Iterative Selection Sort www.hndit.com Iterative selection sort algorithm //Sort the first n elements of an array Input: array A, int n Output: selectionSort1(A, n){ for(index = 0; index<n-1; index++){ indexOfNextSmallest = the index of the smallest value among A[index], A[index+1], …A[n-1] interchange the value of A[index] and A[indexOfNextSmallest] } 98

Implementing iterative selection sort in Java void selectionSort ( int a [ ] , int nElems ) { int out, in, min; for(out=0; out<nElems-1; out++) // outer loop min = out; // minimum for(in=out+1; in<nElems; in++) // inner loop if(a[in] < a[min] ) { // if min greater, min = in; // we have a new min swap(out, min); // swap them } } // end for(out) } // end selectionSort() private void swap(int one, int two) long temp = a[one]; a[one] = a[two]; a[two] = temp; www.hndit.com

void selectionSort (int arr[ ] , int n) { int i , j , minIndex , tmp; www.hndit.com void selectionSort (int arr[ ] , int n) {       int i , j , minIndex , tmp;           for (i = 0; i < n - 1; i++) {             minIndex = i;             for (j = i + 1; j < n; j++)                   if (arr[j] < arr[minIndex])                         minIndex = j;             if (minIndex != i) {                   tmp = arr[i];                   arr[i] = arr[minIndex];                   arr[minIndex] = tmp;             }       } }

Insertion Sort www.hndit.com Based on the technique used by card players to arrange a hand of cards Player keeps the cards that have been picked up so far in sorted order When the player picks up a new card, he makes room for the new card and then inserts it in its proper place

Insertion Sort Algorithm www.hndit.com For each array element from the second to the last (nextPos = 1) Insert the element at nextPos where it belongs in the array, increasing the length of the sorted subarray by 1

Insertion Sort (cont’d) www.hndit.com To sort an array with k elements, Insertion sort requires k – 1 passes. Example:

Analysis of Insertion Sort www.hndit.com Maximum number of comparisons is O(n*n) In the best case, number of comparisons is O(n) The number of shifts performed during an insertion is one less than the number of comparisons or, when the new value is the smallest so far, the same as the number of comparisons A shift in an insertion sort requires the movement of only one item whereas in a bubble or selection sort an exchange involves a temporary item and requires the movement of three items

Insertion Sort Algorithm www.hndit.com Algorithm Conceptually, incremental add element to sorted array or list, starting with an empty array (list). Incremental or batch algorithm. Analysis In best case, input is sorted: time is O(N) In worst case, input is reverse sorted: time is O(N2). Average case is (loose argument) is O(N2) Inversion: elements out of order critical variable for determining algorithm time-cost each swap removes exactly 1 inversion

Pseudo-code for Insertion Sorting www.hndit.com Pseudo-code for Insertion Sorting Place ith item in proper position: temp = data[i] shift those elements data[j] which greater than temp to right by one position place temp in its proper position

Insertion Sort (cont’d) www.hndit.com public void insertionSort (double arr[ ] , int n) { for (int k = 1 ; k < n; k++) double temp = arr [ k ]; int i = k; while (i > 0 && arr [i-1] > temp) arr [i] = arr [i - 1]; i --; } arr [i] = temp; shift to the right It can be programmed recursively, too.

Insert Action: i=1 www.hndit.com temp 8 20 8 5 10 7 i = 1, first iteration 8 20 20 5 10 7 --- 8 20 5 10 7

Insert Action: i=2 www.hndit.com temp 5 8 20 5 10 7 i = 2, second iteration 5 8 20 20 10 7 5 8 8 20 10 7 --- 5 8 20 10 7

Insert Action: i=3 www.hndit.com temp 10 5 8 20 10 7 i = 3, third iteration 10 5 8 20 20 7 --- 5 8 10 20 7

Insert Action: i=4 www.hndit.com temp 7 5 8 10 20 7 i = 4, forth iteration 7 5 8 10 20 20 7 5 8 10 10 20 7 5 8 8 10 20 --- 5 7 8 10 20

www.hndit.com Divide and Conquer Divide and Conquer cuts the problem in half each time, but uses the result of both halves: cut the problem in half until the problem is trivial solve for both halves combine the solutions

Sorting (cont’d) n2 n log2 n n Time n 10 100 1000 www.hndit.com Sorting (cont’d) Time n2 n log2 n When n is large enough, An2 eventually overtakes Bnlog n no matter how small A is and how large B. n n 10 100 1000 n2 100 10,000 1,000,000 n log2 n 35 700 10,000

Hierarchy of Big-Oh Complexity classes in increasing order of growth: www.hndit.com Complexity classes in increasing order of growth: constant time O(1) logarithmic O(log N) linear O(N) loglinear O(N log N) quadratic O(N2) cubic O(N3) ... exponential O(2N) An algorithm from a lower complexity class will run much faster than one from a higher complexity class when the value of N becomes very large.

Comparison of Quadratic Sorts www.hndit.com None of the algorithms are particularly good for large arrays

Exercises on Sorting www.hndit.com 1. Show the contents of the following integer array: 43, 7, 10, 23, 18, 4, 19, 5, 66, 14 when the array is sorted in ascending order using: (a) bubble sort, (b) selection sort, (c) insertion sort. Explain why insertion sort works well on partially sorted arrays. 3. Implement an insertion sort on an integer array that in each pass places both the minimum and maximum elements in their proper locations.