Sorting algorithms General problem description: Given array A[0..n  1] with n elements that can be compared directly (i.e. for each i and j either A[i]

Sorting algorithms General problem description: Given array A[0..n  1] with n elements that can be compared directly (i.e. for each i and j either A[i] < A[j], or A[i] = A[j], or A[i] > A[j]), rearrange elements of A in such a way that A[0]  A[1]  A[2]  …  A[n  1].

Sorting algorithms To understand algorithms it is sufficient to assume that elements of array A are just integers. In practice elements of array A usually will be more complex and will have to be sorted according to some particular property (Key), e.g. (in C++) they may be defined as classes class Item { int Key;  };

Sorting algorithms If elements of A are complex (use large amount of memory) it may be more efficient to construct new array B of keys, sort it, and then rearrange items in initial array according to the order of keys. Actually elements of B must contain also the information about the initial position of keys, i.e. they should be pairs (Key,Position).

Sorting algorithms procedure InsertionSort(array A[0..n  1]): for i from 1 to n  1 do j  i x  A[i] while j  1 and A[j  1] > x do A[j]  A[j  1] j  j  1 A[j]  x Insertion Sort

Sorting algorithms 82497 24789 24897 24897 28497 Insertion Sort - Example

Sorting algorithms Insertion Sort - Example 2 [Adapted from T.Niemann]

Sorting algorithms procedure InsertionSort(array A[0..n  1]): for i from 1 to n  1 do j  i x  A[i] while j  1 and A[j  1] > x do A[j]  A[j  1] j  j  1 A[j]  x n  1 0..i Insertion Sort - Complexity

Sorting algorithms Time (number of operations) – worst case: T(n) = (c 1 + c 2 ) + (c 1 + 2c 2 ) + … + (c 1 + (n  1) c 2 ) T(n) = n c 1 + (1/2 n (n  1)) c 2  T(n) =  (n 2 ) Space (memory) – worst case: M(n) = n + const Insertion Sort - Complexity

Sorting algorithms Average complexity? For space – obviously the same: M(n) = n + const For time – it can be shown that  (n 2 ) operations are required also in average case. Insertion Sort - Complexity

Sorting algorithms Insertion Sort - Complexity (empirical) [Adapted from M.Lamont]

Sorting algorithms procedure BubbleSort(array A[0..n  1]): for i from 1 to n do for j from 1 to n  1 do if A[j] > A[j  1] then A[j]  A[j  1] Bubble Sort (version 1)

Sorting algorithms 82497 24879 24789 24789 24789 24789 Bubble Sort - Example

Sorting algorithms procedure BubbleSort(array A[0..n  1]): for i from 1 to n do for j from 1 to n  1 do if A[j] > A[j  1] then A[j]  A[j  1] n n1n1 Bubble Sort - Complexity

Sorting algorithms Worst case: T(n) = n(n –1) const =  (n 2 ) M(n) = n + const Average complexities are the same as in worst case Bubble Sort Complexity

Sorting algorithms Bubble Sort Complexity (empirical) [Adapted from M.Lamont]

Sorting algorithms procedure BubbleSort(array A[0..n  1]): stop = 0 while stop = 0 do stop = 1 for j from 1 to n  1 do if A[j] > A[j  1] then stop = 0 A[j]  A[j  1] Bubble Sort (version 2)

Sorting algorithms Simple sorting algorithms - empirical complexity [Adapted from M.Lamont]

Sorting algorithms Merge Sort - Idea [Adapted from H.Lang]

Sorting algorithms procedure MergeSort(array A[l..r], B[l..r]): if l < r then m   l + (r  l)/2  MergeSort(A[l..m], B[l..m]) MergeSort(A[m+1..r], B[m+1..r]) Merge(B[l..r], A[l..m], A[m+1..r]) A[l..r]  B[l..r] Merge Sort

Sorting algorithms procedure Merge(array B[l..r], A[l..m], A[m+1..r]) apos  l; bpos  l; cpos  m + 1 while bpos  m and cpos  r do if A[bpos] < A[cpos] then B[apos]  A[bpos]; bpos  bpos + 1; apos  apos + 1 else B[apos]  A[cpos]; cpos  cpos + 1; apos  apos + 1 while bpos  m do B[apos]  A[pbos]; bpos  bpos + 1; apos  apos + 1 while cpos  r do B[apos]  A[cbos]; cpos  cpos + 1; apos  apos + 1 Merge Sort

Sorting algorithms 82497 82497 824 97 7 9 9 7 79 42 8 47928 24789 Merge Sort - Example

Sorting algorithms Worst case: Merge procedure: T m (n) = n const =  (n) MergeSort procedure: T(n) = 2 T(n/2) + const + T m (n) = 2 T(n/2) +  (n) =  (n log n) Merge Sort - Complexity

Sorting algorithms Worst case: MergeSort procedure: T(n) = 2 T(n/2) + const + T m (n) = 2 T(n/2) +  (n) T(n) =  (n log n) M(n) = 2 n + const (can be impoved up to 3/2 n + const) Average complexities are the same as in worst case Merge Sort - Complexity

Sorting algorithms Quick Sort Sort - Idea [Adapted from H.Lang]

Sorting algorithms procedure QuickSort(array A[0..n  1]): if n > 1 then x  A[0] (or i  Random(n); x  A[i]) bpos  0; cpos  0 for i from 0 to n  1 do if x > A[i] then B[bpos] = A[i]; bpos  bpos + 1 if x  A[i] then C[cpos] = A[i]; cpos  cpos + 1 QuickSort(B[0..bpos  1]); QuickSort(C[0..cpos  1]) Append(x,A[0..n  1], B[0..bpos  1], C[0..cpos  1]) Quick Sort (not very efficient version)

Sorting algorithms procedure Append(x,A[0..n  1], B[0..bpos  1], C[0..cpos  1]) for i from 0 to bpos  1 do A[i]  B[i] A[bpos] = x for i from 0 to cpos  1 do A[bpos+i+1]  B[i] Quick Sort (not very efficient version)

Sorting algorithms Quick Sort - Example [Adapted from T.Niemann]

Sorting algorithms 82497 2479 7 7 47 247 24789 8 47 2 4 4 2 9 8 Quick Sort - Example 2

Sorting algorithms Worst case: Append procedure: T a (n) = n const =  (n) QuickSort procedure: T(n) = T(bpos) + T(cpos) +  (n) + T a (n) = = T(bpos) + T(cpos) +  (n) Quick Sort - Complexity

Sorting algorithms QuickSort procedure: T(n) = T(bpos) + T(cpos) +  (n) Case 1: bpos = cpos T(n) = 2 T(n/2) +  (n) Case 2: bpos = 1, cpos = n – 1 T(n) = T(1) + T(n – 1) +  (n) =  (n log n) =  (n 2 ) Quick Sort - Complexity

Sorting algorithms It can be shown that the worst situation is in Case 2 (or with bpos and cpos reversed), thus QuickSort worst case time complexity is T(n) =  (n 2 ) Quick Sort - Complexity

Sorting algorithms Good news: Average time complexity for QuickSort is  (n log n) Quick Sort - Complexity This (inefficient) version uses 2n + const of space, but that can be improved to n + const On average QuickSort is practically the fastest sorting algorithm

Sorting algorithms Quick Sort - Complexity [Adapted from K.Åhlander]

Sorting algorithms procedure QuickSort(array A[l..r]): if l < r then i  l; j  r + 1; x  A[l] (or select x randomly) while i < j do i  i + 1 while i  r and A[i] < x do i  i + 1 j  j – 1 while j  l and A[j] > x do j  j – 1 A[i]  A[j] A[i]  A[j]; A[j]  A[l] QuickSort(A[l..j–1]) QuickSort(A[j+1..r]) Quick Sort (efficient version)

Sorting algorithms ij After selecting of x move pointers i and j from start and array correspondingly and swap all pairs of elements for which A[i] > x and A[j] < x Quick Sort (efficient version)

Sorting algorithms Keys:9 111171318412145 >><>>< after 1  while:9151713184121411 ><<<><<< after 2  while:9154131817121411 <><< after 3  while: 9151341817121411 2 extra swaps:4159131817121411 sort sub-arrays:1459111213141718 Positions:0123456789 Quick Sort (efficient version) - example

Sorting algorithms Complexity comparison Tw(n)Tw(n)Ta(n)Ta(n)M(n) InsertionSort (n2)(n2) (n2)(n2) n BubbleSort (n2)(n2) (n2)(n2) n MergeSort  (n log n) 3/2 n QuickSort (n2)(n2)  (n log n) n

Sorting algorithms h 1 = 1 < h 2 < h 3 < h 4 < h 5 <  - an increasing sequence of integers For given n select some h k < n. Repeatedly for inc = h k, h k-1, , h 1 = 1 and for s from 0 to inc – 1 InsertionSort the sequences A[s], A[s + inc],A[s + 2 inc], A[s + 3 inc],  Shell Sort - Idea

Sorting algorithms Shell Sort - Idea [Adapted from H.Lang]

Sorting algorithms procedure ShellSort(array A[0..n–1]): inc  InitialInc(n) while inc  1 do for i from inc to n – 1 do j  i x  A[i] while j  inc and A[j–inc] > x do A[j]  A[j–inc] j  j – inc A[j]  x inc  NextInc(inc,n) Shell Sort

Sorting algorithms Shell Sort - Example [Adapted from T.Niemann]

Sorting algorithms Keys:9 111171318412145 sort for inc = 5:91111713 18 412145 9111145 18 4121713 sort for inc = 3:9111145 18 4121713 411195 17 13121814 sort for inc = 1: 411195 17 13121814 145911 12 13141718 Positions:0123456789 Shell Sort - Example 2

Sorting algorithms sequence 1, 2, 4, 8,  (h i = 2 i-1 ) is not particularly good Shell Sort - Good increment sequences? the “best” increment sequence is not known sequence 1, 4, 13, 40, 121,  (h i = (3 i – 1)/2 ) is one (of several known) that gives good results in practice

Sorting algorithms Shell Sort - Complexity for sequence 1, 2, 4, 8,  we have T(n) =  (n 2 ) it is known that for sequence 1, 4, 13, 40, 121,  T(n) = O(n 1.5 ), but in practice it tends to perform comparably with n log n time algorithms there exists sequences for which it has been proven that T(n) = O(n (log n) 2 ) in general the finding of time complexity for a given increment sequence is very difficult

Sorting algorithms Heap Sort - procedure "Heapify" [Adapted from T.Giang]

Sorting algorithms procedure Heapify(array A[i..j]): if RC(j)  i and A[RC(j)]  A[LC(j)] and A[RC(j)] < A[j] then A[j]  A[RC(j)] Heapify(A[i.. RC(j)]) else if LC(j)  i and A[LC(j)] < A[j] then A[j]  A[LC(j)] Heapify(A[i.. LC(j)]) Here LC(j) = 2j – n (left child) and RC(j) = 2j – n – 1 (right child) Heap Sort - procedure "Heapify"

Sorting algorithms Heap Sort - building a heap [Adapted from T.Giang]

Sorting algorithms procedure InitialiseHeap(array A[0..n–1]): for i from 1 to n – 1 do Heapify(A[0..i]) Heap Sort - building a heap

Sorting algorithms procedure HeapSort(array A[0..n–1]): InitialiseHeap(A[0..n–1]) for i from 0 to n – 2 do A[i]  A[n–1] Heapify(A[i+1..n–1]) Heap Sort

Sorting algorithms Keys:9 111171318412145 InitialiseHeap:1814111713951241 1  Heapify:1141817139111254 2  Heapify:1418171314111295 3  Heapify:1451713141812119 4  Heapify:1459131718121411 5  Heapify:1459111718131412 6  Heapify:1459111218171413 Positions:0123456789 Heap Sort - Example

Sorting algorithms 7  Heapify:1459111213171814 8  Heapify:1459111213141817 9  Heapify:1459111213141718 6  Heapify:1459111218171413 Heap Sort - Example

Sorting algorithms Worst case: Heapify procedure: T h (n) =  (log n) InitialiseHeap procedure: T i (n)  (n – 1) T h (n)  T i (n) = O(n log n) Heap Sort - Complexity

Sorting algorithms InitialiseHeap procedure - a closer look: T i (n) =  h = 1..log n h (n/2 h )   n  h = 1..  (h/2 h ) Heap Sort - Complexity

Sorting algorithms 1  1/2 + 2  1/4 +3  1/8 + 4  1/16 + 5  1//32 +  = = 1/2 + 1/4 +1/8+ 1/16+ 1/32+ 1/64 +  + 1/4 +1/8+ 1/16+ 1/32+ 1/64 +  +1/8+ 1/16+ 1/32+ 1/64 +  + 1/16+ 1/32+ 1/64 +  + 1/32+ 1/64 +  Heap Sort - Complexity = 1 = 1/2 = 1/4 = 1/8 = 1/16  = 2

Sorting algorithms InitialiseHeap procedure - a closer look: T i (n) =  h = 1..log n h (n/2 h )   n  h = 1..  (h/2 h ) Heap Sort - Complexity = 2 n T i (n) > 1/2 n T i (n) =  (n)

Sorting algorithms HeapSort procedure: T(n)  T i (n) + ((n – 1)  (log n) = =  (n) +  (n log n) =  (n log n) M(n) = n + const Heap Sort - Complexity

Sorting algorithms Complexity comparison Tw(n)Tw(n)Ta(n)Ta(n)M(n) InsertionSort (n2)(n2) (n2)(n2) n BubbleSort (n2)(n2) (n2)(n2) n MergeSort  (n log n) 3/2 n QuickSort (n2)(n2)  (n log n) n ShellSort  (n log 2 n) n HeapSort  (n log n) n

Sorting algorithms Sorting algorithms - empirical complexity [Adapted from M.Lamont]

Sorting algorithms Best complexity? Is it possible to design a sorting algorithm with time complexity T(n) = o(n log n)? Apparently T(n) =  (n) Can we have T(n) =  (n)?

Sorting algorithms Complexity - the theoretical lower bound Theorem Any comparison- based sorting algorithm has the worst case time complexity T(n) =  (n log n).

Sorting algorithms Complexity - the theoretical lower bound

Sorting algorithms Complexity - the theoretical lower bound There are n! possible different permutation of n elements (if all of them are different). There are 2 k possible outcomes of k comparisons, i.e. with k comparisons we can distinguish up to 2 k different permutations. Therefore we must have 2 k  n! k =  (n log n)  T(n) =  (n log n)

Sorting algorithms Digital sorting In this case there is some additional information available about the data to be sorted (in addition to the information that will be extracted by comparisons) CountingSort -keys to be sorted are integers from the range 0..k – 1 RadixSort - keys to be sorted are integers of restricted length, i.e. they are using no more than m “units” of memory each

Sorting algorithms Counting Sort procedure CountingSort(array A[0..n–1], B[0..n–1], C[0..k–1], k): for i from 0 to k – 1 do C[i]  0 for i from 0 to n – 1 do C[A[i]]  C[A[i]] + 1 for i from 1 to k – 1 do C[i]  C[i] + C[i – 1] for i from n – 1 downto 0 do B[C[A[i]] – 1]  A[i] C[A[i]]  C[A[i]] – 1

Sorting algorithms Counting Sort - Example k = 3 A B C 21201 21201000 21201122 21201135 2120101122 135

Sorting algorithms Counting Sort - Complexity T(n) =  (k) +  (n) +  (k) +  (n) =  (k + n) M(n) = 2 n + k + const CountingSort is stable – if two keys are equal, they will be in the same order after sorting in which they were before

Sorting algorithms Counting Sort - Complexity (empirical) [Adapted from V.Zabrodsky]

Sorting algorithms Radix Sort - where it came from [Adapted from D.Luebke]

Sorting algorithms Radix Sort - Idea [Adapted from K.Houten]

Sorting algorithms Radix Sort procedure RadixSort(array A[0..n–1], m): for i from 1 to m do StableSort array A[0..n–1] by using as keys i-th lowest “units” of memory

Sorting algorithms Radix Sort - Example 12 84 82 51 23 34 12 23 34 51 82 84 51 12 82 23 84 34

Sorting algorithms Radix Sort - Example 2 [Adapted from K.Houten]

Sorting algorithms Radix Sort needs stable sorting 12 84 82 51 23 34 12 23 34 51 84 82 51 12 82 23 84 34

Sorting algorithms Bucket sort Problem Sort n numbers that are uniformly distributed over the interval [0,1). The method for solving this problem can be also applied to sorting uniformly distributed integers from interval [1,k] (consider integer i as a number i/k).

Sorting algorithms Bucket sort - Idea [Adapted from P.Berenbrink]

Sorting algorithms Bucket sort - Idea A  [0,1) B 1  [0,1/k) B 2  [1/k,2/k) B k  [k–1/k,1)

Sorting algorithms Bucket Sort procedure BucketSort(array A[0..n–1]): for i from 0 to n–1 do insert A[i] into list B[  n A[i]  ] for i from 0 to n–1 do sort B[i] with InsertionSort concatenate the lists B[0], B[1],..., B[n–1] together in order

Sorting algorithms Bucket Sort - Example [Adapted from P.Berenbrink]

Sorting algorithms Bucket Sort - Complexity Worst case: T(n) =  (n 2 ) Average case: T(n) = n  (E[n i 2 ]) E[n i 2 ] = ? E[n i 2 ] = D[n i ] + E 2 [n i ] E 2 [n i ] = 1 D[n i ] = ?

Sorting algorithms Bucket Sort - Binomial distribution The probability that a particular bucket will contain k numbers corresponds to that given by binomial distribution (where p = 1/n): [Adapted from T.Cormen, C.Leiserson, R. Rivest]

Sorting algorithms Bucket Sort - Binomial distribution For p = 1/n we have E[X] = n  1/n = 1 [Adapted from T.Cormen, C.Leiserson, R. Rivest]

Sorting algorithms Bucket Sort - Binomial distribution X i - a random variable describing the success of i-th trial (success of i-th digit falling in this particular bucket) X i have values 0 or 1 E[X i ] = p and E[X i 2 ] = p D[X i ] = E[X i 2 ] – E 2 [X i ] = p – p 2 = p(1–p)

Sorting algorithms Bucket Sort - Binomial distribution X i - a random variable describing the success of i-th trial (success of i-th digit falling in this particular bucket) D[X i ] = p(1–p) Here Var[X] denotes the same as D[X], q = 1–p For p = 1/n we have D[X] = n  1/n  (1–1/n) = 1–1/n [Adapted from T.Cormen, C.Leiserson, R. Rivest]

Sorting algorithms Bucket Sort - Complexity Worst case: T(n) =  (n 2 ) Average case: T(n) = n  (E[n i 2 ]) E[n i 2 ] = D[n i ] + E 2 [n i ] = 1– 1/n + 1 2 = 2 – 1/n T(n) = n  (2 – 1/n) = n  (1) =  (n)

Sorting algorithms General problem description: Given array A[0..n  1] with n elements that can be compared directly (i.e. for each i and j either A[i]

Similar presentations

Presentation on theme: "Sorting algorithms General problem description: Given array A[0..n  1] with n elements that can be compared directly (i.e. for each i and j either A[i]"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Sorting algorithms General problem description: Given array A[0..n  1] with n elements that can be compared directly (i.e. for each i and j either A[i]

Similar presentations

Presentation on theme: "Sorting algorithms General problem description: Given array A[0..n  1] with n elements that can be compared directly (i.e. for each i and j either A[i]"— Presentation transcript:

Similar presentations

About project

Feedback