1. nalysis of lgorithms A A Universidad Nacional de Colombia
Facultad de Ingeniería
Departamento de Sistemas

2. OUTLINE
Getting started
Insertion sort
Merge sort

3. Insertion Sort INPUT: A sequence of numbers <a1,a2,a3,...,an>
INCREMENTAL PARADIGM
Solve for 1
Solve for 2: using the solution for 1
Solve for the next: using the previous solution
Solve for n: using the solution for n-1

4. procedure insertion_sort( A[1..n] )
for j 2 to n do
x  A[j]
i  j-1
while x<A[i] and i>0 do
A[i+1]  A[i]
i  i-1
end-while
A[i+1]  x
end-for
end

5. j= 2 A x 4 3 5 2 1 3
i = j-1 = 1 i j 3 4 5 2 1 3
i = i-1 = 0 i j
Exit while because i<0

6. j=3 A x 3 4 5 2 1 5
i = j-1 = 2 i j
Exit while because x >A[i]

7. j=4 3 4 5 2 1 2
i = j-1 = 3 i j 3 4 5 5 1 2
i = i-1 = 2 i j

8. 3 4 4 5 1 2
i = i-1 =1 j i 2 3 4 5 1 2
i = i-1 = 0 j i
Exit while because i<0

9. j=5 2 3 4 5 1 1
i = j-1 = 4 i j 2 3 4 5 5 1
i = i-1 =3 i j

10. 2 3 4 4 5 1 i
i = i-1 =2 j 2 3 3 4 5 1 i j
i = i-1 =2
Exit while because i<0 1 2 3 4 5 1 i j

11. Using a loop invariant to prove correctness of an algorithms
A loop invariant is a statement about variables of the algorithms that help us to prove why an algorithms is correct. We must show three things about a loop invariant:

12. INITILIZATION
The loop invariant is true prior to the first iteration of the loop.
MAINTENANCE
If it is true before an iteration, it remains true before the next iteration.
TERMINATION
When the loop terminates, the invariant gives an useful property that help us to show that the algorithms is correct ( we usualy use loop invatients that at termination imply the correctness of the algorithm).

13. Loop Invariant for Insertion-sort
At the start of each iteration of the for loop the subarray A[1,..,i-1] consist of the elements originally in A[1,..,i-1] but in sorted order.

14. INITILIZATION
Before the beginning of the loop j=2, then the subarray A[1,..,j-1] consist only of the A[1], which is in fact the original element A[1]. This subarray is sorted (trivially, of course).

15. Before the beginning of iteration j=2
MAINTENANCE
Before the beginning of iteration j=2
... x1 x2 x3
... xn i j
Before the beginning of iteration j=3
min{x1 , x2 }
max{x1 , x2 }
... xn j

16. Before the beginning of iteration j=3
min{x1 , x2 }
max{x1 , x2 } x3
... xn i j
Before the beginning of iteration j=3
xi1
xi2
xi3
... xn
{xi1 ,xi2 ,xi3 }= {x1 ,x2 ,x3 } j
xi1< xi2 < xi3

17. Before the beginning of iteration j=k
xi1
xi2
xi3
...
xik-1 xk
... xn i j
Before the beginning of iteration j=k+1
xi1
xi2
xi3
...
xik-1
xik
... xn j
{xi1 ,xi2 ,...,xik }= {x1 ,x2 ,...,x3 }
xi1< xi2 <...< xi3

18. TERMINATION
At termination j=n+1 the subarray A[1,..,j-1] consist of the elements in original array A[1,..,n] but in a sorted way. Therefore the algorithm is correct.

19. Time complexity cost times procedure insertion_sort( A[1..n] )
for j 2 to n do c1 n
x  A[j] c2 n-1
 insert A[j] into A[1,..,j-1] c3=0 n-1
i  j c4 n-1
while x<A[i] and i>0 do c5 i=2,..,n tj
A[i+1]  A[i] c6 i=2,..,n (ti -1)
i  i-1 c7 i=2,..,n (ti -1)
end-while
A[i+1]  x c8 n-1
end-for
end

20. tj is the number questions ask in the while(x<A[j] ….
time complexity t(n)
t (n) = c1 n +(c2+c4+c8) (n-1) + c5 j=2,..,n tj
+ (c6+c7) j=2,..,n (tj -1)
If ci= 1 for i = 1,…8 and i ≠ 3, c3 =0
t (n) = 2n j=2,..,n tj
tj is the number questions ask in the while(x<A[j] ….
1 ≤ tj ≤ i

21. Best case time complexity tb(n)
In this case tj = 1 then
tb(n) = 2n j=2,..,n 1 = (n).
tb(n) = 5n-4= (n).
Worst case time complexity tw(n)
In this case tj = j then
tw(n) = 2n j=2,..,n j = (n2).
tw(n) = 3/2 n2 + 7/2 n – 4 = (n2).

22. Average case time complexity ta(n)
The only random variable in t(n) is ti , lets study the case with {1,2,3}
t2 t3
Then t2 =1,2 with prob. ½  E(t2) = 3/2
t3 =1,2,3 with prob. 1/3  E(t3) = 2 :
tj =1,2,3,..,j with prob. 1/j  E(ti) = tj=1,..,j tj (1/j) = (j+1)/2

23. Average case time complexity ta(n)
ta(n) = 2n j=2,..,n (j+1)/2 = (n2).
ta(n) = 3/4 n2 + 17/4 n – 4 = (n2).

24. DIVIDE AND CONQUER PARADIGM
Merge-Sort
INPUT: A sequence of numbers <a1,a2,a3,...,an>
DIVIDE AND CONQUER PARADIGM
Divide: the problem into a number of sub-problems
Conquer: the sub-problems by solving them recursively
Combine: the solutions to the sub-problems into the solution for the original problem

25. MERGE-SORT
DIVIDE: Divide the n-element sequence to be sorted into two subsequences of n/2 elements each
CONQUER: sort the two subsequences recursively using merge sort
COMBINE: merge the two sorted subsequences to produce the sorted answer

26. MERGE-SORT STEPS: on a problem of size 1 do nothing
on a problem of size at least 2
Split the sequence into two halves
Sort one half of the numbers
Sort the second half of the numbers
Merge the two sorted lists

27. MERGE n m n+m Given two lists to merge size n and m
(First we study merge )
Given two lists to merge size n and m
Maintain pointer to head of each list
Move smaller element to output and advance pointer n m
n+m

28. MERGE MERGE ( A, p,q, r ) n1 q-p+1 n2  r-q
create arrays L[1..n1+1] and R[1.. n2+1]
for i 1 to n1 do
L[i]  A[p+i-1]
for j 1 to n2 do
R[j]  A[q+j]
L[n1+1]  
R[n2+1]  
i 1
j 1
for k  p to r do
if L[i]  R[j] then
A[k]  L[i]
i  i+1
else
A[k]  R[j]
j  j+1
end_if
end_for
end.

29. MERGE ( A, 5,8,11 ) 4 5 6 7 8 9 10 11 12
...
... 1 2 7 9 3 5 8 k A  1 2 7 9  3 5 8 i j L R

30. 4 5 6 7 8 9 10 11 12
...
... 1 2 7 9 3 5 8 k A  1 2 7 9  3 5 8 i j L R

31. 4 5 6 7 8 9 10 11 12
...
... 1 2 7 9 3 5 8 k A  1 2 7 9  3 5 8 i j L R

32. 4 5 6 7 8 9 10 11 12
...
... 1 2 3 9 3 5 8 k A  1 2 7 9  3 5 8 i j L R

33. 4 5 6 7 8 9 10 11 12
...
... 1 2 3 5 3 5 8 k A  1 2 7 9  3 5 8 i j L R

34. 4 5 6 7 8 9 10 11 12
...
... 1 2 3 5 7 5 8 k A  1 2 7 9  3 5 8 i j L R

35. 4 5 6 7 8 9 10 11 12
...
... 1 2 3 5 7 8 8 k A  1 2 7 9  3 5 8 i j L R

36. 4 5 6 7 8 9 10 11 12
...
... 1 2 3 5 7 8 9 k A  1 2 7 9  3 5 8 i j L R

37. MERGE-Correcteness Loop Invariant
At the start of each iteration of the for loop for k, the subarray A[p,..,k-1] consist of the k-p smallest elements of L[1,.., n1+1] and R[1,.., n2+1] in sorted order. Moreover L[i] and R[j] are the smallest of their arrays that have not been copied back into A.

38. INITILIZATION
Before the beginning of the loop k=p, then the subarray A[p,..,k-1] is empty and consist of the k-p = 0 smallest elements of L[1,.., n1+1] and R[1,.., n2+1]. Since i=j=1 then L[1] and R[1] are the smallest of their arrays that have not been copied back into A.

39. MAINTENANCE
Before the beginning of the l-th iteration of the loop k=p+l, then the subarray A[p,..,k-1] consist of the (k-p = l) smallest elements of L[1,.., n1+1] and R[1,.., n2+1] in sorted order and L[i] and R[j] are the smallest elements of their arrays that have not been copied back into A.

40. Let us first assume that L[i]  R[j] then following the loop L[i] is copied to A[k =p+l] therefore before the beginning of the (l+1)th k=p+l+1 and A[p,..,k-1] = A[p,..,p+l] consist of the (l+1) smallest elements of L[1,.., n1+1] and R[1,.., n2+1] and L[i] and R[j] are the smallest elements of their arrays that have not been copied back into A.

41. Now suppose that R[j] < L[i] then following the loop R[i] is copied to A[k =p+l] therefore before the beginning of the (l+1)th k=p+l+1 and A[p,..,k-1] = A[p,..,p+l] consist of the (l+1) smallest elements of L[1,.., n1+1] and R[1,.., n2+1] and L[i] and R[j] are the smallest elements of their arrays that have not been copied back into A.

42. TERMINATION
At termination k=r+1, then the subarray A[p,..,k-1]= A[p,..,r] is consist of the r smallest elements of L[1,.., n1+1] and R[1,.., n2+1] in sorted order. Since i= n1 +1 and j= n2 +1 then L[i] and R[j] are .

43. MERGE-SORT if p<r then q  (p+r)/2  MERGE-SORT( A, p, q )
procedure MERGE-SORT( A, p, r )
if p<r then
q  (p+r)/2 
MERGE-SORT( A, p, q )
MERGE-SORT( A, q+1, r )
MERGE ( A, p, q, r )
To sort call MERGE(A, 1, length[A]) with A = [a1,a2,a3,...,an ]
MERGE(A, p, r): procedure that takes time (n), where n=r-p+1

44. Initial Sequence divide divide divide merge merge merge
divide
divide
5 2
4 6
1 3
2 6
divide 5 2 4 6 1 3 2 6
merge
2 5
4 6
1 3
2 6
merge
merge
Sorted Sequence

45. Initial Sequence divide divide divide merge merge merge
divide
divide
5 2
4 6
1 3
2 6
divide 5 2 4 6 1 3 2 6
merge
2 5
4 6
1 3
2 6
merge
merge
Sorted Sequence

46. Analyzing divide and conquer algorithms
Time complexity
Analyzing divide and conquer algorithms
The time can often be described by recurrence equation fo the form
(1), if n  c,
T(n) =
aT(n/b)+D(n)+C(n) if n>c
With a,b and c be nonnegative constants.
If the problem is the small enough, say n  c, then the solution takes constant time (1).
If not the problem is divided in a subproblems with (1/b) size of the original. The division takes time D(n) and the combinations of sub-solutions takes time C(n).

47. a=2, b=2, c=1, D(n)=(1) and C(n)= (n)
Analyzing MERGE-SORT
In this case
a=2, b=2, c=1, D(n)=(1) and C(n)= (n)
then
(1), if n 1,
T(n) =
2T(n/2)+ (1)+ (n) if n>1
c if n 1,
2T(n/2)+ cn if n>1

48. Sorted Sequence merge merge merge Initial Sequence 1 2 2 3 4 5 6 6
2 5
2 6
1 3
4 6 5 2 4 6 1 3
merge
merge
merge
Initial Sequence

49. The construction of the recursion tree
Lets suppose n power of two
n=2k
c, if n 1,
T(n) =
2T(n/2)+ cn if n>1
T(n) cn
T(n/2)
T(n/2)

50. cn
cn/2
cn/2
T(n/4)
T(n/4)
T(n/4)
T(n/4)

51. cn
cn/2
cn/2
cn/4
cn/4
cn/4
cn/4
T(n/8)
T(n/8)
T(n/8)
T(n/8)
T(n/8)
T(n/8)
T(n/8)
T(n/8)

52. cn cn cn/2 cn/2 cn cn/4 cn/4 cn/4 cn/4 cn cn/8 cn/8 cn/8 cn/8 cn/8
lg n+1
cn/8
cn/8
cn/8
cn/8
cn/8
cn/8
cn/8
cn/8 cn . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . cn c c c c c c c c c c c c c c c c n
Total: cn (lg n + 1)
cn lg n + cn

53. Recursive substitution
Lets assume that n is power of two, i.e.,
n=2k, k = lg n
T(n) = 2T(n/2)+ cn
= 2[2T(n/4)+ cn/2]+ cn = 4T(n/4)+ cn+ cn
= 4[2T(n/8)+cn/4]+ cn+ cn= 8T(n/8)+cn+ cn+ cn .
= 2kT(n/2k)+cn cn =
= 2kT(1)+k(cn)
= cn+cn lg n
k times

54. Exact recursive recurrence
In total including other operations let’s say each merge costs 3 per element output
T(n)=T(n/2)+T(n/2)+3n for n2
T(1)=1
Can use this to figure out T for any value of n
T(5)=T(3)+T(2)+3x =(T(2)+T(1)+3x3)+(T(1)+T(1)+3x2) = ((T(1)+T(1)+6)+1+9)+(1+1+6) =
= 41
“ceiling” round up
“floor” round down