1. CMPT 438 Algorithms

2. Why Study Algorithms? Necessary in any computer programming problem
Improve algorithm efficiency: run faster, process more data, do something that would otherwise be impossible
Solve problems of significantly large size
Technology only improves things by a constant factor
Compare algorithms
Algorithms as a field of study
Learn about a standard set of algorithms
New discoveries arise
Numerous application areas
Learn techniques of algorithm design and analysis

3. What are Algorithms?
An algorithm is a sequence of computational steps that transform the input into the output.
An algorithm is also a tool for solving a well- specified computational problem.
E.g., sorting problem:
<31, 26, 41, 59, 58> is an instance of the sorting problem.

4. An algorithm is correct if, for every input instance, it halts with the correct output.

5. Analyzing Algorithms Predict the amount of resources required:
memory: how much space is needed?
computational time: how fast the algorithm runs?
FACT: running time grows with the size of the input
Input size (number of elements in the input)
Size of an array, # of elements in a matrix, # of bits in the binary representation of the input, vertices and edges in a graph
Def: Running time = the number of primitive operations (steps) executed before termination
Arithmetic operations (+, -, *), data movement, control, decision making (if, while), comparison

6. Algorithm Efficiency vs. Speed
E.g.: sorting n numbers (n = 106)
Friend’s computer = 109 instructions/second
Friend’s algorithm = 2n2 instructions
(insertion sort)
Your computer = 107 instructions/second
Your algorithm = 50nlgn instructions
(merge sort)

7. Algorithm Efficiency vs. Speed
To sort 100 million numbers:
Insertion sort takes more than 23 days
Merge sort takes under 4 hours

8. Typical Running Time Functions
1 (constant running time):
Instructions are executed once or a few times
logN (logarithmic)
A big problem is solved by cutting the original problem in smaller sizes, by a constant fraction at each step
N (linear)
A small amount of processing is done on each input element
N logN
A problem is solved by dividing it into smaller problems, solving them independently and combining the solution

9. Typical Running Time Functions
N2 (quadratic)
Typical for algorithms that process all pairs of data items (double nested loops)
N3 (cubic)
Processing of triples of data (triple nested loops)
NK (polynomial)
2N (exponential)
Few exponential algorithms are appropriate for practical use

10. Why Faster Algorithms?

11. Insertion Sort Idea: like sorting a hand of playing cards
Remove one card at a time from the table, and insert it into the correct position in the left hand
compare it with each of the cards already in the hand, from right to left

12. Example of insertion sort

14. INSERTION-SORT (A, n) ⊳ A[1 . . n] for j ←2 to n do key ← A[ j]
sorted
key n j 1
INSERTION-SORT (A, n) ⊳ A[1 . . n]
for j ←2 to n
do key ← A[ j]
i ← j –1
while i > 0 and A[i] > key
do A[i+1] ← A[i]
i ← i –1
A[i+1] = key
Insertion sort sorts the elements in place.

15. Analysis of Insertion Sort

16. Analysis of Insertion Sort

17. Running time
•Parameterize the running time by the size of the input, since short sequences are easier to sort than long ones.

18. Kinds of analyses Worst-case: • T(n) =maximum time of algorithm on any
input of size n.
Average-case:
• T(n) =expected time of algorithm over all inputs
of size n.
• Need assumption of statistical distribution of inputs.
Best-case:
• Cheat with a slow algorithm that works fast on
some input.

19. Machine-independent time
What is insertion sort’s worst-case time?
•It depends on the speed of our computer
BIG IDEA:
•Ignore machine-dependent constants.
•Look at growth of T(n) as n→∞.
“Asymptotic Analysis”

20. Θ-notation Math: Θ(g(n)) = { f (n): there exist positive
constants c1, c2, and n0
such that 0 ≤c1g(n) ≤f (n) ≤c2g(n)
for all n≥n0}

21. Θ-notation Engineering:
•Drop low-order terms; ignore leading constants.
•Example: 3n3 + 90n2–5n = Θ(n3)

22. Best Case Analysis

23. Best Case Analysis The array is already sorted
A[i] ≤ key upon the first time the while loop test is run (when i = j -1)
tj = 1

24. Worst Case Analysis

25. Worst Case Analysis The array is in reverse sorted order
Always A[i] > key in while loop test
Have to compare key with all elements to the left of the j-th position
compare with j-1 elements
tj = j

26. Average Case?
All permutations equally likely.

27. Insertion Sort Summary
Advantages
Good running time for “almost sorted” arrays θ(n)
Disadvantages
θ(n2) running time in worst and average case
Is insertion sort a fast sorting algorithm?
•Moderately so, for small n.
•Not at all, for large n.

28. Worst-Case and Average-Case
We usually concentrate on finding only the worst-case running time
an upper bound on the running time
For some algorithms, the worst case occurs often.
E.g., searching when information is not present in the DB
The average case is often as bad as the worst case.

29. Merge Sort

30. Merge Sort MERGE-SORT A[1 . . n] 1.If n= 1, done.
2.Recursively sort A[ n/2]and
A[ [n/2] n ] .
3.“Merge” the 2 sorted lists.

31. Example

32. Divide-and-Conquer Divide the problem into a number of subproblems
Similar sub-problems of smaller size
Conquer the sub-problems
Solve the sub-problems recursively
Sub-problem size small enough to solve the problems in straightforward manner
Combine the solutions to the sub-problems
Obtain the solution for the original problem

33. Merge Sort Approach To sort an array A[p . . r]: Divide Conquer
Divide the n-element sequence to be sorted into two subsequences of n/2 elements each
Conquer
Sort the subsequences recursively using merge sort
When the size of the sequences is 1 there is nothing more to do
Combine
Merge the two sorted subsequences

34. Merge sort

35. Merge sort

36. Analyzing merge sort MERGE-SORT A[1 . . n] T(n) 1.If n= 1, done. Θ(1)
2.Recursively sort A[ 「 n/2 」]
and A[「n/2」 n ] .
3.“Merge”the 2sorted lists
T(n)
Θ(1)
2T(n/2) ?
Sloppiness: Should be T(「 n/2 」) + T(「n/2」) ,
but it turns out not to matter asymptotically.

37. Merging two sorted arrays

38. Merging two sorted arrays
20 12
13 11
7 9 2
20 12
13 11
7 9
20 12
13 11 9
20 12
13 11
20 12 13
20 12
13 11
7 9
2 1 1 2 11 12 7 9
Time?
In place sort?

39. Merging two sorted arrays
20 12
13 11
7 9 2
20 12
13 11
7 9
20 12
13 11 9
20 12
13 11
20 12 13
20 12
13 11
7 9
2 1 1 2 11 12 7 9
Time = Θ(n) to merge a total
of n elements (linear time).

40. Analyzing Divide and Conquer Algorithms
T(n) = aT(n/b) + D(n) + C(n)
The recurrence is based on the three steps of the paradigm:
T(n) – running time on a problem of size n
Divide the problem into a subproblems, each of size n/b: takes D(n)
Conquer (solve) the subproblems: takes aT(n/b)
Combine the solutions: takes C(n)

41. MERGE – SORT Running Time
T(n) = 2T(n/2) + θ(n) if n > 1
Divide:
compute q as the average of p and r: D(n) = θ(1)
Conquer:
recursively solve 2 subproblems, each of size n/2 -> 2T (n/2)
Combine:
MERGE on an n-element subarray takes θ(n) time C(n) = θ(n)

42. Recurrence for merge sort
Θ(1) if n= 1;
2T(n/2)+ Θ(n) if n> 1.
T(n) =

43. Recursion tree
Solve T(n) = 2T(n/2) + cn, where c > 0 is constant.

44. Conclusions • Θ(n lg n) is better than Θ(n2).
• Therefore, merge sort asymptotically beats insertion sort in the worst case.
Disadvantage
Requires extra space Θ (n)

45. Divide-and-Conquer Example: Binary Search
Find an element in a sorted array: 1. Divide: Check middle element. 2. Conquer: Recursively search 1 subarray. 3. Combine: Trivial. A[8] = {1, 2, 3, 4, 5, 7, 9, 11} Find 7

46. For an ordered array A, finds if x is in the array A[lo…hi]

47. Analysis of Binary Search?

48. Divide-and-Conquer Example: Powering a Number? ? ? ?

49. Homework 1
Quiz