1. Merge Sort Algorithm Techniques
11/19/2018 4:16 AM
Algorithms
Algorithm Techniques

2. Steps in Designing Algorithms(1)
Understand the problem
Specify the range of inputs the algorithm should handle
Learn about the model of the implementation technology
RAM (Random-access machine), sequential execution
Choosing between an exact and an approximate solution
Some problems cannot be solved exactly: nonlinear equations, evaluating definite integrals
Exact solutions may be unacceptably slow
Choose the appropriate data structures

3. Steps in Designing Algorithms(2)
Choose an algorithm design technique
General approach to solve problems algorithmically that is applicable to a variety of computational problems
Provide guidance for developing solutions to new problems
Specify the algorithm
Pseudocode: mixture of natural and programming language
Prove the algorithm’s correctness
Algorithm yields the correct result for any legitimate input, in a finite amount of time
Mathematical induction, loop-invariants

4. Steps in Designing Algorithms(3)
8. Analyze the Algorithm
Predicting the amount of resources required:
memory: how much space is needed?
computational time: how fast the algorithm runs?
Input size (number of elements in the input)
Size of an array, polynomial degree, # of elements in a matrix, # of bits in the binary representation of the input, vertices and edges in a graph
Def: Running time = the number of primitive operations (steps) executed before termination
Arithmetic operations (+, -, *), data movement, control, decision making (if, while), comparison

5. Steps in Designing Algorithms(4)
Coding the algorithm
Verify the ranges of the input
Efficient/inefficient implementation
It is hard to prove the correctness of a program (typically done by testing)

6. Classification of Algorithms
By problem types
Sorting
Searching
String processing
Graph problems
Combinatorial problems
Geometric problems
Numerical problems
By design paradigms
Divide-and-conquer
Greedy algorithms
Incremental
Dynamic programming
Randomized/probabilistic

7. fundamental design paradigms
Merge Sort
11/19/2018 4:16 AM
fundamental design paradigms
Greedy method
Often used in problems involving
weighted graphs
data compression problems
Divide and conquer method
Already seen this is merge-sort and quick-sort
Here we’ll concentrate on the analyzing of problems solved by this method by solving recurrence relations
Dynamic programming
Very powerful technique IF we can build a certain characterization
Used in solving many problems that superficially do not seem to have much in common.
There are other paradigms, but these are really quite basic.

8. Merge Sort
11/19/2018 4:16 AM
The Greedy Method

9. Greedy algorithms
Suppose it is possible to build a solution through a sequence of partial solutions
At each step, we focus on one particular partial solution and we attempt to extend that solution
Ultimately, the partial solutions should lead to a feasible solution which is also optimal 9

10. The Greedy Method Technique
Merge Sort
11/19/2018 4:16 AM
The Greedy Method Technique
The greedy method is a general algorithm design paradigm, built on the following elements:
configurations: different choices, collections, or values to find
an objective function: a score assigned to configurations, which we want to either maximize or minimize
A globally-optimal solution can always be found by a series of local improvements from a starting configuration.

11. Making change Consider this commonplace example:
Making the exact change with the minimum number of coins
Consider the Euro denominations of 1, 2, 5, 10, 20, 50 cents
Stating with an empty set of coins, add the largest coin possible into the set which does not go over the required amount 11

12. Making change Total €0.50 To make change for €0.72: Start with €0.5012

13. Making change Total €0.70 To make change for €0.72: Start with €0.50
Add a €0.20
Total €0.70 13

14. Making change Total €0.72 To make change for €0.74: Start with €0.50
Add a €0.20
Skip the €0.10 and the €0. 05 but add a €0.02
Total €0.72 14

15. Making change Notice that each digit can be worked with separately
The maximum number of coins for any digit is three
Thus, to make change for anything less than €1 requires at most six coins
The solution is optimal 15

16. Definition A greedy algorithm is an algorithm which has:
A set of partial solutions from which a solution is built
An objective function which assigns a value to any partial solution
Then given a partial solution, we
Consider possible extensions of the partial solution
Discard any extensions which are not feasible
Choose that extension which minimizes the object function
This continues until some criteria has been reached

17. Example Given: A set S of n items, with each item i having
Merge Sort
11/19/2018 4:16 AM
Example
Given: A set S of n items, with each item i having
bi - a positive benefit
wi - a positive weight
Goal: Choose items with maximum total value but with weight at most W.
10 ml
“knapsack”
Solution:
1 ml of item 5
2 ml of item 3
6 ml of item 4
1 ml of item 2
Items: 1 2 3 4 5
Weight:
4 ml
8 ml
2 ml
6 ml
1 ml
Benefit:
$12
$32
$40
$30
$50
Value: 3 4 20 5 50
($ per ml)

18. Problems That Can Be Solved by the Greedy Method
Merge Sort
11/19/2018 4:16 AM
Problems That Can Be Solved by the Greedy Method
A game like chess can be won by thinking ahead.
But, a player focusing entirely on their immediate advantage is usually easy to defeat.
For example, in Scrabble, the player can do quite well by simply making whatever move seems best at the moment and not worrying about future consequences.

19. On each step in the algorithm, the choice must be:
Merge Sort
11/19/2018 4:16 AM
On each step in the algorithm, the choice must be:
Feasible - i.e. it satisfies the problems constraints
Locally optimal – i.e. it has to be the best local choice among all feasible choices available at the step
Irrevocable – i.e. once made, it cannot be changed on subsequent steps of the algorithm

20. Example: Making Change
Merge Sort
11/19/2018 4:16 AM
Problem: A dollar amount to reach and a collection of coin amounts to use to get there.
Configuration: A dollar amount to return to a customer plus the coins already returned
Objective function: Minimize number of coins returned.
Greedy solution: Always return the largest coin you can

21. The Fractional Knapsack Problem
Merge Sort
11/19/2018 4:16 AM
The Fractional Knapsack Problem
Given: A set S of n items, with each item i having
bi - a positive benefit
wi - a positive weight
Goal: Choose items with maximum total value but with weight at most W.
The value of an item is its benefit/weight ratio.
If we are allowed to take fractional amounts, then this is the fractional knapsack problem.
In this case, we let xi denote the amount we take of item i
0  xi  wi
Objective: maximize
Constraint:

22. The Fractional Knapsack Algorithm
Merge Sort
11/19/2018 4:16 AM
The Fractional Knapsack Algorithm
Algorithm fractionalKnapsack (S, W)
Input: set S of items with benefit bi and weight wi; max. weight W
Output: amount xi of each
item i to maximize benefit with weight at most W
for each item i in S
xi  0
vi  bi / wi {value}
w  0 {total weight}
while w < W
remove item i with highest vi
xi  min{wi , W - w}
w  w + min{wi , W - w}
Greedy choice: Keep taking item with highest value (benefit to weight ratio bi / wi )
Run time: O(n log n). Why?
Use a max-heap priority queue

23. Optimal and sub-optimal examples
An implementation of the greedy algorithm is straight-forward:
void greedy( int value, int *coins, int *rep, int n ) {
for ( int i = n - 1; i >= 0; --i ) {
rep[i] = 0;
while ( coins[i] <= value ) {
value -= coins[i];
++( rep[i] ); //++rep[i] also works }

24. Project management (26 weeks) 0/1 knapsack problem
Product ID
Completion Time (wks)
Expected Revenue (1000 $) A 15
210 B 12
220 C 10
180 D 9
120 E 8
160 F 7
170 G 5 90 H 4 40 J 3 60 K 1

25. Project management 0/1 knapsack problem
Greedy-by-time (make use of all 26 wks):
Project A: 15 wks
Project C: 10 wks
Project J: 1 wk
Total time: 26 wks
Expected revenue: $
Product ID
Completion Time (wks)
Expected Revenue
(1000 $) A 15
210 B 12
220 C 10
180 D 9
120 E 8
160 F 7
170 G 5 90 H 4 40 I 3 60 J 1

26. Project management 0/1 knapsack problem
Greedy-by-revenue (best-paying projects):
Project B: $220K
Project C: $180K
Project H: $ 60K
Project K: $ 10K
Total time: 26 wks
Expected revenue: $
Product ID
Completion Time (wks)
Expected Revenue
(1000 $) B 12
220 A 15
210 C 10
180 F 7
170 E 8
160 D 9
120 G 5 90 J 3 60 H 4 40 K 1

27. Project management 0/1 knapsack problem
Greedy-by-revenue-density:
Project F: $24 286/wk
Project E: $20 000/wk
Project J: $20 000/wk
Project G: $18 000/wk
Project K: $10 000/wk
Total time: 24 wks
Expected revenue: $
Bonus: 2 weeks for bug fixing
Product ID
Completion Time (wks)
Expected Revenue (1000 $)
Revenue Density
($/wk) F 7
170
24 286 E 8
160
20 000 J 3 60 B 12
220
18 333 C 10
180
18 000 G 5 90 A 15
210
14 000 D 9
120
13 333 H 4 40
10 000 K 1

28. Project management 0/1 knapsack problem
Using brute force, we find that the optimal solution is:
Project C: $
Project E: $
Project F: $
Project K: $
Total time: 26 wks
Expected revenue: $
Product ID
Completion Time (wks)
Expected Revenue (1000 $)
Revenue Density
($/wk) A 15
210
14 000 B 12
220
18 333 C 10
180
18 000 D 9
120
13 333 E 8
160
20 000 F 7
170
24 286 G 5 90 H 4 40
10 000 J 3 60 K 1

29. Process scheduling Consider the following processes:
Ref: Weiss, DS&AA in C++, 2nd Ed., p.410
Process (i)
Time (ti) 1
15 ms 2
8 ms 3
3 ms 4
10 ms

30. Process scheduling
The process time for each of the processes is summarized in this table:
Process
Time i1 i2 i3 i4
Sum

31. Process scheduling
For example, given 12 processes and three processors, we could schedule the processes as follows:

32. Process scheduling
For example, consider the processes and completion times listed in this table
Suppose we can run these processes on three different processors (assuming that the processes are not interdependent)
Process
Time
(ms) 1 3 2 5 6 4 10 11 14 7 15 8 18 9 20

33. Other Problems That Can Use the Greedy Method
Merge Sort
11/19/2018 4:16 AM
Other Problems That Can Use the Greedy Method
Here are a few:
You are to network a collection of computers by linking selected pairs of them. Each link as a maintenance cost, reflected in a weight attached to the link. What is the cheapest possible network?
The MP3 audio compression scheme encodes a sound signal by using something called a Huffman encoding. In simple terms, given symbols A, B, C, and D, what is the shortest way that a binary string can encode the letters so any string can be decoded unambiguously?

34. Task Scheduling A start time, si A finish time, fi (where si < fi)
Merge Sort
11/19/2018 4:16 AM
Task Scheduling
Given: a set T of n tasks, each having:
A start time, si
A finish time, fi (where si < fi)
Goal: Perform all the tasks using a minimum number of “machines.”
Two tasks can execute on the same machine only if
fi<=sj or fj <=si. (called non-conflicting) 1 9 8 7 6 5 4 3 2
Machine 1
Machine 3
Machine 2

35. Example Given: a set T of n tasks, each having: A start time, si
Merge Sort
11/19/2018 4:16 AM
Example
Given: a set T of n tasks, each having:
A start time, si
A finish time, fi (where si < fi)
Goal: Perform all tasks on min. number of machines
Assume T is [4,7],[7,8],[1,4],[1,3],[2,5],[3,7],[6,9]
Machine 3
Machine 2
Machine 1 1 2 3 4 5 6 7 8 9
Order by the start time:
[1,4], [1,3], [2,5], [3,7], [4,7], [6,9], [7,8]

36. Task Scheduling Algorithm
Merge Sort
11/19/2018 4:16 AM
Task Scheduling Algorithm
Greedy choice: consider tasks by their start time and use as few machines as possible with this order.
Run time: O(n log n). Why?
Correctness: Suppose there is a better schedule.
We can use k-1 machines
The algorithm uses k
Let i be first task scheduled on machine k
Machine i must conflict with k-1 other tasks
But that means there is no non-conflicting schedule using k-1 machines
Algorithm taskSchedule(T)
Input: set T of tasks w/ start time si and finish time fi
Output: non-conflicting schedule with minimum number of machines
m  0 {no. of machines}
while T is not empty
remove task i w/ smallest si
if there’s a machine j for i then
schedule i on machine j
else
m  m + 1
schedule i on machine m

37. Greedy Algorithms
A greedy algorithm always makes the choice that looks best at the moment
The hope: a locally optimal choice will lead to a globally optimal solution
For some problems, it works
Dynamic programming can be overkill; greedy algorithms tend to be easier to code

38. Activity Selection: A Greedy Algorithm
So actual algorithm is simple:
Sort the activities by time
Schedule the first activity
Then schedule the next activity in sorted list which starts after previous activity finishes
Repeat until no more activities
Intuition is even more simple:
Always pick the shortest ride available at the time

39. Activity Selection i j j i
Schedule n activities that require exclusive use of a common resource
S = {a1, , an} – set of activities
ai needs resource during period [si , fi)
si = start time and fi = finish time of activity ai ……0  si < fi < 
Activities ai and aj are compatible if the intervals [si , fi) and [sj, fj) do not overlap
fi  sj
fj  si i j j i

40. Activity Selection Problem
Select the largest possible set of nonoverlapping (mutually compatible) activities.
E.g.: i 1 2 3 4 5 6 7 8 9 10 11 si 12 fi 13 14
Activities are sorted in increasing order of finish times
A subset of mutually compatible activities: {a3, a9, a11}
Maximal set of mutually compatible activities:
{a1, a4, a8, a11} and {a2, a4, a9, a11}

41. Characterizing the Subproblems
The original problem: find the maximum subset of mutually compatible activities for S = S0, n+1
Activities are sorted by increasing finish time
a0, a1, a2, a3, …, an+1
We always choose an activity with the earliest finish time
Greedy choice maximizes the unscheduled time remaining
Finish time of activities selected is strictly increasing

42. Example
k sk fk  a1
m=1 a0 a2 a1 a3 a1 a4
m=4 a1 a5 a1 a4 a6 a1 a4 a7 a1 a4 a8 a1 a4
m=8 a9 a1 a4 a8
a10 a1 a4 a8
a11 a1 a4 a8
m=11 a1 a4 a8
a11 1 2
COSC3101A 3 4 5 6 7 8 9 10 11 12 13 14

43. Steps Toward Our Greedy Solution
Determine the optimal substructure of the problem
Develop a recursive solution
Prove that one of the optimal choices is the greedy choice
Show that all but one of the subproblems resulted by making the greedy choice are empty
Develop a recursive algorithm that implements the greedy strategy
Convert the recursive algorithm to an iterative one

44. Designing Greedy Algorithms
Cast the optimization problem as one for which:
we make a choice and are left with only one subproblem to solve
Prove that there is always an optimal solution to the original problem that makes the greedy choice
Making the greedy choice is always safe
Demonstrate that after making the greedy choice:
the greedy choice + an optimal solution to the resulting subproblem leads to an optimal solution

45. Divide-and-Conquer 7 2  9 4  2 4 7 9 7  2  2 7 9  4  4 9 7  7
Merge Sort
11/19/2018 4:16 AM
Divide-and-Conquer
7 2  
7  2  2 7
9  4  4 9
7  7
2  2
9  9
4  4

46. Merge Sort
11/19/2018 4:16 AM
Divide-and-Conquer
Divide-and conquer is a general algorithm design paradigm:
Divide: divide the input data S in two or more disjoint subsets S1, S2, …
Recur: solve the subproblems recursively
Conquer: combine the solutions for S1, S2, …, into a solution for S
The base case for the recursion are subproblems of constant size
Analysis can be done using recurrence equations

47. Many Problems Fall to Divide and Conquer
Merge Sort
11/19/2018 4:16 AM
Many Problems Fall to Divide and Conquer
Mergesort and quicksort were mentioned earlier.
Compute gcd (greatest common divisor) of two positive integers.
Compute the median of a list of numbers.
Multiplying two polynomials of degree 2d
i.e. (1 + 2x + 3x^2) * (5 -4x + 8x^2)
FFT - Fast Fourier Transform used in signal processing.
(Closest Pair) Given points in the plane, find two that have the minimum distance between them.

48. Merge-Sort Algorithm mergeSort(S, C)
11/19/2018 4:16 AM
Merge-Sort
Algorithm mergeSort(S, C)
Input sequence S with n elements, comparator C
Output sequence S sorted
according to C
if S.size() > 1
(S1, S2)  partition(S, n/2)
mergeSort(S1, C)
mergeSort(S2, C)
S  merge(S1, S2)
Merge-sort on an input sequence S with n elements consists of three steps:
Divide: partition S into two sequences S1 and S2 of about n/2 elements each
Recur: recursively sort S1 and S2
Conquer: merge S1 and S2 into a unique sorted sequence

49. Searching an ordered matrix
Consider the following search for 19:
Search across until ai,j + 1 > 19
Alternate between
Searching down until ai,j > 19
Searching back until ai,j < 19
This requires us to check at most 3n entries: O(n)

50. Grade School Algorithm
2345
6789 x
21105

51. Grade School Algorithm
2345
6789 x
21105
18760
shift by 1 digit

52. Grade School Algorithm
2345
6789 x
21105
18760
shift by 1 digit
16415
shift by 2 digits

53. Observe that even the total work for shifting is
Grade School Algorithm
Observe that even the total work for shifting is
( … + n-1) = O(n2)!
2345
6789 x
21105
18760
shift by 1 digit
16415
shift by 2 digits
14070
shift by 3 digits +
Multiplying two digits => 1 operation
Addition of two digits => 1 operation
Shift by 1 digit => 1 operation (shift by x digits, x ops)
TOTAL WORK: O(n2)

54. Question: Can we do better?
Upshot: A set of mysterious looking multiplications, additions and subtractions giving the correct answer!
X = (a10n/2 + b); Y = (c10n/2 + d) a
23 45 b
each of a, b, c, d is of size n/2 c
67 89 d
Ex: a=23, b=45, so 2345 = 23*10n/2 + 45
XY = ac10n + ad10n/2 + bc10n/2 + bd
XY = ac10n + (ad + bc)10n/2 + bd
Call this expression (★)

55. Integer multiplication
Calculate the product of two 16-digit integers ×
Multiplying two n-digit numbers requires Q(n2) multiplications of two decimal digits: × n

56. Integer multiplication
Rewrite the product × as
( × ) × ( × )
which requires four multiplications of 8-digit integers:
( × )×1016 +
( × × )×108 +
( × )
Adding two n-digit integers is a Q(n) operation

57. Problem: Big Integer Multiplication
Merge Sort
11/19/2018 4:16 AM
Problem: Big Integer Multiplication
Problem: Given two n-bit integers, I and J, that can’t be handled by the hardware of a machine, devise an algorithm with good complexity that multiplies these two numbers.
Applications:
Encryption schemes used in security work.
Can we do better?
We will assume n is a power of 2; otherwise, pad with zeroes.
Note: This provides an alternate way of doing what we did in the first homework assignment which tacitly assumed the hardware could handle the products.

58. Integer multiplication
Thus, the recurrence relation is:
Again, we solve the recurrence relation using Maple:
> rsolve( {T(n) = 4*T(n/2) + n, T(1) = 1}, T(n) );
This is still Q(n2)

59. MATRIX OPERATIONS: Example
Merge Sort
11/19/2018 4:16 AM
MATRIX OPERATIONS: Example
Matrix-matrix multiplication: Given: A is n X r and B is r X m r
Then, C = AB = [c(i,j)] where c[i,j] =  a[i,k] b[k,j]
k=1
For example,c[2,3]=
a[2,1]b[1,3] +
a[2,2]b[2,3]=
1*1 + 2*0 = 1
Note that the following is undefined:
because a 2 X 4 matrix can't
be multiplied by a 3 X 2 matrix.
A 4 X m matrix is required.

60. Matrix Multiplication
Merge Sort
11/19/2018 4:16 AM
Matrix Multiplication
In trying to improve this, a first pass would look at the following:
Assume n is a power of 2 and view an array as made up of submatrices, i.e.
These can be handled
recursively by viewing this as
a 4 X 4 matrix as shown
and then breaking the 4 X 4
matrices into 2 X 2 matrices.

61. Matrix Multiplication
Merge Sort
11/19/2018 4:16 AM
Matrix Multiplication
Thus,
where
I= AE + BG
J = AF + BH
K = CE + DG
L = CF + DH
Then, use this idea to divide and conquer.

62. Matrix Multiplication
Merge Sort
11/19/2018 4:16 AM
Matrix Multiplication
With this approach,
T(n) = 8T(n/2) + bn^2
Unfortunately, the master theorem only gives us that T(n) is O(n^3), which isn't any improvement.
However, there is an algorithm called Strassen's Algorithm which is able to handle the multiplication in just seven recursive calls.
The technique can be verified (see pgs ) although the algebra is messy.

63. Divide and Conquer
We have now looked at a number of divide-and-conquer algorithms, and come up with a number of different run times:
Binary search T(n) = T(n/2) + Q(1) O(ln(n))
Tree traversals T(n) = 2T(n/2) + Q(1) Q(n)
Merge sort T(n) = 2T(n/2) + Q(n) Q(n ln(n))
Ordered-matrix search T(n) = 3T(n/2) + Q(1) O(nlg(3))
Integer multiplication T(n) = 3T(n/2) + Q(n) Q(nlg(3))
Toom-3 integer multiplication T(n) = 5T(n/3) + Q(n) Q(nlog3(5))
Matrix multiplication T(n) = 7T(n/2) + Q(n2) Q(nlg(7))
Fast Fourier transform T(n) = 2T(n/2) + Q(n) Q(n ln(n))