1. Advanced Algorithms Analysis and DesignBy
Dr. Nazir Ahmad Zafar
Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

2. Lecture No. 24 Optimal Binary Search Trees
Constructing Dynamic Programming
Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

3. Construction: Optimal Substructure
One of the keys in ki, …,kj, say kr, i ≤ r ≤ j, must be the root of an optimal subtree for these keys.
Left subtree of kr contains ki,...,kr-1.
Right subtree of kr contains kr+1, ...,kj.
To find an optimal BST:
Examine all candidate roots kr , for i ≤ r ≤ j
Determine all optimal BSTs containing ki,...,kr-1 and containing kr+1,...,kj kr ki
kr-1
kr+1 kj
Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

4. Construction: Optimal Substructure
Find optimal BST for ki,...,kj, where i ≥1, j ≤ n, j ≥ i1
When j = i1, the tree is empty.
Define e[i, j ] = expected search cost of optimal BST for ki,...,kj.
If j = i1, then e[i, j ] = qi-1.
If j ≥ i,
Select a root kr, for some i ≤ r ≤ j .
Recursively make an optimal BSTs
for ki,..,kr1 as the left subtree, and
for kr+1,..,kj as the right subtree.
Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

5. Lemma
Prove that when OPT tree becomes a sub-tree of a node then expected search cost increases by
Proof
Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

6. Construction: Optimal Substructure
When OPT subtree becomes a subtree of a node:
Depth of every node in OPT subtree goes up by 1.
Expected search cost increases by
If kr is the root of an optimal BST for ki,..,kj
e[i, j ] = pr + (e[i, r1] + w(i, r1))+(e[r+1, j] + w(r+1, j))
= e[i, r1] + e[r+1, j] + w(i, j).
But, we don’t know kr. Hence,
Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

7. Algorithm: Optimal Binary Search
OPTIMAL-BST(p, q, n)
for i ← 1 to n + 1
do e[i, i 1] ← qi-1.
w[i, i 1] ← qi-1.
for l ← 1 to n
do for i ← 1 to nl + 1
do j ←i + l1
e[i, j ]←∞
w[i, j ] ← w[i, j1] + pj + qj
for r ←i to j
do t ← e[i, r1] + e[r + 1, j ] + w[i, j ]
if t < e[i, j ]
then e[i, j ] ← t
root[i, j ] ←r
return e and root
Consider all trees with l keys.
Consider all trees with l keys.
Fix the first key.
Fix the first key.
Fix the last key
Fix the last key
Determine the root of the optimal sub-tree
Determine the root of the optimal sub-tree
Dr. Nazir A. Zafar Advanced Algorithms Analysis and Design

8. Greedy Algorithms
Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

9. Today Covered Introduction to Greedy Algorithms Why Greedy Algorithm?
Where Greedy?
Where Greed Algorithm do not work?
Activity Selection Problem
Steps developing activity selection algorithm
Dynamic programming solution
Greedy choice
Recursive algorithm
Iterative algorithm
Conclusion
Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

10. Why Greedy Algorithm?
The algorithms we have studied in dynamic programming are relatively inefficient, for example
Cost of 0-1 knapsack problem: O(nW)
Cost of matrix chain multiplication: O(n3)
Cost in longest common subsequence: O(mn)
Optimal binary search trees: O(n3)
This is because
We have many choices computing optimal solution.
We check all of them in dynamic programming.
We must think which choice is the best or
At least restrict the choices we have to try.
Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

11. What are Greedy Algorithm?
In greedy algorithms, we do the same thing
Mostly optimization algorithms go through a sequence of steps, with a set of choices at each step.
In dynamic programming best choices is ignored
Some times a simpler and efficient algorithm required.
Greedy algorithms make best choice at a moment.
It makes a locally optimal choice in the hope that this choice will lead to a globally optimal solution.
Greedy algorithms do not always yield an optimal solutions, but mostly they do
They tend to be easier to implement.
Dr. Nazir A. Zafar Advanced Algorithms Analysis and Design

12. Where Greedy Algorithms do not work?
Greedy choice:
We make the choice that looks best at a moment.
Every time make a choice, greedily maximize value
An example where this does not work
Find longest monotonically increasing subsequence
Given sequence < >
Longest such subsequence is < >.
The greedy choice after choosing <3 4 5> is to choose 17, which is an unwanted element, results in the sequence < > which is suboptimal.
Dr. Nazir A. Zafar Advanced Algorithms Analysis and Design

13. Activity Selection Problem
Dr. Nazir A. Zafar Advanced Algorithms Analysis and Design

14. Some Definitions Closed Interval = [a, b] = {x R | a ≤ x ≤ b}
Open Interval =
(a, b) = {x R | a < x < b}
Left Semi Open =
(a, b] = {x R | a < x ≤ b}
Right Semi Open =
[a, b) = {x R | a ≤ x < b}
Dr. Nazir A. Zafar Advanced Algorithms Analysis and Design

15. Activity Selection Problem
The problem involves scheduling of several competing activities that require exclusive use of common resource
Problem Statement
The problem can be stated as, suppose we have a set: S = {a1, a2, ..., an} of n proposed activities.
Each activity wish to use a resource which can be used by only one activity at a time.
Each activity ai has starting time si, finishing time fi
where, 0 ≤ si < fi < ∞
Objective in activity-selection problem is to select a maximum-size subset of mutually compatible activities.
Dr. Nazir A. Zafar Advanced Algorithms Analysis and Design

16. Steps in Designing Activity Selection Algorithm
Steps to solve the problem
We will formulate the dynamic-programming solution in which
Combine optimal solutions to two subproblems to form an optimal solution to original problem
Check several choices when determining which subproblems to use in an optimal solution
Then needed to make a greedy choice in which
One of the subproblems guaranteed empty, so that only one nonempty subproblem remains
Then a recursive greedy algorithm is developed and converted to an iterative one
Dr. Nazir A. Zafar Advanced Algorithms Analysis and Design

17. Application : Scheduling Problem
A classroom can be used for one class at a time.
There are n classes that want to use the classroom.
Every class has a corresponding time interval Ij = [sj, fj) during which the room would be needed for this class.
Our goal is to choose a maximal number of classes that can be scheduled to use the classroom without two classes ever using the classroom at the same time.
Assume that the classes are sorted according to increasing finish times; that is, f1 < f2 < … < fn. 1 2 3 4 5 6 7 8
Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

18. Designing Activity Selection Problem
Compatible Activity
If a selected activity ai is required to take place during the half-open time interval [si, fi). And activity aj is required to take place during the half-open time interval [sj, fj). Then the activities ai and aj are compatible if the intervals [si, fi) and [sj, fj) do not overlap i.e
si ≥ fj or sj ≥ fi
Compatible Activities
Not Compatible Activities ai aj bi bj
Dr. Nazir A. Zafar Advanced Algorithms Analysis and Design

19. Compatible not Maximal1 2 3 4 5 6 7 8 9 10 11 si 12 fi 13 14
A set S of activities sorted in increasing order of finish time
The subset consisting of mutually compatible activities are
{a3, a9, a11} but it is not a maximal subset,
{a1, a4, a8, a11} is larger.
{a2, a4, a9, a11} is another largest subset.
Dr. Nazir A. Zafar Advanced Algorithms Analysis and Design

20. Optimal Substructure of Activity Selection Problem
The first step is to find optimal substructure and then use it to construct an optimal solution to the problem from optimal solutions to subproblems.
Let us start by defining sets
Sij = {ak  S : fi ≤ sk < fk ≤ sj}
Sij is the subset of activities in S that can start after activity ai finishes and finish before activity aj starts.
The fictitious activities a0 and an+1 are added and adopt the conventions that f0 = 0 and sn+1 = ∞. Then S = S0.n+1, and the ranges for i and j are given by
0 ≤ i, j ≤ n + 1.
Let us assume that Aij is a solution to Sij
Dr. Nazir A. Zafar Advanced Algorithms Analysis and Design

21. Optimal Substructure of Problem
Let us assume that the activities are sorted in monotonically increasing order of finish times of the activities:
f0 ≤ f1 ≤ f2 ≤ … ≤ fn < fn+1
Assuming that we have sorted set of activities, our space of subproblems is
to select a maximum-size subset of mutually compatible activities from Sij, for 0 ≤ i < j ≤ n + 1,
knowing that all other Sij are empty.
Dr. Nazir A. Zafar Advanced Algorithms Analysis and Design

22. Decomposition: Optimal Substructure of Problem
Suppose that a solution to Sij is non-empty and includes some activity ak, so that
fi ≤ sk < fk ≤ sj.
After choosing activity ak, it decomposes Sij into two subproblems, Sik and Skj
Sik = activities that start after ai finishes and finish before ak starts and
Skj = activities that start after ak finishes and finish before aj starts,
Each of Sik and Skj are subset of the activities in Sij
Dr. Nazir A. Zafar Advanced Algorithms Analysis and Design

23. Solution to Si,j : Optimal Substructure of Problem
Our solution to Sij is the union of the solutions to Sik and Skj, along with the activity ak.
Thus, the number of activities in our solution to Sij is the size of our solution to Sik, plus the size of our solution to Skj , plus ak.
Solution(Sij) = Solution(Sik)  Solution(Skj) {ak}
Aij = Aik  Akj {ak}
Suppose we now that an optimal solution Aij includes activity ak, then the solutions Aik and Akj used within this optimal solution must be optimal.
Dr. Nazir A. Zafar Advanced Algorithms Analysis and Design

24. Why Ai k and Ak j are Optimal?
Proof Why Aik and Akj are Optimal?
If we had a solution A’ik to Sik that included more activities than Aik, we could cut out Aik from Aij and paste A’ik in Aij , thus producing another solution A’ij to Sij with more activities than Aij.
Because we assumed that Aij is an optimal solution, we have derived a contradiction.
Similarly, if we had a solution A’kj to Skj with more activities than Akj, we could replace Akj by A’kj to produce solution to Sij with more activities than Aij
Dr. Nazir A. Zafar Advanced Algorithms Analysis and Design

25. Further Decomposition to Find S0, n+1
Now we can build a maximum-size subset of mutually compatible activities in Sij
by splitting the problem into two subproblems, mutually compatible activities in Sik and Skj
finding maximum-size subsets Aik and Akj of theses activities for these subproblems and then
forming maximum-size subset Aij as
Aij = Aik  {ak}  Akj
Optimal solution to entire problem is: A0,n+1.
Dr. Nazir A. Zafar Advanced Algorithms Analysis and Design

26. A Recursive Solution
Let c[i, j] = number of activities in a maximum-size subset of mutually compatible activities in Sij.
We have c[i, j] = 0 whenever Sij = Ø;
In particular we have, c[i, j] = 0 for i ≥ j.
Since Aij = Aik  {ak}  Akj
Therefore the recurrence relation for the problem is
c[i, j ] = c[i, k] + c[k, j ] + 1.
Since value of k varies between i + 1, ..., j – 1, and hence there are (j - 1) - (i - 1) + 1 possible values of k, i.e., j - 1 – i = j – i - 1
Dr. Nazir A. Zafar Advanced Algorithms Analysis and Design

27. A Recursive Solution
Since maximum-size subset of Sij must use one of these values for k, we check them all to find the best.
Thus, our full recursive definition of c[i, j] becomes
Now it is a straightforward exercise to write a tabular, bottom-up, dynamic programming algorithm based on recurrence relation defined in previous slide.
Dr. Nazir A. Zafar Advanced Algorithms Analysis and Design