1. Advanced Algorithms Analysis and DesignBy
Dr. Nazir Ahmad Zafar
Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

2. Lecture No. 23 Longest Common Subsequence. (Dynamic Algorithm)
Lecture No Longest Common Subsequence (Dynamic Algorithm) Optimal Binary Search Trees
Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

3. Theorem: Optimal Substructure of an LCS
If X = (x1, x2,. . ., xm), and Y = (y1, y2, . . ., yn) be sequences and let us suppose that Z = (z1, z2, . . ., zk) be a longest common sub-sequence of X and Y
if xm = yn, then zk = xm and Zk – 1 is LCS of Xm – 1, Yn-1.
If xm  yn, then zk  xm implies that Z is LCS of Xm – 1 and Y
If xm  yn, then zk  yn implies that Z is LCS of X and Yn – 1
Dr. Nazir A. Zafar Advanced Algorithms Analysis and Design

4. Example
Problem
If X = <A, B, C, B, D, A, B>, and Y = <B, D, C, A, B, A> are two sequences then compute a maximum-length common subsequence of X and Y.
Solution:
Let c(i, j) = length of LCS of Xi and Yj, now we have to compute c(7, 6).
The recursive mathematical formula computing LCS is given below
Dr. Nazir A. Zafar Advanced Algorithms Analysis and Design

5. Example
If X = <A, B, C, B, D, A, B>, Y = <B, D, C, A, B, A>
c(1, 1) = max (c(0, 1), c(1, 0)) = max (0, 0) = 0
b[1, 1] = 
c(1, 2) = max (c(0, 2), c(1, 1)) = max (0, 0) = 0
b[1, 2] = 
c(1, 3) = max (c(0, 3), c(1, 2)) = max (0, 0) = 0
b[1, 3] = 
c(1, 4) = c(0, 3) + 1 = = 1; b[1, 4] =
c(1, 5) = max (c(0, 5), c(1, 4)) = max (0, 1) = 1
b[1, 5] = 
c(1, 6) = c(0, 5) + 1 = = 1; b[1, 6] = ↖
Dr. Nazir A. Zafar Advanced Algorithms Analysis and Design

6. Example
If X = <A, B, C, B, D, A, B>, Y = <B, D, C, A, B, A>
c(2, 1) = c(1, 0) + 1 = = 1; b[2, 1] =
c(2, 2) = max (c(1, 2), c(2, 1)) = max (0, 1) = 1
b[2, 2] = 
c(2, 3) = max (c(1, 3), c(2, 2)) = max (0, 1) = 1 b[2, 3] = 
c(2, 4) = max (c(1, 4), c(2, 3)) = max (1, 1) = 1 b[2, 4] = 
c(2, 5) = c(1, 4) + 1 = = 2; b[2, 5] =
c(2, 6) = max (c(1, 6), c(2, 5)) = max (1, 2) = 2
b[2, 6] =  ↖ ↖
Dr. Nazir A. Zafar Advanced Algorithms Analysis and Design

7. Example
If X = <A, B, C, B, D, A, B>, Y = <B, D, C, A, B, A>
c(3, 1) = max (c(2, 1), c(3, 0)) = max (1, 0) = 1 b[3, 1] = 
c(3, 2) = max (c(2, 2), c(3, 1)) = max (1, 1) = 1
b[3, 2] = 
c(3, 3) = c(2, 2) + 1 = = 2; b[3, 3] =
c(3, 4) = max (c(2, 4), c(3, 3)) = max (1, 2) = 2
b[3, 4] = 
c(3, 5) = max (c(2, 5), c(3, 4)) = max (2, 2) = 2 b[3, 5] = 
c(3, 6) = max (c(2, 6), c(3, 5)) = max (2, 2) = 2 b[2, 6] =  ↖
Dr. Nazir A. Zafar Advanced Algorithms Analysis and Design

8. Example
If X = <A, B, C, B, D, A, B>, Y = <B, D, C, A, B, A>
c(4, 1) = c(3, 0) + 1 = = 1; b[4, 1] =
c(4, 2) = max (c(3, 2), c(4, 1)) = max (1, 1) = 1
b[4, 2] = 
c(4, 3) = max (c(3, 3), c(4, 2)) = max (2, 1) = 2
b[4, 3] = ↑
c(4, 4) = max (c(3, 4), c(4, 3)) = max (2, 2) = 2
b[4, 4] = 
c(4, 5) = c(3, 4) + 1 = = 3; b[4, 5] =
c(4, 6) = max (c(3, 6), c(4, 5)) = max (2, 3) = 3
b[4, 6] =  ↖ ↖
Dr. Nazir A. Zafar Advanced Algorithms Analysis and Design

9. Example
If X = <A, B, C, B, D, A, B>, Y = <B, D, C, A, B, A>
c(5, 1) = max (c(4, 1), c(5, 0)) = max (1, 0) = 1
b[5, 1] = ↑
c(5, 2) = c(4, 1) + 1 = = 2; b[5, 2] =
c(5, 3) = max (c(4, 3), c(5, 2)) = max (2, 2) = 2
b[5, 3] = 
c(5, 4) = max (c(4, 4), c(5, 3)) = max (2, 2) = 2
b[5, 4] = 
c(5, 5) = max (c(4, 5), c(5, 4)) = max (3, 2) = 3
b[5, 5] = ↑
c(5, 6) = max (c(4, 6), c(5, 5)) = max (3, 3) = 3
b[5, 6] =  ↖
Dr. Nazir A. Zafar Advanced Algorithms Analysis and Design

10. Example
If X = <A, B, C, B, D, A, B>, Y = <B, D, C, A, B, A>
c(6, 1) = max (c(5, 1), c(6, 0)) = max (1, 0) = 1
b[6, 1] = ↑
c(6, 2) = max (c(5, 2), c(6, 1)) = max (2, 1) = 2
c(6, 3) = max (c(5, 3), c(6, 2)) = max (2, 2) = 2
b[6, 3] = 
c(6, 4) = c(5, 3) + 1 = = 3; b[6, 4] =
c(6, 5) = max (c(5, 5), c(6, 4)) = max (2, 3) = 3
b[6, 5] = 
c(6, 6) = c(5, 5) + 1 = = 4; b[6, 6] = ↖ ↖
Dr. Nazir A. Zafar Advanced Algorithms Analysis and Design

11. Example
If X = <A, B, C, B, D, A, B>, Y = <B, D, C, A, B, A>
c(7, 1) = c(6, 0) + 1 = = 1; b[7, 1] =
c(7, 2) = max (c(6, 2), c(7, 1)) = max (2, 1) = 2
b[7, 2] = ↑
c(7, 3) = max (c(6, 3), c(7, 2)) = max (2, 2) = 2
b[7, 3] = 
c(7, 4) = max (c(6, 4), c(7, 3)) = max (3, 2) = 3
b[7, 4] = ↑
c(7, 5) = c(6, 4) + 1 = = 4; b[7, 5] =
c(7, 6) = max (c(6, 6), c(7, 5)) = max (4, 4) = 4
b[7, 6] =  ↖ ↖
Dr. Nazir A. Zafar Advanced Algorithms Analysis and Design

12. Results j 1 2 3 4 5 6 j 1 2 3 4 5 6 i yj B D C A B A i yj B D C A B A1 2 3 4 5 6 j 1 2 3 4 5 6 i yj B D C A B A i yj B D C A B A xi xi • 1 A • ← ← 1 ↖ 1 ← 1 ↖ 1 A ← 2 B 1 2 • ↖ ← ↑ 2 B 3 C 3 C 4 B • 1 ↖ 1 ← 2 ↑ 2 ← 3 ↖ 3 ← 4 B 5 D 1 2 3 4 • ↑ ↖ ← 5 D 6 A 6 A 7 B 7 B
Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

13. Computable Tables j 1 2 3 4 5 6 j 1 2 3 4 5 6 i yj B D C A B A i yj B1 2 3 4 5 6 j 1 2 3 4 5 6 i yj B D C A B A i yj B D C A B A xi xi • • • • • • • A 1 1 1 1 1 A • ← ← ← ↖ ← ↖ B 2 1 1 1 1 2 2 2 B • ↖ ← ← ← ↖ ← C 3 1 1 2 2 2 2 3 C • ↑ ← ↖ ← ← ← B 4 1 1 2 2 3 3 4 B • ↖ ← ↑ ← ↖ ← D 5 1 2 2 2 3 3 5 D • ↑ ↖ ← ← ↑ ← A 6 1 2 2 3 3 4 6 A • ↑ ↑ ← ↖ ← ↖ B 7 1 2 2 3 4 4 7 B • ↖ ↑ ← ↑ ↖ ←
Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

14. Computable Tables j 1 2 3 4 5 6 j 1 2 3 4 5 6 i yj B D C A B A i yj B1 2 3 4 5 6 j 1 2 3 4 5 6 i yj B D C A B A i yj B D C A B A xi xi • • • • • • • 1 A 1 1 1 1 A • ← ← ← ↖ ← ↖ 2 B 1 1 1 1 2 2 2 B • ↖ ← ← ← ↖ ← 3 C 1 1 2 2 2 2 3 C • ↑ ← ↖ ← ← ← 4 B 1 1 2 2 3 3 4 B • ↖ ← ↑ ← ↖ ← 5 D 1 2 2 2 3 3 5 D • ↑ ↖ ← ← ↑ ← 6 A 1 2 2 3 3 4 6 A • ↑ ↑ ← ↖ ← ↖ 7 B 1 2 2 3 4 4 7 B • ↖ ↑ ← ↑ ↖ ←
Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

15. Computable Tables Table size: O(n.m)
Every entry takes O(1) time to compute.
The algorithm takes O(n.m) time and space.
The space complexity can be reduced to 2 · min(m, n) + O(1). j 1 2 3 4 5 6 i yj B D C A B A xi 1 A 1 1 1 2 B 1 1 1 1 2 2 3 C 1 1 2 2 2 2 4 B 1 1 2 2 3 3 5 D 1 2 2 2 3 3 6 A 1 2 2 3 3 4 7 B 1 2 2 3 4 4
Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

16. Longest Common Subsequence Algorithm
c[i, j] = c(i-1, j-1) if xi = yj;
c[i, j] = max( c(i-1, j), c(i, j-1)) if xi ≠ yj;
c[i, j] = if (i = 0) or (j = 0).
function LCS(X, Y)
m  length [X]
n  length [Y]
for i  1 to m
do c[i, 0]  0;
for j  1 to n
do c[0, j]  0;
Dr. Nazir A. Zafar Advanced Algorithms Analysis and Design

17. Longest Common Subsequence Algorithm
c[i, j] = c(i-1, j-1) if xi = yj;
c[i, j] = max( c(i-1, j), c(i, j-1)) if xi ≠ yj;
c[i, j] = if (i = 0) or (j = 0).
for i  1 to m
do for j  1 to n
do if (xi = = yj)
then c[i, j]  c[i-1, j-1] + 1
b[i, j]  “ ”
else if c[i-1, j]  c[i, j-1]
then c[i, j]  c[i-1, j]
b[i, j]  “↑”
else c[i, j]  c[i, j-1]
b[i, j]  “”
Return c and b;
Dr. Nazir A. Zafar Advanced Algorithms Analysis and Design

18. Construction of Longest Common Subsequence
c[i, j] = c(i-1, j-1) if xi = yj;
c[i, j] = min( c(i-1, j), c(i, j-1)) if xi ≠ yj;
c[i, j] = if (i = 0) or (j = 0).
procedure PrintLCS(b, X, i, j)
if (i == 0) or (j == 0)
then return
if b[i, j] == “ ”
then PrintLCS(b, X, i-1, j-1)
Print xi
else if b[i, j]  “↑”
then PrintLCS(b, X, i-1, j)
else PrintLCS(b, X, i, j-1)
Dr. Nazir A. Zafar Advanced Algorithms Analysis and Design

19. Relationship with shortest common supper-sequence
Shortest common super-sequence problem is closely related to longest common subsequence problem
Shortest common super-sequence
Given two sequences:
X = < x1,...,xm > and
Y = < y1,...,yn >
A sequence U = < u1,...,uk > is a common super-sequence of X and Y if U is a super-sequence of both X and Y.
The shortest common supersequence (scs) is a common supersequence of minimal length.
Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

20. Relationship with shortest common supper-sequence
Problem Statement
The two sequences X and Y are given and task is to find a shortest possible common supersequence.
Shortest common supersequence is not unique.
Easy to make SCS from LCS for 2 input sequences.
Example,
X[1..m] = abcbdab
Y[1..n] = bdcaba
LCS = Z[1..r] = bcba
Insert non-lcs symbols preserving order, we get
SCS = U[1..t] = abdcabdab.
Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

21. Optimal Binary Search Trees
Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

22. Binary Search Trees
Binary search tree (BST) is a binary data structure which has the following properties:
Each node has a value.
An order is defined on these values.
Left sub-tree of node contains values less than node value
Right sub-tree of a node contains only values greater than or equal to the node’s value.
Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

23. Optimal Binary Search Trees
Example: A translator from English to, say, Urdu.
Use a binary search tree to store all the words in our dictionary, together with their translations.
The word “the” is much more likely to be looked up than the word “ring”
So we would like to make the search time for the word “the” very short, possibly at the expense of increasing the search time for the word “ring.”
Problem Statement: We are given a probability distribution that determines, for every key in the tree, the likelihood that we search for this key.
The objective is to minimize the expected search time of the tree.
Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

24. Optimal Binary Search Trees
We need is known as an optimal binary search tree.
Formally, we are given a sequence K = <k1, k2, ..., kn> of n distinct keys in sorted order i.e. k1 < k2 < ··· < kn, and we wish to build a binary search tree from these keys.
For each ki, we have a probability pi that search is for ki.
Some searches may not be successful, so we have n + 1 “dummy keys” d0, d1, d2, ..., dn representing values not in K
In particular
d0 = represents all values less than k1,
dn = represents all values greater than kn, and
di = represents all values between ki and ki+1,
 i = 1, 2, ..., n -1
Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

25. Optimal Binary Search Trees
For each dummy key di, we have a probability qi that a search will correspond to di.
Each ki is an internal node, each dummy key di is a leaf.
Every search is either successful (finding some key ki) or failure (finding some dummy key di), and so we have d0 d1 d2 d3 d5 d6 d4 d7 d8 k1 k2 k3 k4 k5 k6 k8 k7
Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

26. Total Cost: Optimal Binary Search Trees
Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

27. Example Expected cost = 2.8 Let pi = probability of searching k1
qi = probability representing di i pi qi
Define cost of a search is as number of nodes examined in a search
Cost of a key = depth of the key + 1
Expected cost = 2.75 k2 k1 k5 k4 k3 d0 d1 d2 d3 d4 d5
Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

28. Brute Force Solution k1 k2 k4 k3 k5 d0 d3 d1 d2 d4 d5 k1 k2 k4 k3 k5 d0 d3 d1 d2 d5
Total number of binary trees will be exponential as in case of chain matrix problem
Brute force approach is not economical k2 k1 k5 k4 k3 d0 d1 d2 d3 d4 d5 k2 k1 k5 k4 k3 d0 d1 d2 d3 d4 d5
Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

29. Brute Force Solution Observations:
Optimal BST may not have smallest height.
Optimal BST may not have highest-probability key at root.
Build by exhaustive checking?
Construct each n-node BST.
For each, assign keys and compute expected search cost.
But there are (4n/n3/2) different BSTs with n nodes.
Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

30. Constructing Dynamic Programming
Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

31. Optimal Substructure Observation:
Any subtree of a BST contains keys range ki, ..., kj for some 1 ≤ i ≤ j ≤ n.
Lemma
If T is an optimal BST and T contains subtree T’ with keys ki, ... ,kj , then T’ must be an optimal BST for keys ki, ..., kj.
Proof:
Cut and paste method. T T’
Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

32. Limitations of Dynamic Programming
Dynamic programming can be applied to any problem that observes the principle of optimality.
Generally, it means that partial solutions can be optimally extended with regard to the state after the partial solution instead of the partial solution itself.
The biggest limitation using dynamic programming is number of partial solutions we must keep track of
For all examples we have seen, partial solutions can be described by stopping places in the input.
This is because combinatorial objects e.g. strings, numerical sequences, and polygons etc., all have an implicit order defined upon their elements.
Dr Nazir A. Zafar Advanced Algorithms Analysis and Design

33. Limitations of Dynamic Programming
This order cannot be changed without completely changing the origional problem.
Once order fixed, there are relatively few possible stopping places, and we get an efficient algorithms.
If objects are not firmly ordered then we have an exponential number of possible partial solutions
And we get an infeasible amount of memory resulting an infeasible solution.
Dr Nazir A. Zafar Advanced Algorithms Analysis and Design