UNIT-6 Recurrence Relations Yen (NTUEE) Discrete Mathematics 2012

Recurrence Relations Definition A recurrence relation for {an} is an equation that expresses an in terms of a0, . . . , an−1. A sequence is called a solution of a recurrence relation if its terms satisfy the recurrence relation. For instance, the relation fn = fn−1 + fn−2 of Fibonacci numbers is a recurrence relation. Yen (NTUEE) Discrete Mathematics 2012

Tower of Hanoi Example (The Tower of Hanoi) Consider moving a stack of disks with different sizes with three pegs. Initially, all disks are sorted and placed on the first peg. One can only move the topmost disk from a peg to another peg. At any time, disks on each peg must be sorted as well. How many steps does it take to move all disks from one peg to another? Solution: Let Hn be the number of steps to solve the problem of n disks. Clearly, H1 = 1. Now consider moving n disks from peg 1 to peg 2. If we can move the topmost n − 1 disks from peg 1 to peg 3, we can solve the puzzle by moving the bottom disk from peg 1 to peg 2 and then the n − 1 disks from peg 3 to peg 2. In other words, Hn = Hn−1 + 1 + Hn−1. We have Hn = 2Hn−1 + 1. Yen (NTUEE) Discrete Mathematics 2012

Tower of Hanoi Hence Hn = 2Hn−1 + 1 = 2(2Hn−2 + 1) + 1 = · · · = 2n−1H1 + 2n−2 + · · · + 21 + 20 n−1 X 2i i =0 = 2n − 1 = = 2n − 1 2 − 1 Yen (NTUEE) Discrete Mathematics 2012

= P Catalan numbers Example Find the number of ways to parenthesize the product of n + 1 numbers x0, x1, . . . , xn. Solution: Let Cn denote the number of ways to parenthesize the product of n + 1 numbers. For instance, C3 = 5 because x0 · (x1 · (x2 · x3)) x0 · ((x1 · x2) · x3) (x0 · (x1 · x2)) · x3 ((x0 · x1) · x2) · x3 are all the possible ways of multiplying x0 · x1 · · · · · x3. Clearly C0 = 1. Consider x0 · x1 · · · · · xn. We can compute it by (x0 · · · · · xk ) · (xk+1 · · · · · xn) for any k = 0, . . . , n. Therefore Cn = C0Cn−1 + C1Cn−2 + · · · + Cn−1C0. The sequence {C }, C = P n−1 n n C C 1, is called Catalan numbers. k=0 k n−k− We’ll give a closed form for Catalan numbers later. Yen (NTUEE) Discrete Mathematics 2012

Homogeneous Linear Recurrence Relations with Constant Coefficients Definition A linear homogeneous recurrence relation of degree k with constant coefficients is of the form an = c1an−1 + c2an−2 + · · · + ck an−k , where c1, c2, . . . , ck ∈ R with ck =6 0. For instance, the recurrence for Fibonacci numbers fn = fn−1 + fn−2 is a linear homogeneous recurrence relation of degree two. Theorem Let c1, c2 ∈ R. Suppose x 2 = c1x + c2 has two distinct roots r1 and r2. Then {an} is a solution of the recurrence relation an = c1an−1 + c2an−2 if and only if a = α r + α r for n ∈ N and some constants α , α . n n n 1 1 2 2 1 2 Yen (NTUEE) Discrete Mathematics 2012

Solving Linear Recurrence Relations Proof. (⇐) Suppose a = α r + α r . We want to verify a = c a n n n 1 1 2 2 n 1 n 1 − 2 n−2 + c a . c1an−1 c2an−2 c1an−1 + c2an−2 = c1α1r1 + c1α2r2 = c2α1r1 + c2α2r2 = = an n−1 n−1 n−2 n−2 α1r1 (c1r1 + c2) + α2r2 (c1r2 + c2) n−2 n−2 α1rn−2 2 r1 + α2r2 r2 n−2 2 1 α1rn + α2rn 1 2 (Cont’d on Next Page) Yen (NTUEE) Discrete Mathematics 2012

Solving Linear Recurrence Relations Proof. (Cont’d) (⇒) From the first part of proof, we know a = α r + α r n n n 1 1 2 2 satisfies the recurrence relation an = c1an−1 2 n−2 + c a . It remains to show an = α1rn + α2rn satisfies initial conditions for some α1, α2. The theorem 1 2 then follows from the uniqueness of solution to linear homogeneous recurrence relation. To see an = α1rn + α2rn satisfies the initial conditions for some α1, α2. 1 2 Consider a1 = α1r1 + α2r2 and a0 = α1 + α2. This is a linear system of two variables α1 and α2. The solutions are a1 − a0r2 α1 = , r − r 1 2 and a r − a α2 = 0 1 1 . r1 − r2 Yen (NTUEE) Discrete Mathematics 2012

1 + √5 !n √ n √ n Fibonacci numbers Example 5 , we Recall the recurrence relation for Fibonacci numbers fn = fn−1 + fn−2 with f0 = 0 and f1 = 1. Find an explicit formula for the Fibonacci numbers. Solution: The solutions to x 2 = x + 1 are 1± 5 . Hence √ 2 1 + √5 !n 1 √ n − 5 ! fn = α1 + α2 2 2 √ √ for some α1 and α2. Solving 0 = α1 + α2, and 1 = α1 1+ 5 + α2 1− 5 , we 2 2 have α1 = and α = − . Therefore 1 1 √5 2 √5 1 1 + √5 !n 1 − 1 √ n 5 ! fn = √5 . 2 − √5 2 Yen (NTUEE) Discrete Mathematics 2012

Characteristic Equations with Multiple Roots Theorem Let c1, c2 ∈ R with c2 =6 0. Suppose x 2 = c1x + c2 has only one root r . Then {an} is a solution of the recurrence relation an = c1an−1 + c2an−2 if and only if an = α1rn + α2nrn for n ∈ N and some constants α1, α2. Proof. (⇐) 2r = c1 and c1r + 2c2 = 0. Let an = α1rn + α2nrn. Then c1an−1 c2an−2 ∴ c1an−1 + c2an−2 = c1α1r + c1α2(n − 1)r = c2α1r + c2α2(n − 2)r = n−1 n−1 n−2 n−2 α1r (c1r + c2) + α2r (c1(n − 1)r + c2(n − 2)) n−2 n−2 α1rn−2r 2 + α2rn−2(c1nr − c1r + c2n − 2c2) α1rn + α2rn−2(c1nr + c2n) α1rn + α2rn−2n(c1r + c2) α1rn + α2nrn−2r 2 = Yen (NTUEE) Discrete Mathematics 2012 α rn + α nrn

Characteristic Equations with Multiple Roots Proof. (Cont’d) (⇒) It remains to show there are α1 and α2 satisfying the initial conditions. We have a0 = α1. Therefore a1 − a0r r α2 = . Yen (NTUEE) Discrete Mathematics 2012

An Example Example What is the solution of the recurrence relation an = 6an−1 − 9an−2 with a0 = 1 and a1 = 6? Solution: Since 3 is the multiple root of x 2 = 6x − 9, we have an = α13n + α2n3n. Moreover, α1 = a0 = 1 and α2 = (6 − 1 · 3)/3 = 1. We have an = 3n + n3n. Yen (NTUEE) Discrete Mathematics 2012

Characteristic Equations with Distinct Roots Theorem Let c1, c2, . . . , ck ∈ R. Suppose the characteristic equation rk − c1rk−1 − · · · − ck = 0 has k distinct roots r1, . . . , rk . Then {an} is a solution of the recurrence relation an = c1an−1 + c2an−2 + · · · + ck an−k if and only if a = α r + α r + · · · + α r n 1 2 2 k k n n n 1 for n ∈ N and constants α1, α2, . . . , αk . Yen (NTUEE) Discrete Mathematics 2012

Characteristic Equations with Multiple Roots Theorem Let c1, c2, . . . , ck ∈ R. Suppose the characteristic equation rk − c1rk−1 − · · · − ck = 0 has t distinct roots r1, . . . , rt with multiplicities m1, . . . , mt respectively such that mi ≥ 1 and m1 + m2 + · · · + mk = k. Then {an} is a solution of the recurrence relation an = c1an−1 + c2an−2 + · · · + ck an−k if and only if an = (α1,0 + α1,1n + · · · + α1,m1−1n )r1 + (α2,0 + α2,1n + · · · + α2,m2−1n )r2 + · · · m1−1 n m2−1 n + (αt,0 + αt,1n + · · · + αt,mt −1n )rt for n ∈ N and constants αi ,j . mt −1 n Yen (NTUEE) Discrete Mathematics 2012

Solving Nonhomogeneous Linear Recurrence Relations Definition A linear nonhomogeneous recurrence relation with constant coefficients is of the form an = c1an−1 + c2an−2 + · · · + ck an−k + F (n), where c1, c2, . . . , ck ∈ R and F : N → Z. The recurrence relation an = c1an−1 + c2an−2 + · · · + ck an−k is called the associated homogeneous recurrence relation. Yen (NTUEE) Discrete Mathematics 2012

Solving Linear Recurrence Relations Let {an } be a particular solution of an = c1an−1 + c2an−2 + · · · + ck an−k + F (n), Then every solution is of the form {an + an }, where {an } is a solution Theorem (p) (p) (h) (h) of an = c1an−1 + c2an−2 + · · · + ck an−k . (Cont’d on Next Page) Yen (NTUEE) Discrete Mathematics 2012

Solving Nonhomogeneous Linear Recurrence Relations Proof. Suppose {bn} be any solution of the nonhomogeneous recurrence relation such that bn = c1bn−1 + c2bn−2 + · · · + ck bn−k + F (n). Then Since {an } is a particular solution, we have (p) a(p) = c1a(p) (p) (p) n−1 + c2an−2 + · · · + ck an−k + F (n). n bn − an = c1(bn−1 − an−1) + c2(bn−2 − an−2) + · · · + ck (bn−k − an−k ). That is, {an − bn} is a solution of the associated homogeneous (p) (p) (p) (p) (p) recurrence relation. Thus, bn = a(p) + a(h). n n Yen (NTUEE) Discrete Mathematics 2012

An Example Example Find all solutions of an = 3an−1 + 2n with a1 = 3. Solution: Let us guess a(p) = cn + d . Then n cn + d = 3(c(n − 1) + d ) + 2n. We have cn + d = (3c + 2)n + (3d − 3c). Solve c = 3c + 2 and d = 3d − 3c. We obtain c = −1 and d = − . 3 2 Therefore a = −n − is a particular solution. (p) 3 n 2 By Theorem 9, we have a(h) = α3n as solutions to the associated n homogeneous recurrence relation. Therefore all solutions are of the form a = −n − + α · 3 where α is a constant. 3 n n 2 Finally, 3 = a = −1 − + 3α. α = . Therefore 3 11 1 2 6 3 11 n an = −n − 2 + 6 3 . Yen (NTUEE) Discrete Mathematics 2012

a(p) = 49 n 7n An Example Example 20 · 7 is a particular solution. Find solutions of an = 5an−1 − 6an−2 + 7 . (p) n Solution: Let us guess a = c · 7 . Then n n c · 7 = 5c · 7 − 6c · 7 + 7 . Hence 49c = 35c − 6c + 49, c = . n n−1 n−2 n 49 20 a(p) = 49 n n 20 · 7 is a particular solution. By Theorem 9, we have a(h) = α 2n + α 3n as solutions of the associated n 1 2 homogeneous recurrence relation. Therefore an = α12n + α23n + 49 20 7n are all solutions. Yen (NTUEE) Discrete Mathematics 2012

Solving Particular Solutions Theorem Suppose {an} satisfies an = c1an−1 + c2an−2 + · · · + ck an−k + F (n), where c1, c2, . . . , ck ∈ R and F (n) = (btnt + bt n + · · · + b n + b )s , t−1 n −1 1 0 where s, b0, b1, . . . , bt ∈ R. When s is not a root of the characteristic equation of the associated linear homogeneous recurrence relation, there is a particular solution of the form (ptnt + pt n + · · · + p n + p )s . t−1 n −1 1 0 When s is a root of the characteristic equation and its multiplicity is m, there is a particular solution of the form nm(ptnt + pt n + · · · + p n + p )s . t−1 n −1 1 0 Yen (NTUEE) Discrete Mathematics 2012

Solving Nonhomogeneous Linear Recurrence Relations Example Solve an = an−1 + 3n − 3n + 1. with a0 = 0. 2 Solution: Since an = an−1 + 3n − 3n + 1, F (n) = (3n − 3n + 1)1 . 2 2 n By Theorem 9, we have a = α · 1 = α as homogeneous solutions. (h) n n By Theorem 15, we let a(p) = n(p n2 + p n + p ) as a particular solution. n 2 1 0 Substitute a(p) in the recurrence relation, we have n n(p2n2 + p1n + p0) = (n − 1)(p2(n − 1)2 + p1(n − 1) + p0) + 3n2 − 3n + 1. Simplify both sides of the equation and compare their coefficients: p2 = p1 p0 0 −3p2 + p1 + 3 3p2 − 2p1 + p0 − 3 −p2 + p1 − p0 + 1 Yen (NTUEE) Discrete Mathematics 2012

Solving Nonhomogeneous Linear Recurrence Relations (Cont’d) Solve the linear system and get p0 = 0, p1 = 0, p2 = 1. Hence a(p) = n(1 · n2 + 0 · n + 0) = n3 is a particular solution. n Thus, an = n3 + α are all solutions. Since a0 = 0, we have α = 0. And we conclude an = n3. Notice that n3 − (n − 1)3 = 3n2 − 3n + 1. Hence Pn 3k − 3k + 1 = n . Alternatively, note that n an = X 3k2 − 3k + 1, k=1 2 3 k=1 you can solve it by computing the summation. Yen (NTUEE) Discrete Mathematics 2012

Solving Recurrence Relations by Domain Transformation Example T (n) = 2T (n/2) + nlog2n, n > 1 where n is a power of 2. Solutions: Replace n by 2k , we have T (2k ) = 2T (2k−1) + k2k Let tk = T (2k ). The above can be rewritten as tk = 2tk−1 + k2 Hence, k tk = c12k + c2k2k + c3k22k T (n) = c1n + c2nlogn + c3nlog 2n Yen (NTUEE) Discrete Mathematics 2012

Solving Recurrence Relations by Range Transformation Example an = 3a2 subject to a0 = 1 n−1 Solution: Let bn = log an. Then the above can be rewritten as bn = 2bn−1 + log 3 subject to b0 = 0. Hence bm = (2n − 1)log 3 an = 2(2 −1)log 3 = 32 −1 n n Yen (NTUEE) Discrete Mathematics 2012

Divide-and-Conquer Algorithms and Recurrence Relations Recall the merge sort algorithm. The algorithm divides the sequence in halves, solves each half recursively, and then merges sorted results to get the solution. This type of algorithms are called divide-and-conquer algorithms. In the complexity analysis of divide-and-conquer algorithms, we obtain the recurrence relation of the form n f (n) = af ( ) + g (n), b where f (n) is the number of steps needed in solving an instance of size n. We will prove a theorem to help us solve the recurrence relation. Yen (NTUEE) Discrete Mathematics 2012

Divide-and-Conquer Algorithms and Recurrence Relations Theorem Let f be a non-decreasing function satisfying n f (n) = af ( ) + c b whenever b|n, where a ≥ 1, b ∈ Z+, b > 1, and c ∈ R+. Then logb a f (n) = O(n ) if a > 1 if a = 1 O(lg n) Furthermore, when n = bk and a > 1, k ∈ Z+, f (n) = C1nlogb a + C2, where C1 = c/(a − 1) + f (1) and C2 = −c/(a − 1). Yen (NTUEE) Discrete Mathematics 2012

Divide-and-Conquer Algorithms and Recurrence Relations Proof: We first consider n = bk . Then bk f (n) = af ( ) + c b = af (bk−1) + c = a(af (bk−2) + c) + c = a2f (bk−2) + ac + c = · · · = ak f (1) + ak−1c + ak−2c + · · · + ac + c k−1 = ak f (1) + c X ai i =0 When a = 1, we have f (n) = f (1) + ck = f (1) + c logb n. Furthermore, if bk < n < bk+1, we have f (bk ) ≤ f (n) ≤ f (bk+1) and bk+1 < nb. Thus, f (n) ≤ f (bk+1) = f (1)+c(k +1) < f (1)+c(1+logb n) = f (1)+c +c logb n. Therefore f (n) = O(lg n) when a = 1. Yen (NTUEE) Discrete Mathematics 2012

Divide-and-Conquer Algorithms and Recurrence Relations (Cont’d) When a > 1 and n = bk , then nlogb a = bk logb a = ak . We have k f (n) = ak f (1) + c a − 1 a − 1 = ak f (1) + c c a − 1 − a − 1 = For bk < n < bk+1, we have nlogb aC1 + C2 f (n) ≤ f (bk+1) = C1ak+1 + C2 = aC1ak + C2 ≤ aC1nlogb a + C2 Hence, f (n) = O(nlogb a). Yen (NTUEE) Discrete Mathematics 2012

An Example Example Let f (n) = 5f (n/2) + 3. Estimate f (n). Solution: By Theorem 19, we have f (n) = O(nlog2 5) = O(nlg 5) since a = 5 and b = 2. For merge sort algorithm, we have T (n) = 2T (n/2) + Θ(n). Theorem 19 is not applicable. The following theorem will be proved in your algorithm class: Yen (NTUEE) Discrete Mathematics 2012

Divide-and-Conquer Algorithms and Recurrence Relations Theorem Let f be a non-decreasing function satisfying n f (n) = af ( ) + cnd b whenever n = bk , k ∈ Z+, b ∈ Z+ with a ≥ 1, b > 1, and c, d ∈ R with c > 0, d ≥ 0. Then  O(nd ) f (n) =  O(nd lg n)  O(nlogb a) if a < bd if a = bd if a > bd Therefore, the complexity of merge sort algorithm is O(n lg n). Yen (NTUEE) Discrete Mathematics 2012