Topic 3: Prob. Analysis Randomized Alg. CSE 2331/5331 Topic 3: Prob. Analysis Randomized Alg. CSE 2331/5331

Expected Complexity Probabilistic method: Randomized algorithm: Given a distribution for all possible inputs Derive expected time based on distribution Randomized algorithm: Add randomness in the algorithm Analyze the expected behavior of the algorithm CSE 2331/5331

First Example What is worst case time complexity? What is expected / average time complexity? CSE 2331/5331

Expected Running Time Expected / average running time 𝐸𝑇 𝑛 = 𝐼 Pr 𝐼 𝑡 𝐼 Pr 𝐼 = probability of event I 𝑡 𝐼 = running time given event I To analyze, need to assume a probabilistic distribution for all inputs CSE 2331/5331

SeqSearch Alg Expected running time = Pr K ∉A t K∉A + 𝑖=1 𝑛 Pr 𝐴 𝑖 =𝐾 𝑡 𝐴 𝑖 =𝐾 If we assume Pr K ∉A = 0 All permutations are equally likely implies Pr 𝐴 𝑖 =𝐾 = 1 𝑛 Then expected running time = 𝑖 1 𝑛 ∗𝑐𝑖 =Θ 𝑛 CSE 2331/5331

Remarks For probabilistic analysis Randomized algorithm An input probabilistic distribution input model has be assumed! For a fixed input, the running time is fixed. The average / expected time complexity is for if we consider running it for a range of inputs, what the average behavior is. Randomized algorithm No assumption in input distribution! Randomness is added in the algorithm For a fixed input, the running time is NOT fixed. The expected time is what we can expect when we run the algorithm on ANY SINGLE input. CSE 2331/5331

Randomized Algorithms CSE 2331/5331

Expectation X is a random variable The expectation of X is 𝐸 𝑋 = 𝐼 Pr 𝑋=𝐼 𝐼 E.g, coin flip Linearity of expectation: 𝐸 𝑋 1 + 𝑋 2 =𝐸 𝑋 1 +𝐸 𝑋 2 Conditional expectation: 𝐸 𝑋 =𝐸 𝑋 𝑌) Pr 𝑌 +𝐸(𝑋 𝑁𝑜𝑡 𝑌 1 − Pr 𝑌 CSE 2331/5331

Linearity of Expectation 𝐸 𝑇 1 𝑛 =𝑛 𝐸 𝑇 2 𝑛 + cn 𝐸 𝑇 2 𝑛 = expected running time for func2 What is 𝐸 𝑇 1 𝑛 ? CSE 2331/5331

Conditional Expectation 𝐸 𝑇 1 𝑛 = Pr ℎ𝑒𝑎𝑑 𝐸 𝑇 2 𝑛 +(1 − Pr ℎ𝑒𝑎𝑑 ) 𝐸 𝑇 3 𝑛 + 𝑐 𝐸 𝑇 2 𝑛 = expected running time of func2 𝐸 𝑇 3 𝑛 = expected running time of func3 What is the expected running time of func1? CSE 2331/5331

A Randomized Example Worst case complexity? Expected case? Worst case \Theta(n^2) Expected case: T(n) = \sum(1/n) ck^2 = \Theta(n^2) too CSE 2331/5331

Running Time Analysis Worst Case: Expected running time: 𝑇 𝑛 =Θ( 𝑛 2 ) 𝐸𝑇 𝑛 = 𝑖=1 𝑛 Pr 𝑘=𝑖 ⋅(𝑐 𝑖 2 ) = 𝑖=1 𝑛 1 𝑛 ⋅(𝑐 𝑖 2 ) = 𝑐 𝑛 𝑖=1 𝑛 𝑖 2 =Θ 𝑛 2 CSE 2331/5331

Another Example CSE 2331/5331

Running time analysis Worst case: 𝑇 𝑛 = 𝑖=1 𝑛 𝑐 𝑖 3 2 =Θ 𝑛 2.5 . Expected running time: Lines 4—8 takes time: 𝑐 𝑘 3 2 Let 𝑋 𝑖 denote the running time for the i-th iteration of outer for-loop (i.e, from line 3 – 8) 𝐸 𝑋 𝑖 = 𝑞=1 𝑖 𝑃𝑟𝑜𝑏 𝑘=𝑞 𝑖𝑛 𝑡ℎ𝑒 𝑖𝑡ℎ 𝑖𝑡𝑒𝑟𝑎𝑡𝑖𝑜𝑛 𝑇𝑖𝑚𝑒 𝑙𝑖𝑛𝑒𝑠 3−8 𝑘=𝑞 = 𝑞=1 𝑖 1 𝑖 ⋅𝑐 𝑞 3 2 = 𝑐 𝑖 𝑞=1 𝑖 𝑞 3 2 =Θ( 𝑖 1.5 ) Hence we can assume that 𝐸 𝑋 𝑖 = 𝑐 ′ 𝑖 1.5 𝐸𝑇 𝑛 =𝐸 𝑖=1 𝑛 𝑋 𝑖 = 𝑖=1 𝑛 𝐸 𝑋 𝑖 = 𝑖=1 𝑛 𝑐 ′ 𝑖 1.5 =Θ 𝑛 2.5 CSE 2331/5331

CSE 2331/5331

Time Analysis Worst case: Expected running time: 𝑇 𝑛 =Θ 𝑛 2 𝑇 𝑛 =Θ 𝑛 2 Expected running time: 𝐸𝑇 𝑛 = Pr 𝑘< 𝑛 ⋅𝐸𝑥𝑝𝑇𝑖𝑚𝑒(𝑘< 𝑛 )+ Pr 𝑘≥ 𝑛 ⋅𝐸𝑥𝑝𝑇𝑖𝑚𝑒 𝑘≥ 𝑛 = 𝑛 𝑛 ⋅ 𝑐 𝑛 2 + 1− 𝑛 𝑛 ⋅( 𝑐 ′ 𝑛 lg 𝑛) ⇒𝐸𝑇(𝑛)=Θ( 𝑛 1.5 ) CSE 2331/5331

A more complicated variation. CSE 2331/5331

Analysis Worst case: Expected case – analysis (1) 𝑇 𝑛 =Θ(𝑛 lg 𝑛 ) 𝐸𝑇 𝑛 = Pr 𝑘< 𝑛 ⋅𝐸𝑇 𝑙𝑖𝑛𝑒𝑠 4−7 𝑘< 𝑛 + Pr 𝑘≥ 𝑛 ⋅𝐸𝑇 𝑙𝑖𝑛𝑒𝑠 11−17∣ 𝑘≥ 𝑛 = 𝑛 𝑛 𝑖=1 𝑛 Pr 𝑘=𝑖∣𝑘< 𝑛 ⋅𝐸𝑇 𝑙𝑖𝑛𝑒𝑠 4−7∣𝑘=𝑖 + = 1 𝑛 𝑖=1 𝑛 1 𝑛 ⋅ 𝑐 𝑖 2 + 1− 𝑛 𝑛 ⋅( 𝑐 ′ 𝑛 lg 𝑛) =Θ( 𝑛 ) +Θ 𝑛 lg 𝑛 =Θ(𝑛 lg 𝑛) CSE 2331/5331

Analysis Expected case – analysis (2) 𝐸𝑇 𝑛 = 𝑖=1 𝑛 Pr 𝑘=𝑖 ⋅𝐸𝑇 𝑙𝑖𝑛𝑒𝑠 4−7∣𝑘=𝑖 + Pr 𝑘≥ 𝑛 ⋅𝐸𝑇 𝑙𝑖𝑛𝑒𝑠 11−17∣𝑘≥ 𝑛 = 𝑖=1 𝑛 1 𝑛 ⋅ 𝑐 𝑖 2 + 1− 𝑛 𝑛 ⋅( 𝑐 ′ 𝑛 lg 𝑛) =Θ( 𝑛 ) +Θ 𝑛 lg 𝑛 =Θ(𝑛 lg 𝑛) CSE 2331/5331

An Example with Recursion CSE 2331/5331

Expected running time 𝐸𝑇 𝑛 Worst case: T n =T n−1 +cn= Θ 𝑛 2 Expected running time 𝐸𝑇 𝑛 𝐸𝑇 𝑛 =𝑐𝑛+ 1 2 𝐸𝑇 𝑛−1 Solving this we have 𝐸𝑇 𝑛 = 1 2 ET n−1 +cn≤ 1 2 𝐸𝑇 𝑛 +𝑐n ⇒ 1 2 𝐸𝑇 𝑛 ≤𝑐𝑛⇒𝐸𝑇 𝑛 ≤2𝑐𝑛⇒𝐸𝑇 𝑛 =𝑂 𝑛 ⇒𝐸𝑇 𝑛 = Θ 𝑛 CSE 2331/5331

Another Example with Recursion CSE 2331/5331

Analysis Worst case: Expected running time: 𝑇 𝑛 =𝑐+𝑇 𝑛−1 +𝑇 𝑛−2 𝑇 𝑛 =Ω 2 𝑛/2 ! Expected running time: 𝐸𝑇 𝑛 =𝑐+ 1 4 𝐸𝑇 𝑛−1 +𝐸𝑇 𝑛−2 ⇒𝐸𝑇 𝑛 ≤𝑐+ 1 2 ET n ⇒ 1 2 ET n ≤𝑐 ⇒𝐸𝑇 𝑛 ≤2𝑐⇒𝐸𝑇 𝑛 =O(1) 𝐸𝑇 𝑛 =Θ 1 ! CSE 2331/5331

Another Example CSE 2331/5331

Analysis Worst case: Expected running time: 𝑇 𝑛 =2𝑇 𝑛 2 +𝑐𝑛=Θ 𝑛 lg 𝑛 𝐸𝑇 𝑛 = 1 4 𝐸𝑇 𝑛 2 +𝐸𝑇 𝑛 2 +𝑐𝑛= 1 2 𝐸𝑇 𝑛 2 +𝑐𝑛 ⇒𝐸𝑇 𝑛 ≤ 1 2 𝐸𝑇 𝑛 + 𝑐𝑛⇒ 1 2 𝐸𝑇 𝑛 ≤𝑐𝑛 ⇒𝐸𝑇 𝑛 =𝑂 𝑛 ⇒𝐸𝑇 𝑛 =Θ 𝑛 CSE 2331/5331

Example 5 CSE 2331/5331

𝐸𝑇(𝑛) = expected running time of 𝐹𝑢𝑛𝑐5 on 𝑛 values CSE 2331/5331

Bounding 𝐸𝑇(𝑛) 𝑃𝑟 𝑘< 𝑛 4 = 1 4 ; 𝑃𝑟 𝑛 4 ≤𝑘≤ 3𝑛 4 = 1 2 ; 𝑃𝑟 𝑘< 𝑛 4 = 1 4 ; 𝑃𝑟 𝑛 4 ≤𝑘≤ 3𝑛 4 = 1 2 ; 𝑃𝑟 𝑘> 3𝑛 4 = 1 4 ; CSE 2331/5331

Upper Bound CSE 2331/5331

Upper Bound cont. 𝐸𝑇 𝑛 ≤𝑐𝑛+ 1 2 𝐸𝑇 𝑛 + 1 2 (𝐸𝑇 𝑛 4 +𝐸𝑇 3𝑛 4 ) 𝐸𝑇 𝑛 ≤𝑐𝑛+ 1 2 𝐸𝑇 𝑛 + 1 2 (𝐸𝑇 𝑛 4 +𝐸𝑇 3𝑛 4 ) CSE 2331/5331

Lower Bound In the best case, 𝑘= 𝑛 2 Hence Putting upper and lower bound together, we have: 𝐸𝑇 𝑛 =Θ 𝑛 𝑙𝑔 𝑛 CSE 2331/5331

Example 6 CSE 2331/5331

Recurrence for Func6 𝐸𝑇(𝑛) = expected running time of 𝐹𝑢𝑛𝑐6 on 𝑛 values CSE 2331/5331

Bounding ET(n) CSE 2331/5331

Upper Bound CSE 2331/5331

Lower Bound Whatever 𝑘 is, 𝐸𝑇(𝑘) is at least 𝑐𝑛. Hence 𝐸𝑇(𝑛) =Ω(𝑛) Putting upper and lower bounds together, we have 𝐸𝑇(𝑛) =Θ(𝑛) CSE 2331/5331