1. Introduction to the Design and Analysis of Algorithms
ElSayed Badr
Benha University
September 20, 2016

2. Problem:

3. Problem:

4. Big-oh

5. Big-omega

6. Big-theta

7. Establishing order of growth using the definition
f(n) is in O(g(n)), denoted f(n)  O(g(n)), if order of growth of
f(n) ≤ order of growth of g(n) (within constant multiple), i.e.,
there exist a positive constant c and non-negative integer n0 such that
f(n) ≤ c g(n) for every n ≥ n0
Examples:
10n is in O(n2)
5n+20 is in O(n)
0-notation
The bound is asymptotically tight
But the bound is not. But we can write little o-notation
Example :
Examples:
10n is O(n2)
since 10n ≤ 10n2 for n ≥ 1 or 10n ≤ n2 for n ≥ 10
c n0
5n+20 is O(10n)
since 5n+20 ≤ 10 n for n ≥ 4

8. -notation Formal definition
A function t(n) is said to be in (g(n)), denoted t(n)  (g(n)), if t(n) is bounded below by some constant multiple of g(n) for all large n, i.e., if there exist some positive constant c and some nonnegative integer n0 such that t(n)  cg(n) for all n  n0
Examples:
10n2  (n2)
0.3n2 - 2n  (n2)
0.1n3  (n2)
Example :

9. -notation Formal definition
A function t(n) is said to be in (g(n)), denoted t(n)  (g(n)), if t(n) is bounded both above and below by some positive constant multiples of g(n) for all large n, i.e., if there exist some positive constant c1 and c2 and some nonnegative integer n0 such that
c2 g(n)  t(n)  c1 g(n) for all n  n0
Examples:
10n2  (n2)
0.3n2 - 2n  (n2)
(1/2)n(n+1)  (n2)

10. >=
(g(n)), functions that grow at least as fast as g(n) =
(g(n)), functions that grow at the same rate as g(n)
g(n)
<=
O(g(n)), functions that grow no faster than g(n)

11. Logarithm Review

12. Logarithm Review

15. Useful summation formulas and rules
l  I n1 = 1+1+…+1 = n - l + 1
In particular, lI n1 = n = n  (n)
1in i = 1+2+…+n = n(n+1)/2  n2/2  (n2)
1in i2 = …+n2 = n(n+1)(2n+1)/6  n3/3  (n3)
0in ai = 1 + a +…+ an = (an+1 - 1)/(a - 1) for any a  1
In particular, 0in 2i = …+ 2n = 2n  (2n )
(ai ± bi ) = ai ± bi cai = cai liuai = limai + m+1iuai

16. Example 1: Maximum element

17. Example 2: Element uniqueness problem

18. Example 3: Matrix multiplication

19. Example 4: Gaussian elimination
Algorithm GaussianElimination(A[0..n-1,0..n])
//Implements Gaussian elimination on an n- by-(n + 1) matrix A
for i  0 to n - 2 do for j  i + 1 to n - 1 do for k  i to n do
A[j,k]  A[j,k] - A[i,k]  A[j,i] / A[i,i]

20. Plan for Analysis of Recursive Algorithms
Decide on a parameter indicating an input’s size.
Identify the algorithm’s basic operation.
Check whether the number of times the basic op. is executed may vary on different inputs of the same size. (If it may, the worst, average, and best cases must be investigated separately.)
Set up a recurrence relation with an appropriate initial condition expressing the number of times the basic op. is executed.
Solve the recurrence (or, at the very least, establish its solution’s order of growth) by backward substitutions or another method.

21. Example 1: Recursive evaluation of n!
Definition: n ! = 1  2  … (n-1)  n for n ≥ 1 and 0! = 1
Recursive definition of n!: F(n) = F(n-1)  n for n ≥ 1 and F(0) = 1
Size:
Basic operation:
Recurrence relation:
Note the difference between the two recurrences. Students often confuse these!
F(n) = F(n-1) n
F(0) = 1
for the values of n!
M(n) =M(n-1) + 1
M(0) = 0
for the number of multiplications made by this algorithm n
multiplication
M(n) = M(n-1) + 1
M(0) = 0

22. Solving the recurrence for M(n)
M(n) = M(n-1) + 1, M(0) = 0
M(n) = M(n-1) + 1
= (M(n-2) + 1) = M(n-2) + 2
= (M(n-3) + 1) = M(n-3) + 3 …
= M(n-i) + i
= M(0) + n
= n
The method is called backward substitution.

23. Smoothness Rule
Let f(n) be a nonnegative function defined on the set of natural numbers. f(n) is call smooth if it is eventually nondecreasing and
f(2n) ∈ Θ (f(n))
Functions that do not grow too fast, including logn, n, nlogn, and n where >=0 are smooth.
Smoothness rule
Let T(n) be an eventually nondecreasing function and f(n) be a smooth function. If
T(n) ∈ Θ (f(n)) for values of n that are powers of b,
where b>=2, then
T(n) ∈ Θ (f(n)) for any n.

24. Fibonacci numbers The Fibonacci numbers:
0, 1, 1, 2, 3, 5, 8, 13, 21, …
The Fibonacci recurrence:
F(n) = F(n-1) + F(n-2)
F(0) = 0
F(1) = 1
In fact, the Fibonacci numbers grow almost as fast as the powers of 2:
for example, F30 is over a million, and F100 is already 21 digits long!
In general, Fn ≈ 20:694n
But what is the precise value of F100, or of F200 ?
Fibonacci himself would surely have wanted to know such things.

25. Fibonacci numbers function fib1(n) if n = 0: return 0
return fib1(n - 1) + fib1(n - 2)
we immediately see that
At this time, the fastest computer in the world is the NEC Earth Simulator, which clocks 40 trillion steps per second. Even on this machine, fib1(200) would take at least 292 seconds.

27. Fibonacci numbers function fib2(n) if n = 0 return 0
create an array f[0 …n]
f[0] = 0, f[1] = 1
for i = 2 …n:
f[i] = f[i - 1] + f[i - 2]
return f[n]
General 2nd order linear homogeneous recurrence with
constant coefficients: aX(n) + bX(n-1) + cX(n-2) = 0

28. Solving aX(n) + bX(n-1) + cX(n-2) = 0
Set up the characteristic equation (quadratic)
ar2 + br + c = 0
Solve to obtain roots r1 and r2
General solution to the recurrence
if r1 and r2 are two distinct real roots: X(n) = αr1n + βr2n
if r1 = r2 = r are two equal real roots: X(n) = αrn + βnr n
Particular solution can be found by using initial conditions

29. Application to the Fibonacci numbers
F(n) = F(n-1) + F(n-2) or F(n) - F(n-1) - F(n-2) = 0
Characteristic equation:
Roots of the characteristic equation:
General solution to the recurrence:
Particular solution for F(0) =0, F(1)=1:

30. Computing Fibonacci numbers
Definition-based recursive algorithm
Nonrecursive definition-based algorithm
Explicit formula algorithm
Logarithmic algorithm based on formula:
F(n-1) F(n)
F(n) F(n+1)
0 1
1 1 = n
for n ≥ 1, assuming an efficient way of computing matrix
powers.

31. Important Recurrence Types
Decrease-by-one recurrences
A decrease-by-one algorithm solves a problem by exploiting a relationship between a given instance of size n and a smaller size n – 1.
Example: n!
The recurrence equation for investigating the time efficiency of such algorithms typically has the form
T(n) = T(n-1) + f(n)
Decrease-by-a-constant-factor recurrences
A decrease-by-a-constant-factor algorithm solves a problem by dividing its given instance of size n into several smaller instances of size n/b, solving each of them recursively, and then, if necessary, combining the solutions to the smaller instances into a solution to the given instance.
Example: binary search.
The recurrence equation for investigating the time efficiency of such algorithms typically has the form
T(n) = aT(n/b) + f (n)

32. Decrease-by-one Recurrences
One (constant) operation reduces problem size by one.
T(n) = T(n-1) + c T(1) = d
Solution:
A pass through input reduces problem size by one.
T(n) = T(n-1) + c n T(1) = d
T(n) = (n-1)c + d linear
T(n) = [n(n+1)/2 – 1] c + d quadratic

33. Master Theorem