1. Basic Concepts in Algorithmic Analysis
Chapter 1
Basic Concepts in Algorithmic Analysis

2. Questions Given a problem
How do we find an efficient algorithm for its solution?
How can we compare the algorithms that solve the same problem?
How should we judge the goodness of an algorithm?

3. Definition of algorithm
An algorithm is a procedure that consists of a finite set of instructions which, given an input from some set of possible inputs, enable us to obtain an output if such an output exists or else obtain nothing at all if there is no output for that particular input through a systematic execution of the instructions

4. Requirements for algorithm
An algorithm should halt on every input, which implies that each instructions requires a finite amount of time, and each input has a finite length.
The output of a legal input should be unique.

5. Linear search Algorithm 1.1 LINEARSEARCH
Input: An array A[1..n] of n elements and an element x
Output: j if x=A[j], 1jn, and 0 otherwise
1. j1
2. while (j<n) and (xA[j])
jj+1
4. end while
5. if x=A[j] then return j else return 0

6. Special situation What should we do if the elements in A are sorted?
Example:
A[1..14]=1, 4, 5, 7, 8, 9, 10, 12, 15, 22, 23, 27, 32, 35

7. Binary search Algorithm 1.2 BINARYSEARCH
Input: An array A[1..n] of n elements sorted in nondecreasing order and an element x
Output: j if x=A[j], 1jn, and 0 otherwise
1. low1, highn, j0
2. while (lowhigh) and (j=0)
mid(low+high)/2
if x=A[mid] then jmid
else if x<A[mid] then highmid-1
else lowmid+1
7. end while
8. return j

8. Decision tree

9. Theorem 1.1
The number of comparisons performed by Algorithm BINARYSEARCH on a sorted array of size n is at most logn+1

10. Selection sort Algorithm 1.4 SELECTIONSORT
Input: An array A[1..n] of n elements
Output: A[1..n] sorted in nondecreasing order
1. for i1 to n-1
ki
for ji+1 to n {Find the ith smallest element}
if A[j]<A[k] then kj
end for
if kj then interchange A[i] and A[k]
7. end for

11. Selection sort
Observation: The number of element comparisons performed by Algorithm SELECTIONSORT is n(n-1)/2. The number of element assignments is between 0 and 3(n-1)

12. Insertion sort Algorithm 1.5 INSERTIONSORT
Input: An array A[1..n] of n elements
Output: A[1..n] sorted in nondecreasing order
1. for i2 to n
xA[i]
ji-1
while (j>0) and (A[j]>x)
A[j+1]A[j]
jj-1
end while
A[j+1]x
9. endfor

13. Insertion sort
Observation: The number of element comparison performed by Algorithm INSERTIONSORT is between n-1 and n(n-1)/2. The number of element assignments is equal to the number of element comparison plus n-1

14. Bottom-up merge sorting

15. Merge
Algorithm 1.3 MERGE
Input: An array A[1..m] of elements and three indices p, q and r, with 1pq<rm, such that both the subarrays A[p..q] and A[q+1..r] are sorted individually in nondecreasing order
Output: A[p..r] contains the result of merging the two subarrays A[p..q] and A[q+1..r]. A[p..r] is sorted in nondecreasing order

16. Merge 1. comment: B[p..r] is an auxiliary array 2. sp, tq+1, kp
3. while sq and tr
4. if A[s]A[t] then
B[k]A[s]
ss+1
else
B[k]A[t]
tt+1
10. end if
11. kk+1
12. endwhile
13. if s=q+1 then B[k..r]A[t..r]
14. else B[k..r]A[s..q]
15. endif
16. A[p..r]B[p..r]

17. Observation
The number of element comparisons performed by Algorithm MERGE to merge two nonempty arrays of size n1 and n2, respectively, where n1n2, into one sorted array of size n=n1+n2 is between n1 and n-1. In particular, if the two array sizes are n/2 and n/2, the number of comparisons needed is between n/2 and n-1
The number of element assignments performed by Algorithm MERGE to merge two arrays into one sorted array of size n is exactly 2n

18. Bottom-up Merge Sorting
Algorithm 1.6 BOTTOMUPSORT
Input: An array A[1..n] of n elements
Output: A[1..n] sorted in nondecreasing order
1. t1
2. while t<n
st, t2s, i0
while i+tn
MERGE(A, i+1, i+s, i+t)
ii+t
end while
if i+s<n then MERGE(A, i+1, i+s, n)
9. endwhile

19. Observation
The total number of element comparisons performed by Algorithm BOTTOMUPSORT to sort an array of n elements, where n is a power of 2, is between (nlogn)/2 and nlogn-n+1. The total number number of element assignments done by the algorithm is exactly 2nlogn

20. Time complexity Running time of a program is determined by:
1) input size
2) quality of the code
3) quality of the computer system
4) time complexity of the algorithm
We are mostly concerned with the behavior of the algorithm under investigation on large input instances.
So we may talk about the rate of growth or the order of growth of the running time

21. Running times vs Input sizes

22. Running times vs Input sizes
complexity
function
With present
computer
With computer
100 times faster
With computer
1000 times faster n N1
100N1
1000N1 N2
31.6N2 n2
10N2 N3 n3
4.64N3
10N3 n5 N4
2.5N4
3.98N4 2n N5 N N 3n N6 N N

23. Elementary operation
Definition: We denote by an “elementary operation” any computational step whose cost is always upperbounded by a constant amount of time regardless of the input data or the algorithm used.
Example:
Arithmetic operations: addition, subtraction, multiplication and division
Comparisons and logical operations
Assignments, including assignments of pointers when, say, traversing a list or a tree

24. O-notation
Definition: Let f(n) and g(n) be two functions from the set of natural numbers to the set of nonnegative real numbers. f(n) is said to be O(g(n)) if there exists a natural number n0 and a constant c>0 such that
nn0, f(n)cg(n)
Consequently, if limf(n)/g(n) exists, then
limf(n)/g(n) implies f(n)=O(g(n))
Informally, this definition says that f grows no faster than some constant times g.

25. -notation
Definition: Let f(n) and g(n) be two functions from the set of natural numbers to the set of nonnegative real numbers. f(n) is said to be (g(n)) if there exists a natural number n0 and a constant c>0 such that
nn0, f(n)cg(n)
Consequently, if limf(n)/g(n) exists, then
limf(n)/g(n)0 implies f(n)=(g(n))
Informally, this definition says that f grows at least as fast as some constant times g.
f(n)=(g(n)) if and only if g(n)=O(f(n))

26. -notation
Definition: Let f(n) and g(n) be two functions from the set of natural numbers to the set of nonnegative real numbers. f(n) is said to be (g(n)) if there exists a natural number n0 and two positive constants c1 and c2 such that
nn0, c1g(n)f(n)c2g(n)
Consequently, if limf(n)/g(n) exists, then
limf(n)/g(n)=c implies f(n)=(g(n))
where c is a constant strictly greater than 0
f(n)=(g(n)) if and only if f(n)=(g(n)) and f(n)=O(g(n))

27. o-notation
Definition: Let f(n) and g(n) be two functions from the set of natural numbers to the set of nonnegative real numbers. f(n) is said to be o(g(n)) if for every constant c>0 there exists a positive integer n0 such that f(n)<cg(n) for all nn0
Consequently, if limf(n)/g(n) exists, then
limf(n)/g(n)=0 implies f(n)=o(g(n))
Informally, this definition says that f(n) becomes insignificant relative to g(n) as n approaches infinity.
f(n)=o(g(n)) if and only if f(n)=O(g(n)) but g(n)O(f(n))

28. Space complexity
The work space cannot exceed the running time of an algorithm, as writing into each memory cell requires at least a constant amount of time.
S(n)=O(T(n))
In many problems there is a time-space tradeoff: The more space we allocate for the algorithm the faster it runs, and vice versa. But this is within limits: increasing the amount of space does not always result in a noticeable speedup in the algorithm running time. However it is always the case that decreasing the amount of work space required by an algorithm results in a degradation in the algorithm’s speed.

29. Optimal algorithm
In general, if we can prove that any algorithm to solve problem II must be (f(n)), then we call any algorithm to solve problem II in time O(f(n)) an optimal algorithm for problem II

30. How to estimate the running time of an algorithm
1. Counting the number of iterations
2. Counting the frequency of basic operations
Definition: An elementary operation in an algorithm is called a basic operation if it is of highest frequency within a constant factor among all other elementary operations
3. Using recurrence relations

31. Basic operation
Algorithm 1.5 INSERTIONSORT Input: An array A[1..n] of n elements Output: A[1..n] sorted in nondecreasing order 1. for i2 to n 2. xA[i] 3. ji-1 4. while (j>0) and (A[j]>x) 5. A[j+1]A[j] 6. jj-1 7. end while 8. A[j+1]x 9. endfor
Algorithm 1.12 MODINSERTIONSORT
Input: An array A[1..n] of n elements
Output: A[1..n] sorted in nondecreasing order
1. for i2 to n
xA[i]
kMODBINARYSEARCH(A[1..i-1],x)
for j  i-1 downto k
A[j+1]A[j]
endfor
A[k]x
8. endfor

32. Worst case and average case analysis
For many problems, the performance of the algorithm is not only a function of n, but also a function of the form of the input elements
worst case analysis, average case analysis, best case analysis

33. Amortized analysis
In amortized analysis, we average out the time taken by the operation throughout the execution of the algorithm, and refer to this average as the amortized running time of that operation. Amortized analysis guarantees the average cost of the operation, and thus the algorithm, in the worst case. This is to be contrasted with the average time analysis in which the average is taken over all instances of the same size. Moreover, unlike the average case analysis, no assumptions about the probability distribution of the input are needed.

34. Input size
Sorting and searching: number of entries in the array or list
Graph: the number of vertices or edges in the graph, or both
Computational geometry: the number of points, vertices, edges, line segments, polygons, etc
Matrix operation: the dimensions of the input matrices
Number theory and cryptography: the number of bits
These “heterogeneous” measures have brought about some inconsistencies when comparing the amount of time or space required by two algorithms

35. Input size Algorithm 1.13 FIRST
Input: A positive integer n and an array A[1..n] with A[j]=j, 1jn
Output: ∑A[j]
1. sum←0
2. for j←1 to n
sum←sum+A[j]
4. end for
5. return sum
Algorithm 1.14 SECOND
Input: A positive integer n
Output: ∑j=1,…,nj
sum←sum+j
Both algorithms run in time (n). But…