1. Lecture 2 MAS 714 Hartmut Klauck
Theory of Computing
Lecture 2
MAS 714
Hartmut Klauck

2. O, , £
Let f,g be two monotone increasing functions that sends N to R+
f=O(g) if n0,c 8 n>n0: f(n)<c g(n)
Example: f(n)=n, g(n)=1000n+100) g(n)=O(f(n))
Set c=1001 and n0=100
Example: f(n)=n log n, g(n)=n2

3. O, , £ Let f,g be two monotone increasing functions that send N to R+
f = (g) iff g=O(f)
Definition by Knuth
f = £(g) iff [ f=O(g) and g=O(f) ]
o, !: asymptotically smaller/larger
f=o(g) iff lim f(n)/g(n)=0
E.g., n=o(n2)
But 2n n=£(n2)

4. Sorting Computers spend a lot of time sorting!
Assume we have a list of numbers x1,…,xn from a universe U
For simplicity assume the xi are distinct
The goal is to compute a permutation ¼ such that x ¼(1)< x¼(2) <  < x¼(n)
Think of a deck of cards

5. InsertionSort An intuitive sorting algorithm
The input is provided in A[1…n]
Code: for i = 2 to n, for (k = i; k > 1 and A[k] < A[k-1]; k--) swap A[k,k-1] → invariant: A[1..i] is sorted end
Clear: O(n2) comparisons and O(n2) swaps
Algorithm works in place, i.e., uses linear space

6. Correctness By induction
Base case: n=2: We have one conditional swap operation, hence the output is sorted
Induction Hypothesis: After iteration i the elements A[1]…A[i] are sorted
Induction Step: Consider Step i+1. A[1]…A[i] are sorted already. Inner loop starts from A[i+1] and moves it to the correct position. After this A[1]…A[i+1] are sorted.

7. Best Case In the worst case Insertion Sort takes time (n2)
If the input sequence is already sorted the algorithms takes time O(n)
The same is true if the input is almost sorted
important in practice
Algorithm is simple and fast for small n

8. Worst Case On some inputs InsertionSort takes (n2) steps
Proof: consider a sequence that is decreasing, e.g., n,n-1,n-2,…,2,1
Each element is moved from position i to position 1
Hence the running time is at least i=1,…,n i = (n2)

9. Can we do better?
Attempt 1: Searching for the position to insert the next element is inefficient, employ binary search
Ordered search:
Given an array A with n numbers in a sorted sequence, and a number x, find the smallest i such that A[i]¸ x
Use A[n+1]=1

10. Linear Search Simplest way to search
Run through A (from 1 to n) and compare A[i] with x until A[i]¸ x is found, output i
Time: £(n)
Can also be used to search unsorted Arrays

11. Binary Search
If the array is sorted already, we can find an item much faster!
Assume we search for a x among A[1]<...<A[n]
Algorithm (to be called with l=1 and r=n):
BinSearch(x,A,l,r]
If r-l=0 test if A[l]=x, end
Compare A[(r-l)/2+l] with x
If A[(r-l)/2+l]=x output (r-l)/2+l, end
If A[(r-l)/2+l]> x BinSearch(x,A,l,(r-l)/2+l)
If A[(r-l)/2+l]<x Bin Search(x,A,(r-l)/2+l,r)

12. Time of Binary Search
Define T(n) as the time/number of comparisons needed on Arrays of length n
T(2)=1
T(n)=T(n/2)+1
Solution: T(n)=log(n)
All logs have basis 2

13. Recursion
We just described an algorithm via recursion: a procedure that calls itself
This is often convenient but we must make sure that the recursion eventually terminates
Have a base case (here r=l)
Reduce some parameter in each call (here r-l)

14. Binary Insertion Sort
Using binary search in InsertionSort reduces the number of comparisons to O(n log n)
The outer loop is executed n times, each inner loop now uses log n comparisons
Unfortunately the number of swaps does not decrease:
To insert an element we need to shift the remaining array to the right!

15. Quicksort Quicksort follows the „Divide and Conquer“ paradigm
The algorithm is best described recursively
Idea:
Split the sequence into two
All elements in one sequence are smaller than in the other
Sort each sequence
Put them back together

16. Quicksort Quicksort(A,l,r) If l=r return A
Choose a pivot position j between l and r
u=1,v=1, initialize arrays B,C
for (i=l…r): If A[i]<A[j] then B[u]=A[i], u If A[i]>A[j] then C[v]=A[i], v++
Run Quicksort(B,1,u) and Quicksort(C,1,v) and return their output (concatenated), with A[j] in the middle

17. How fast is it?
The quality of the algorithm depends on how we split up the sequence
Intuition:
Even split will be best
Questions:
How are the asymptotics?
Are approximately even splits good enough?

18. Worst Case Time
We look at the case when we really just split into the pivot and the rest (maximally uneven)
Let T(n) denote the number of comparisons for n elements
T(2)=1
T(n)<T(n-1)+n-1
Solving the recurrence gives T(n)=O(n2)

19. Best Case Time Every pivot splits the sequence in half T(2)=1
T(n)=2T(n/2)+n-1
Questions:
How to solve this?
What if the split is 3/4 vs. 1/4 ?