1. Lecture 3 Nearest Neighbor Algorithms Shang-Hua Teng

2. What is Algorithm? A computable set of steps to achieve a desired result from a given input Example: –Input: An array A of n numbers –Desired result Pseudo-code of Algorithm SUM

3. Complexity: Input Size n Number of steps: n-1 additions

4. Example 2: Integer Multiplication c = a b When do we need to multiply two very large numbers? –In Cryptography and Network Security message as numbers encryption and decryption need to multiply numbers

5. How to multiply 2 n-bit numbers ************ ************************

6. Asymptotic Notation of Complexity as input size grows, how fast does the running time grow. –T 1 (n) = 100 n –T 2 (n) = n 2 Which algorithm is better? When n < 100 is small then T 2 is smaller As n becomes larger, T 2 grows much faster To solve large-scale problem, algorithm1 is preferred.

7. Asymptotic Notation (Removing the constant factor) The  Notation  (g(n)) = { f(n): there exist positive c 1 and c 2 and n 0 such that for all n > n 0 } For example T(n) = 4nlog n + n =  (nlog n) For example n – 1 =  (n)

8. Asymptotic Notation (Removing the constant factor) The  Big  Notation O(g(n)) = { f(n): there exist positive c and n 0 such that for all n > n 0 } For example T(n) = 4nlog n + n =  (nlog n) But also T(n) = 4nlog n + n =  (n 2 )

9. Nearest Neighbor Problem: General Formulation

10. Nearest Neighbor Problem

11. Applications Points could be web-page, closest neighbor is the most similar web-page Points could be people, closest neighbor could be the best friend Points could be biological spices, the closest neighbor could be the closest spice …

12. An O(dn 2 ) time Algorithm Why O(dn 2 ) time?

13. Can We do better? Yes, Handout #4, by Jon Louis Bentley

14. One-Dimensional Geometry If we can order points from small to large, then we just need to look at the left neighbor and right neighbor of each point to find its nearest neighbor

15. Reduce to Sorting Input: Array A[1...n], of elements in an arbitrary order; array size n Output: Array A[1...n] of the same elements, but in the non-decreasing order

16. Divide and Conquer Divide the problem into a number of sub-problems (similar to the original problem but smaller); Conquer the sub-problems by solving them recursively (if a sub-problem is small enough, just solve it in a straightforward manner. Combine the solutions to the sub-problems into the solution for the original problem

17. Algorithm Design Paradigm I Solve smaller problems, and use solutions to the smaller problems to solve larger ones –Divide and Conquer Correctness: mathematical induction

18. Merge Sort Divide the n-element sequence to be sorted into two subsequences of n/2 element each Conquer: Sort the two subsequences recursively using merge sort Combine: merge the two sorted subsequences to produce the sorted answer Note: during the recursion, if the subsequence has only one element, then do nothing.

19. Merge-Sort(A,p,r) A procedure sorts the elements in the sub-array A[p..r] using divide and conquer Merge-Sort(A,p,r) –if p >= r, do nothing –if p< r then Merge-Sort(A,p,q) Merge-Sort(A,q+1,r) Merge(A,p,q,r) Starting by calling Merge-Sort(A,1,n)

20. A = MergeArray(L,R) Assume L[1:s] and R[1:t] are two sorted arrays of elements: Merge-Array(L,R) forms a single sorted array A[1:s+t] of all elements in L and R. A = MergeArray(L,R) – –for k 1 to s + t do if –then –else

21. Complexity of MergeArray At each iteration, we perform 1 comparison, 1 assignment (copy one element to A) and 2 increments (to k and i or j ) So number of operations per iteration is 4. Thus, Merge-Array takes at most 4(s+t) time. Linear in the size of the input.

22. Merge (A,p,q,r) Assume A[p..q] and A[q+1..r] are two sorted Merge(A,p,q,r) forms a single sorted array A[p..r]. Merge (A,p,q,r) –

23. Merge-Sort(A,p,r) A procedure sorts the elements in the sub-array A[p..r] using divide and conquer Merge-Sort(A,p,r) –if p >= r, do nothing –if p< r then Merge-Sort(A,p,q) Merge-Sort(A,q+1,r) Merge(A,p,q,r)

24. Running Time of Merge-Sort Running time as a function of the input size, that is the number of elements in the array A. The Divide-and-Conquer scheme yields a clean recurrences. Assume T(n) be the running time of merge- sort for sorting an array of n elements. For simplicity assume n is a power of 2, that is, there exists k such that n = 2 k.

25. Recurrence of T(n) T(1) = 1 for n > 1, we have if n = 1 if n > 1

26. Solution of Recurrence of T(n) T(n) = 4 nlog n + n = O(nlog n) Picture Proof by Recursion Tree