Computer Algorithms CISC4080 CIS, Fordham Univ. Instructor: X. Zhang Lecture 1

Acknowledgement The set of slides have use materials from the following resources Slides for textbook by Dr. Y. Chen from Shanghai Jiaotong Univ. Slides from Dr. M. Nicolescu from UNR Slides sets by Dr. K. Wayne from Princeton which in turn have borrowed materials from other resources

Outline What is algorithm: word origin, first algorithms, algorithms of today’s world Introduction to algorithm analysis: fibonacci seq calculation counting number of “computer steps” recursive formula for running time of recursive algorithm math help: math. induction Asymtopic notations Algorithm running time classes: P, NP

What are Algorithms?

Algorithms Etymology CS 477/677 - Lecture 1

Decimal System Imagine adding two Roman numerals? What is Decimal system, invented in India around AD600 Uses only 10 symbols (0,1,…9) write large number compactly easy to perform arithmetic operations

Algorithms Al Khwarizmi laid out basic methods for adding, multiplying and dividing numbers extracting square roots calculating digits of pi, … These procedures were precise, unambiguous, mechanical, efficient, correct. i.e., they were algorithms, a term coined to honor Al Khwarizmi after decimal system was adopted in Europe many centuries later.

Why study algorithms? Internet: web search, packet routing, distributed file sharing, … Biology: human genome project, protein folding, … Computers: circuit layout, databases, caching,networking, compilers, ... Computer graphics. Movies, video games, virtual reality, … Security. Cell phones, e-commerce, voting machines,… Multimedia. MP3, JPG, DivX, HDTV, face recognition, … Social networks. Recommendations, news feeds, advertisements, ... Physics. N-body simulation, particle collision simulation, ...

Introduction to algorithm Consider calculation of Fibonacci sequence, in particular, the n-th number in sequence: 0, 1, 1, 2, 3, 5, 8, 13, 21, 34, …

Fibonacci Sequence 0, 1, 1, 2, 3, 5, 8, 13, 21, 34, … Formally, Problem: How to calculate n-th term, e.g., what is F100, F200?

A recursive algorithm Three questions: Is it correct? yes, as the code mirrors the definition… How much time does it take? Can we do better? (faster?)

In pursuit of better algorithms We want to solve problems using less resource: Space: how much memory is needed? Time: how fast can we get the result? Usually, the bigger input, the more memory it takes and longer it takes larger n in Fib1(), longer it takes longer to sort larger array more memory, and longer to add larger matrices Efficient algorithms are critical for large input size/problem instance Finding F100, searching Web … Two different approaches to evaluate efficiency of algorithms: Measure vs. analyze

Experimental approach Measure how much time elapses from algorithm starts to finishes needs to implement, instrument and deploy results are realistic, specific and random specific to language, run time system (Java VM, OS), caching effect, other processes running cannot shed light on: larger input size(not always possible to test all input size), faster CPU, … Measurement is important for a “production” system/end product; but not informative for algorithm efficiency studies/comparison/prediction

Analytic approach how running time grows when input size grows? constant running time, linearly, logarithmically, quadratically, … exponentially? approach: analyze pseudocode, express total number of steps in terms of input size, and study its order of growth results are general: not specific to language, run time system, caching effect, other processes sharing computer shed light on effects of larger problem size, faster CPU, … Analysis is appropriate for algorithm efficiency studies, comparison/prediction

Running time analysis Given an algorithm in pseudocode or actual program Express total number of computer steps (primitive operations) executed in terms of size of input (n) size of input: size of an array, polynomial degree, # of elements in a matrix, vertices and edges in a graph, or # of bits in the binary representation of input Computer steps: arithmetic operations, data movement, control, decision making (if, while), comparison,… each step take a constant amount of time, i.e., independent of input size Ignore: overhead of function calls (call stack frame allocation, passing parameters, and return values) Later: use estimated execution times of computer steps to estimate total “running time”

Case Studies: Fib1(n) Can you see that T(n) > Fn ? How big is T(n)? Let T(n) be number of computer steps needed to compute fib1(n) T(0)=1: when n=0, first step is executed T(1)=2: when n=1, first two steps are executed For n >1, T(n)=T(n-1)+T(n-2)+3: first two steps are executed, fib1(n-1) is called (with T(n-1) steps), fib1(n-2) is called (T(n-2) steps), return values are added (1 step) Can you see that T(n) > Fn ? How big is T(n)?

Running Time analysis Analyze running time of recursive algorithm Let T(n) be number of computer steps to compute fib1(n) T(0)=1 T(1)=2 T(n)=T(n-1)+T(n-2)+3, n>1 Analyze running time of recursive algorithm first, write a recursive formula for its running time then, recursive formula => closed formula, asymptotic result How fast does T(n) grow? Can you see that T(n) > Fn ? How big is T(n)?

Mathematical Induction F0=0, F1=1, Fn=Fn-1+Fn-2 We will show that Fn >= 20.5n, for n >=6 using strong mathematical induction technique Intuition of basic mathematical induction it’s like Domino effect if one push 1st card, all cards fall because 1) 1st card is pushed down 2) every card is close to next card, so that when one card falls, next one falls

Mathematical Induction Sometimes, we needs the multiple previous cards to knock down next card… Intuition of strong mathematical induction it’s like Domino effect: if one push first two cards, all cards fall because the weights of two cards falling down knock down the next card Generalization: 2 => k

Fibonacci numbers F0=0, F1=1, Fn=Fn-1+Fn-2 We are going to show that Fn >= 20.5n, for n >=6 using strong mathematical induction technique basis step: show it’s true when n=6, 7 inductive step: show if it’s true for k, k-1, then it’s true for k+1 if given Fk-1>=20.5(k-1), Fk>=20.5k, can we show Fk+1>=20.5(k+1) ? In fact, Fn is approximately 20.694n

Exponential algorithms Running time of Fib1: T(n)> 20.694n Running time of Fib1 is exponential in n calculate F200, it takes 2138 computer steps on NEC Earth simulator, which executes 40 trillion (1012) steps per second, this takes at least 292 seconds. Moore’s law (computer speeds double about every 18 months) only allows us to calculate one more term next year… Algorithms with exponential running time are not efficient

Can we do better? Correctness? Analyze running time of iterative (non-recursive) algorithm: T(n)=1 // if n=0 return 0 +n // create an array of f[0…n] +2 // f[0]=0, f[1]=1 +(n-1) // for loop: repeated for n-1 times = 2n+2 T(n) is a linear function of n, or fib2(n) has linear running time

Alternatively… How long does it take for fib2(n) finish? Estimation based upon CPU: takes 1000us, takes 200n us each assignment takes 60us addition and assignment takes 800us… How long does it take for fib2(n) finish? T(n)=1000 +200n+2*60+(n-1)*800=1000n+320 // in unit of us What about for loop itself? We could take that into account by estimating its time too… Again: T(n) is a linear function of n Constants are not important: on different computers/CPU? Complexity in systems (caching, OS scheduling) makes it pointless to do such fine-grained analysis anyway!

Time for exercises/examples 1. Reading algorithms in pseudocode 2. Writing algorithms in pseudocode 3. Analyzing algorithms

Algorithm Analysis: Example What’s the running time of MIN? Algorithm/Function.: MIN (a[1…n]) input: an array of numbers a[1…n] output: the minimum number among a[1…n] m = a[1] for i=2 to n: if a[i] < m: m = a[i] return m How do we measure the size of input for this algorithm? How many computer steps when the input’s size is n?

Algorithm Analysis: bubble sort Algorithm/Function.: bubblesort (a[1…n]) input: an array of numbers a[1…n] output: a sorted version of this array for i=1 to n-1: for j=1 to n-i: if a[j] > a[j+1]: swap (a[j], a[j+1]) return a Does it work? How do you choose to measure the size of input, i.e., array a? How many computer steps when the input’s size is n? To count steps in nested loop: when i==1, inner loop iterates for n-1 times; …

Algorithm Analysis: bubble sort Algorithm/Function.: bubblesort (a[1…n]) input: an array of numbers a[1…n] output: a sorted version of this array for i=1 to n-1: for j=1 to n-i: if a[j] > a[j+1]: swap (a[j], a[j+1]) return a i=1: inner loop (for j=1 to n-1) repeats for n-1 times i=2: inner loop repeats for n-2 times i=3: … i=n-1: inner loop (for j=1 to n-(n-1)) repeats for 1 times Total # of steps: T(n) = (n-1)+(n-2)+(n-3)+…+1

How big is T(n)? T(n) = (n-1)+(n-2)+(n-3)+…+1 Can you write big sigma notation for T(n)? Can you write simplified formula for T(n)? Can you prove the above using math. induction? 1) when n=2, left hand size =1, right hand size is 2) if the equation is true for n=k, then it’s also true for n=k+1

Algorithm Analysis: Binary Search Algorithm/Function.: search (a[L…R], value) input: an array of numbers a[L…R] sorted in ascending order, a number value output: the index of value in array a (if value is in it), or -1 if not found if (L>R): return -1 m = (L+R)/2 if (a[m]==value): return m else: if (a[m]>value): return search (a[L…m-1], value) return search (a[m+1…R], value) What’s the size of input in this algorithm? Let T(n) be number of steps to search an array of size n (in worst case). Write recursive formula for T(n) Observation: running time depends on where/whether value appears in the array => we usually focus on worst case or average case

mini-summary Running time analysis identifying input size n counting, and recursive formula: number of steps in terms of n Chapter 2: Solving recursive formula Next: More careful analysis: what is basic computer step, what is not basic step? Asymptotic running time

A more careful analysis Each computer step take a constant amount of time (independent from input size n) Recall: Fn is about 20.694n, i.e., about 0.694n bits long e.g., F100 is about 70 bits long, cannot fit in an int, or long int Adding two arbitrarily large numbers takes longer time, not a constant time any more How would you add 100 bits long integer? Chapter 2: addition of two n-bit integers take time that’s roughly proportional n, i.e., cn, c is some constant

A more careful analysis Given addition of two n-bit integers take time that’s roughly proportional n, i.e., cn, c is some constant T(0)=1 T(1)=2 T(n)=T(n-1)+T(n-2)+2+0.695cn, n>1 We will learn Master Theorem (chapter 2) that allows us to obtain T(n) from above recursive formula… fib1’s running time is proportional to nFn Fn-1 is roughly 0.694n bits long running time to add Fn-1+Fn-2 is

A more careful analysis fib2’s running time T(n) 1: if n=0 return 0 n: create an array of f[0…n] 2: f[0]=0, f[1]=1 Sig summation term: for loop fib2’s running time is proportional to n2

Remove Clutter constant factors are neither reliable nor important different steps might take different time to execute in older/newer computer, it can be bigger or smaller lower order terms (n,3,…) is insignificant when n is big To focus on how fast T(n) grows as n grows, we leave out lower-order terms: n, 3 (which becomes insignificant as n becomes big) constant coefficient of n2 Fib2(n) has a running time that is quadratic in n, proportional to n2

Order (Growth Rate) of functions Growth rate of functions of n (from low to high): log(n) < n < nlog(n) < n2 < n3 < n4 < ….< 2n

Running time in n We have: Abstracts away low-order terms and constant factors to focus on running time of an alg. as n grows to ∞ What does it mean? Constants are still important, you still want to optimize your code to bring down constant! lowering constants does not change growth rate of running time, or, scalability of algorithm Constant factors do not affect “order of the function”

Algorithm Efficiency vs. Speed E.g.: sorting n numbers Friend’s computer = 109 instructions/second Friend’s algorithm = 2n2 computer steps Your computer = 107 instructions/second Your algorithm = 50nlog(n) computer steps To sort 106 numbers, Your friend: You: Your algorithm finishes 20 times faster! More importantly, the gap becomes larger with larger n!

What’s next? Compare asymptotic running time of algorithms Asymptotically bound from upper and lower: Big Oh notation …

Big-O notations Let f(n) and g(n) be two functions from positive integers to positive reals. f=O(g) means: f grows no faster than g, g is asymptotic upper bound of f = O(g) if there is a constant c>0 such that for all n, or Most books use notations , where O(g) denotes the set of all functions T(n) for which there is a constant c>0, such that

Big-O notations: Example f=O(g) if there is a constant c>0 such that for all n, e.g., f(n)=100n2, g(n)=n3 so f(n)=O(g(n)), or 100n2=O(n3) Exercise: show 100n2+8n=O(n2) and nlog(n)=O(n2)

Exercises/Examples of Big-oh 2. Table of important Big-Oh sets 3. Warning (popular mistakes)

Big-O notations: rule of thumbs f=O(g): f grows no faster than g, g is asymptotic upper bound of f), g dominates f na dominates nb, if a>b any exponential dominates any polynomials any polynomial dominates any logarithm

Big-Ω notations Let f(n) and g(n) be two functions from positive inters to positive reals. f=Ω(g) means: f grows no slower than g, g is asymptotic lower bound of f f = Ω(g) if and only if g=O(f) or, if and only if there is a constant c, such that for all n, What does it mean?

Big-Ω notations f=Ω(g) means: f grows no slower than g, g is asymptotic lower bound of f f = Ω(g) if and only if g=O(f) or, if and only if there is a constant c, such that for all n, e.g., f(n)=100n2, g(n)=n so f(n)=Ω(g(n)), or 100n2=Ω(n) Exercise: show 100n2+8n=Ω(n2) and 2n=Ω(n8)

Big- notations Let f(n) and g(n) be two functions from positive inters to positive reals. Big- notation f= (g) means: f grows no slower and no faster than g, f grows at same rate as g asymptotically f= (g) if and only if = Ω(g) and f=O(g) What does it mean? Function f can be sandwiched between g by a constant factor …

Big- notations f= (g) if and only if = Ω(g) and f=O(g) e.g.,

mini-summary in analyzing running time of algorithms, what’s important is scalability (perform well for large input) therefore, constants are not important (counting if .. then … as 1 or 2 steps does not matter) focus on higher order which dominates lower order parts a three-level nested loop dominates a single-level loop In algorithm implementation, constants matters!

Typical Running Time Functions 1 (constant running time): Instructions are executed once or a few times log(n) (logarithmic), e.g., binary search A big problem is solved by cutting original problem in smaller sizes, by a constant fraction at each step n (linear): linear search A small amount of processing is done on each input element n log(n): merge sort A problem is solved by dividing it into smaller problems, solving them independently and combining the solution

Typical Running Time Functions n2 (quadratic): bubble sort Typical for algorithms that process all pairs of data items (double nested loops) n3 (cubic) Processing of triples of data (triple nested loops) nK (polynomial) 2n (exponential): Fib1 (recursive function Fib1()) Few exponential algorithms are appropriate for practical use

NP-Completeness An algorithm with polynomial running time is considered tractable (feasible to be solved efficiently) Not all problems are tractable Some problems cannot be solved by any computer no matter how much time is provided (Turing’s Halting problem) – such problems are called undecidable Some problems can be solved in exponential time, but have no known polynomial time lag Can we tell if a problem can be solved ? NP, NP-complete, NP-hard

Summary This class focused on algorithm running time analysis start with running time function, expressing number of computer steps in terms of input size Focus on very large problem size, i.e., asymptotic running time big-O notations => focus on dominating terms in running time function Constant, linear, polynomial, exponential time algorithms … NP, NP complete problem

Coming up? Algorithm analysis is only one aspect of the class We will look at different algorithms design paradigm, using problems from a wide range of domain (number, encryption, sorting, searching, graph, …)

Dynamic Programming An algorithm design technique (like divide and conquer) Richard Bellman, optimizing decision processes Applicable to problems with overlapping subproblems E.g.: Fibonacci numbers: Recurrence: F(n) = F(n-1) + F(n-2) Boundary conditions: F(1) = 0, F(2) = 1 Compute: F(5) = 3, F(3) = 1, F(4) = 2 Solution: store the solutions to subproblems in a table Applications: Assembly line scheduling, matrix chain multiplication, longest common sequence of two strings, 0-1 Knapsack problem

Greedy Algorithms Similar to dynamic programming, but simpler approach Also used for optimization problems Idea: When we have a choice to make, make the one that looks best right now Make a locally optimal choice in hope of getting a globally optimal solution Greedy algorithms don’t always yield an optimal solution Applications: Activity selection, fractional knapsack, Huffman codes

Graphs Applications that involve not only a set of items, but also the connections between them Maps Schedules Computer networks Hypertext Circuits

Searching in Graphs Graph searching = systematically follow the edges of the graph so as to visit the vertices of the graph Two basic graph methods: Breadth-first search Depth-first search The difference between them is in the order in which they explore the unvisited edges of the graph Graph algorithms are typically elaborations of the basic graph-searching algorithms u v w x y z

Variants of Shortest Paths Single-source shortest path (Bellman-Ford, DAG shortest paths, Disjkstra) G = (V, E) ⇒ find a shortest path from a given source vertex s to each vertex v ∈ V Single-destination shortest path Find a shortest path to a given destination vertex t from each vertex v Reverse the direction of each edge ⇒ single-source Single-pair shortest path Find a shortest path from u to v for given vertices u and v Solve the single-source problem All-pairs shortest-paths (Matrix multiplication, Floyd-Warshall) Find a shortest path from u to v for every pair of vertices u and v

Number Theoretic Algorithms Secured communication: RSA public-key cryptosystem Easy to find large primes Hard to factor the product of large primes Secure Message eavesdropper Authenticated Message Bob Alice

Readings Chapter 1