Asymptotic Complexity (Big-O Notation) CS 1037 Fundamentals of Computer Science II

What is a “Fast” Algorithm? Best case Average case Worst case Small problems Large problems 2 Best case Average case Worst case Small problems Large problems Takes the fewest seconds? Takes the fewest steps? In practice, these matter. Choice depends on: application requirements computer architecture expected input PP PP In theoretical comp. sci., algorithms ranked by: fewest # of steps, in the worst case, on large problems T

Ranking Running-Time Suppose algorithm A takes an 2 +b steps on problem of size n in worst case Suppose algorithm B takes cnlgn+d steps Which algorithm is faster in practice? – depends on a,b,c,d and typical input Which algorithm is faster in theory? – for large enough n coefficients a,b,c,d don’t matter – algorithm B therefore fundamentally faster than A 3

Example: Algorithm A vs B Force cnlgn+d to have large c... 4 B faster than A for large enough n A B

When is Algorithm A “as fast as” B? Let A(n) be # steps for A on problem size n Let B(n) be # steps for algorithm B – e.g. define functions A(n)= 4n+2, B(n)= n 2 A as fast as B if A(n) · kB(n) for all large enough n and some k > 0 4n+2 · n 2 for all n ¸ 5 4n+2 · 2n 2 for all n ¸ 3 20nlgn · n 2 for all n ¸ n time # items 5

Asymptotic Notation (“Big-O”) Let g(n) be some function of n Define O(g(n)) to be set of all functions as fast as g(n) in theoretical sense: 6 O(g(n)) = f(n) j there exist positive k, n 0 such that 0 · f(n) · kg(n) for all n ¸ n 0 f g n time # items n2n2 f(n) 2 O(n2)f(n) 2 O(n2) n0n0 n 2 O(n) n 2 O(n 2 ) n 2 2 O(n) e.g.

Asymptotic Notation (“Big-O”) Intuition: n O(n) because n+100 · 50n for n ¸ 3 or because n+100 · 2n for n ¸ 100 n 2 2 O(n) because for any k>0 eventually n 2 > kn for large n (i.e. “ n 2 slower than n ”) any constant c 2 O(n) because c · n for n ¸ c 7 f(n) 2 O(g(n)) “ f(n) no slower than kg(n) for large n ”

Asymptotic Notation (“Big-O”) Claim: any function of the form an+b with a ¸ 0 is a member of the set O(n) Proof: choose k = a+ j b j, then an+b · (a+ j b j )n 8 n ¸ 1 Claim: any an 2 +bn+c 2 O(n 2 ) if a ¸ 0 Proof: choose k = a+ j b j + j c j, then an 2 +bn+c · (a+ j b j + j c j )n 2 8 n ¸ 1 8

Asymptotic Notation (“Big-O”) Can say Insertion Sort runs in... – “quadratic time in the worst-case”, or – O(n 2 ) time... “oh of n squared time” Can say Binary Search runs in... – “logarithmic time in the worst-case”, or – O(lgn) time... “oh of log n time” Can say Counting Sort runs in... – “time linear in n and k ”, or – O(n+k) time... “oh of n plus k time” 9 this k means “max range of items”

Asymptotic Notation (“Big-O”) Claim: if f(n) 2 O(g(n)) and g(n) 2 O(h(n)) then f(n) 2 O(h(n)) Proof: We know for some n 1,n 2,k 1,k 2 > 0 0 · f(n) · k 1 g(n) 8 n ¸ n 1 0 · g(n) · k 2 h(n) 8 n ¸ n 2 and therefore 0 · f(n) · k 1 k 2 h(n) 8 n ¸ max f n 1,n 2 g 10

Big-O Notation Used Everywhere 11 DOCUMENTATION COMP. SCI. COURSES ALGORITHMS RESEARCH

More Examples of Big-O Membership n+50 6n n 2 +10n+50 n 3 ¡ 10n 2lgn+50 n+lgn+50 2nlgn+n+lgn 2 n +500n 3 c n +n c O(1) O(n) O(n 2 ) O(n 3 ) O(lgn) O(n) O(nlgn) O(2 n ) O(c n ) if c > 1 We say any constant c 2 O(1), i.e. O(1) = set of all constants Such functions are “exponential in n ” and grow extremely fast! Such algorithms take longer than age of universe to compute medium-sized problems

History of Big-O Invented by mathematician Paul Bachmann in 1894 to characterize error bounds Don Knuth popularized use for running time bounds in 1970s 13 The Art of Computer Programming, Volume 1

Knuth on Importance of Big-O “Students will be motivated to use O(n) notation for two important reasons. First, it significantly simplifies calculations because it allows us to be sloppy — but in a satisfactorily controlled way. Second, it appears in the power series 14 calculations of symbolic algebra systems like Maple and Mathematica, which today’s students will surely be using...” —Don Knuth advocating Big-O in high school, Letter to The American Mathematical Society March

END OF COURSE MATERIAL hooray!!! 15

variables if-else loops arrays pointers functions CS1036 Programming Skills Take Time! 16 effectiveness of you stuff in your brain structs classes templates sorting lists/queues trees CS1037 dictionaries hashing more… CS2210 SE2205

Big Universe of Ideas! 17 variables loops if-else arrays pointers functions exceptions structs classes templates sorting lists queues trees big-O recursion polymorphism inheritance debugging assertions iterators threads networking security optimization encapsulation searching hashing clustering heaps balanced trees caching randomized algorithms graphs matching NP-completeness computability theory finite automata approximation algorithms multisets dynamic programming compression float arithmetic symbolic computation GPUs distributed computing compilers operating systems greedy algorithms interpreters parsing databases dictionaries stacks user interfaces Java Python virtual machines C# backtracking profiling reducibility C data mining assembly file I/O proof techniques portability linear algebra rasterization references physics simulation version control concurrency combinatorics UML

Language Popularity in source:

Goals of CS 1037 Understand how to… – use data structures – implement data structures – characterize an algorithm’s performance – express computation in C++ Develop intuition about… – best data structure / algorithm for situation – good code, poor code, and dangerous code Programming experience, confidence 19

Pearls of Wisdom 20 “Anybody who comes to you and says he has a perfect language is either naive or a salesman.” —Bjarne Stroustrup at Waterloo Comp Sci Club Meeting “A program that has not been tested does not work.” —Bjarne Stroustrup

Pearls of Wisdom 21 “The psychological profiling [of a programmer] is mostly the ability to shift levels of abstraction, from low level to high level. To see something in the small and to see something in the large.” —Don Knuth “Premature optimization is the root of all evil.” —Don Knuth remember that one!!

EXAM REVIEW TIME! hooray? 22