1. Computer Science 212 Title: Data Structures and Algorithms
Instructor: Harry Plantinga
Text: Algorithm Design: Foundations, Analysis, and Internet Examples by Goodrich and Tamassia

2. Computer Science 212 Data Structures and Algorithms
The Heart of Computer Science
Data structures
Study of important algorithms
Algorithm analysis
Algorithm design techniques Plus
Intelligent systems
Course Information: Grades on moodle

3. Why study DS & Algorithms?
Some problems are difficult to solve and good solutions are known
Some “solutions” don’t always work
Some simple algorithms don’t scale well
Data structures and algorithms make good tools for addressing new problems
Fun! Joy! Esthetic beauty!
Example: successor in a BST?
Algorithm example: celebrity problem
Algorithms that don’t scale: e.g. using bubble sort in virtual trackball program, Microsoft Word on MB+ files
Example of bad “solutions” – making change when you run out of nickels (derive dynamic programming solution)
A better algorithm is much to be preferred over a faster computer
Story of how my old minister always used to sing the doxology…

4. Place in the curriculum
The study of interesting areas of computer science: upper-level electives
108, 112: basic tools (like algebra to mathematics)
212: More advanced tools. Introduction to the science of computing. (A little more mathematical than other core CS courses, but not as rigorous as a real math course.)
New way of thinking to many of you. But you’ll get used to it.

5. Example: Search and Replace
Goal: replace one string with another
Which version is better?
#!/usr/bin/perl
my $a = shift;
my $b = shift;
my $input = "";
while (<STDIN>) {
$input .= $_; }
while ($input =~ m/$a/) {
$input =~ s/$a/$b/;
#!/usr/bin/perl
my $a = shift;
my $b = shift;
my $input = "";
while (<STDIN>) {
$input .= $_; }
$input =~ s/$a/$b/g;
Graph data in Excel. Find the curves; fit the functions.
Acolyte: public/examples

6. Empirical Analysis Implement Gather runtime data for various inputs
Analyze runtime vs. size
Example: suppose we test swap1
Input sizes: 1, 2, 4, 8, 16, 32, 64, 128, 256
Runtimes: 70, 110, 250, 560, 810, 1750, 6510, 25680,
Get data – what now?

7. swap1 runtime analysis size Swap1 (ms) 1 70 2 110 4 250 8 560 16 81032
1750 64
6510
128
25680
256
102050
see online: acolyte:public/212/examples/swap/swap1

8. swap2 runtime analysis size swap2 4 70 8 50 16 32 80 64 100 128 120
256
230
512
380
1024
580
2048
910
4096
1680
8192
3130
16384
6160

9. Can I solve the "bad algorithm" problem by just getting a faster computer?

10. How do we find the runtime from the fitted straight line?
Y = x^3
log(y) = 3 log(x)
Slope is exponent

11. Note that slope gives growth rate of runtime Note that first few points are inflated by startup costs

12. Empirical runtime analysis
What can we say about the runtimes of these two algorithms?
Could give runtime as a function of input size, for a particular implementation
But would that apply on different hardware?
What about a different implementation in C++ instead of Perl?
Is there anything we can say that is true in general?
Is version 2 better than version 1 in general? Why?
Have we done science? I.e. have we proven that version 1is better than version 2, in general, i.e. independent of implementation details?

13. Empirical analysis: conclusions
Empirical analyses are implementation-dependent
You have to use representative sample data
You can learn something about the growth rate from the slope or curve of the runtime graph
Bad algorithms can’t be fixed with faster computers
Big companies (e.g. Microsoft) are not immune to bad algorithms

14. What’s the runtime? for i  0 to n  1 do for j  0 to n  1 do
if A[i]  A[j] then
A[j]  A[i]
What's the runtime?
How do you know?
What is n?
What does O(n^2) even mean?

15. Theoretical Analysis
Uses a high-level description of the algorithm instead of an implementation
Characterizes running time as a function of the input size, n.
Takes into account all possible inputs, often analyzing the worst case
Allows us to evaluate the speed of an algorithm independent of the hardware/software environment

16. Pseudocode (§1.1) Example: find max element of an array
Algorithm arrayMax(A, n)
Input array A of n integers
Output maximum element of A
currentMax  A[0]
for i  1 to n  1 do
if A[i]  currentMax then
currentMax  A[i]
return currentMax
Example: find max element of an array
High-level description of an algorithm
More structured than English prose
Less detailed than a program
Preferred notation for describing algorithms
Hides program design issues

17. Pseudocode Details Control flow Method declaration Method call
if … then … [else …]
while … do …
repeat … until …
for … do …
Indentation replaces braces
Method declaration
Algorithm method (arg [, arg…])
Input …
Output …
Method call
var.method (arg [, arg…])
Return value
return expression
Expressions
Assignment (like  in Java)
Equality testing (like  in Java)
n2 Superscripts and other mathematical formatting allowed

18. Questions…
Can a program be asymptotically faster on one type of CPU vs another?
Do all CPU instructions take equally long?

19. The Random Access Machine (RAM) Model
A CPU
An potentially unbounded bank of memory cells, each of which can hold an arbitrary number or character 1 2
What's the purpose of the RAM model?
Memory cells are numbered and accessing any cell in memory takes unit time.

20. Primitive Operations Basic computations performed by an algorithm
Examples:
Evaluating an expression
Assigning a value to a variable
Indexing into an array
Calling a method
Returning from a method
Basic computations performed by an algorithm
Identifiable in pseudocode
Largely independent from any programming language
Exact definition not important (constant number of machine cycles per statement)
Assumed to take a constant amount of time in the RAM model

21. Counting Primitive Operations (§1.1)
By inspecting the pseudocode, we can determine the maximum number of primitive operations executed by an algorithm, as a function of the input size
Algorithm arrayMax(A, n)
currentMax  A[0]
# operations
for i  1 to n  1 do n
if A[i]  currentMax then 2(n  1)
currentMax  A[i] 2(n  1)
{ increment counter i } 2(n  1)
return currentMax
Total 7n  1

22. Estimating Running Time
Algorithm arrayMax executes 7n  1 primitive operations in the worst case. Define:
a = Time taken by the fastest primitive operation
b = Time taken by the slowest primitive operation
Let T(n) be worst-case time of arrayMax. Then a (7n  1)  T(n)  b (7n  1)
Hence, the running time T(n) is bounded by two linear functions

23. Theoretical analysis: conclusion
We can count the number of RAM-equivalent statements executed as a function of input size
All remaining implementation dependencies amount to multiplicative constants, which we will ignore
(But watch out for statements that take more than constant time, such as $input =~ s/$a/$b/g)
Linear, quadratic, etc. runtime is an intrinsic property of an algorithm

24. What’s the runtime? int n; cin >> n; for (int i=0; i<n; i++)
for (int j=0; j<n; j++)
for (int k=0; k<n; k++) {
cout << "Hello, ";
cout << "greetings, ";
cout << "bonjour, ";
cout << "guten tag, ";
cout << "Здравствуйте, ";
cout << "你好 ";
cout << " world!"; }
You've seen big-O notation. What is the runtime of this function in terms of n?
Does the constant matter? Why?

25. What’s the runtime? int n; cin >> n; if (n<1000)
for (int i=0; i<n; i++)
for (int j=0; j<n; j++)
for (int k=0; k<n; k++)
cout << "Hello\n";
else
cout << "world!\n";
What's the runtime here?
Isn't n^3 a constant when n is <1000?

26. Function Growth Rates Growth rates of functions:
Linear  n
Quadratic  n2
Cubic  n3
In a log-log chart, the slope of the line corresponds to the growth rate of the function

27. Constant factors The growth rate is not affected by Examples
constant factors or
lower-order terms
Examples
102n is a linear function
105n n is a quadratic function

28. Asymptotic (big-O) Notation (§1.2)
Given functions f(n) and g(n), we say that f(n) is O(g(n)) if there are positive constants c and n0 such that
f(n)  cg(n) for n  n0
Example: 2n + 10 is O(n)
2n + 10  cn
(c  2) n  10
n  10/(c  2)
Pick c = 3 and n0 = 10

29. Example Example: the function n2 is not O(n) n2  cn n  c
The above inequality cannot be satisfied since c must be a constant

30. More Big-O Examples 3n3 + 20n2 + 5 3 log n + log log n 7n-2
7n-2 is O(n)
need c > 0 and n0  1 such that 7n-2  c•n for n  n0
this is true for c = 7 and n0 = 1
3n3 + 20n2 + 5
3n3 + 20n2 + 5 is O(n3)
need c > 0 and n0  1 such that 3n3 + 20n2 + 5  c•n3 for n  n0
this is true for c = 4 and n0 = 21
3 log n + log log n
3 log n + log log n is O(log n)
need c > 0 and n0  1 such that 3 log n + log log n  c•log n for n  n0
this is true for c = 4 and n0 = 2

31. Asymptotic analysis of functions
Asymptotic analysis is equivalent to
ignoring multiplicative constants
ignoring lower-order terms
“for large enough inputs”
Big-O and growth rate
Big-O gives an upper bound on the growth rate of a function
Think of it as <= [asymptotically speaking]

32. Big-O Rules
If is f(n) a polynomial of degree d, then f(n) is O(nd), i.e.,
Drop lower-order terms
Drop constant factors
Use the smallest possible class of functions, if possible
Say “2n is O(n)” instead of “2n is O(n2)”
(The former is a stronger statement)
Use the simplest expression of the class
Say “3n + 5 is O(n)” instead of “3n + 5 is O(3n)”

33. Asymptotic Algorithm Analysis
Asymptotic analysis: determine the runtime in big-O notation
To perform the asymptotic analysis
Find the worst-case number of primitive operations executed as a function of the input size
Express this function with big-O notation
Example:
We determine that algorithm arrayMax executes at most 7n  1 primitive operations
We say that algorithm arrayMax “runs in O(n) time”
Since constant factors and lower-order terms are eventually dropped anyway, we can disregard them when counting primitive operations

34. Computing Prefix Averages
Runtime analysis example: Two algorithms for prefix averages
The i-th prefix average of an array X is average of the first (i + 1) elements of X:
A[i] = (X[0] + X[1] + … + X[i])/(i+1)
Computing the array A of prefix averages of another array X has applications to financial analysis

35. Prefix Averages (Quadratic)
The following algorithm computes prefix averages in quadratic time by applying the definition
Algorithm prefixAverages1(X, n)
Input array X of n integers
Output array A of prefix averages of X #operations
A  new array of n integers n
for i  0 to n  1 do n
s  X[0] n
for j  1 to i do …+ (n  1)
s  s + X[j] …+ (n  1)
A[i]  s / (i + 1) n
return A
There's that gauss sum. You know the story about Gauss being asked in elementary school to add … + 100?

36. Prefix Averages (Linear)
The following algorithm computes prefix averages in linear time by keeping a running sum
Algorithm prefixAverages2(X, n)
Input array X of n integers
Output array A of prefix averages of X #operations
A  new array of n integers n
s 
for i  0 to n  1 do n
s  s + X[i] n
A[i]  s / (i + 1) n
return A
Algorithm prefixAverages2 runs in O(n) time

37. Arithmetic Progression
The running time of prefixAverages1 is O( …+ n)
The sum of the first n integers is n(n + 1) / 2
There is a simple visual proof of this fact
Thus, algorithm prefixAverages1 runs in O(n2) time

38. What’s the runtime? 2n3+n2+n+2? O(n3) runtime
int n;
cin >> n;
for (int i=0; i<n; i++)
for (int j=0; j<n; j++)
for (int k=0; k<n; k++)
cout << “Hello world!\n”;
2n3+n2+n+2?
O(n3) runtime
Memory leak
What if the last line is replaced by:
string *s=new string(“Hello world!\n”);
O(n3) time and space

39. What’s the runtime? O(n3) + O(n3) = O(n3)
int n;
cin >> n;
for (int i=0; i<n; i++)
for (int j=0; j<n; j++)
for (int k=0; k<n; k++)
cout << “Hello world!\n”;
O(n3) + O(n3) = O(n3)
Statements or blocks in sequence: add

40. What’s the runtime? Loops: add up cost of each iteration
int n;
cin >> n;
for (int i=0; i<n; i++)
for (int j=n; j>1; j/=2)
cout << “Hello world!\n”;
Loops: add up cost of each iteration
(multiply loop cost by number of iterations if they all take the same time)
log n iterations of n steps  O(n log n)

41. What’s the runtime? Loops: add up cost of each iteration
int n;
cin >> n;
for (int i=1; i<=n; i++)
for (int j=1; j<=i; j++)
cout << “Hello world!\n”;
Loops: add up cost of each iteration
… + n = n(n+1)/2 = Q(n2)

42. What’s the runtime? template <class Item>
void insert(Item a[], int l, int r)
{ int i;
for (i=r; i>l; i--) compexch(a[i-1],a[i]);
for (i=l+2; i<=r; i++)
{ int j=i; Item v=a[i];
while (v<a[j-1])
{ a[j] = a[j-1]; j--; }
a[j] = v; }

43. Math you need to Review Summations (Sec. 1.3.1)
Logarithms and Exponents (Sec )
Proof techniques (Sec )
Basic probability (Sec )
properties of logarithms:
logb(xy) = logbx + logby
logb (x/y) = logbx - logby
Logb xa = a logb x
logba = logxa/logxb
properties of exponentials:
a(b+c) = aba c
abc = (ab)c
ab /ac = a(b-c)
b = a logab
bc = a c*logab

44. Relatives of Big-Oh big-Omega
f(n) is (g(n)) if there is a constant c > 0
and an integer constant n0  1 such that
f(n)  c•g(n) for n  n0
big-Theta
f(n) is (g(n)) if there are constants c’ > 0 and c’’ > 0 and an integer constant n0  1 such that c’•g(n)  f(n)  c’’•g(n) for n  n0
little-o
f(n) is o(g(n)) if, for any constant c > 0, there is an integer constant n0  0 such that f(n)  c•g(n) for n  n0
little-omega
f(n) is (g(n)) if, for any constant c > 0, there is an integer constant n0  0 such that f(n)  c•g(n) for n  n0

45. Intuition for Asymptotic Notation
Big-Oh
f(n) is O(g(n)) if f(n) is asymptotically less than or equal to g(n)
big-Omega
f(n) is (g(n)) if f(n) is asymptotically greater than or equal to g(n)
big-Theta
f(n) is (g(n)) if f(n) is asymptotically equal to g(n)
little-oh
f(n) is o(g(n)) if f(n) is asymptotically strictly less than g(n)
little-omega
f(n) is (g(n)) if is asymptotically strictly greater than g(n)

46. Example Uses of the Relatives of Big-Oh
5n2 is (n2)
f(n) is (g(n)) if there is a constant c > 0 and an integer constant n0  1 such that f(n)  c•g(n) for n  n0
let c = 5 and n0 = 1
5n2 is (n)
f(n) is (g(n)) if there is a constant c > 0 and an integer constant n0  1 such that f(n)  c•g(n) for n  n0
let c = 1 and n0 = 1
5n2 is (n)
f(n) is (g(n)) if, for any constant c > 0, there is an integer constant n0  0 such that f(n)  c•g(n) for n  n0
need 5n02  c•n0  given c, the n0 that satisfies this is n0  c/5  0

47. Asymptotic Analysis: Review
What does it mean to say that an algorithm has runtime O(n log n)?
n: Problem size
Big-O: upper bound over all inputs of size n
“Ignore constant factor” (why?)
“as n grows large”
O: like <= for functions (asymptotically speaking)
W: like >=
Q: like =

48. Asymptotic notation: examples
Asymptotic runtime, in terms of O, W, Q?
Suppose the runtime for a function is
n2 + 2n log n + 40
n n1.999
n3 + n2 log n
n n2 log n
2n+ 100 n2
n+ 100 n97

49. Asymptotic comparisons
n2 = O( n1.999 )?
n = O(n log n)?
1.0001n = O(n943)?
lg n = Q(ln n)?
(Evaluate the limit of the quotient of the functions)
No – a polynomial with a higher power dominates one with a lower power
No – all polynomials (n ) dominate any polylog (log n)
No – all exponentials dominate any polynomial
Yes – different bases are just a constant factor difference

50. Estimate the runtime Suppose an algorithm has runtime Q(n3)
suppose solving a problem of size 1000 takes 10 seconds. How long to solve a problem of size 10000?
Suppose an algorithm has runtime Q(n log n)
runtime 10-8 n3; if n=10000, runtime 10000s = 2.7hr
runtime 10-3 n lg n; if n=10000, runtime 133 secs

51. Worst vs. average case
You might be interested in worst, best, or average case analysis of an algorithm
You can have upper, lower, or tight bounds on each of those functions.
Eg. For each n, some problem instances of size n have runtime n and some have runtime n2.
Worst case:
Best case:
Average case:
Q(n2), W(n), W(log n), O(n2), O(n3)
W(n), W(log n), O(n2), Q(n)
W(n), W(log n), O(n2), O(n3)
Average case: need to know distribution of inputs

52. Problem: Extensible Arrays
A data structure for a stack that is…
As fast and easy as an array (usually)
Not limited in size
How?
Extensible Array:
Usually works like a normal array
When it fills, copy the entire contents into a new array of twice the size.
Runtime?

53. Analysis of Extensible Arrays
Worst-case runtime for a push() operation?
Is that the most useful way of describing the result?
Amortized analysis: averaging out the cost over many operations.
Amortized runtime of a push() operation?
Total runtime of m push operations:
m steps when array doesn’t grow
Worst-case cost of grow operations:
…+m < 2m
Total: O(m) time for m operations.
O(1) amortized runtime per operation

54. Theoretical and empirical analysis
Why do we ignore constants in analyzing algorithms theoretically?
Wouldn’t it be better if we knew the actual constants?
Theoretical and empirical analysis gives us
Theoretical knowledge of asymptotic runtime
Empirical knowledge of actual constants for some implementation
Best of both worlds!

55. Analyze this… function fun (int n) if (n<1000)
for (i=1; i<=n; i++)
for (j=1; j<=n; j++)
for (k=1; k<=n; k++)
cout << “Hello world!\n”;
else {
j=i;
while (j >= 1)
j = j / 2;
if (n%2) }
cout << “Have a nice day.\n”;