1. Computer Algorithms CISC4080 CIS, Fordham Univ.
Instructor: X. Zhang
Lecture 1

2. Acknowledgement
The set of slides have use materials from the following resources
Slides for textbook by Dr. Y. Chen from Shanghai Jiaotong Univ.
Slides from Dr. M. Nicolescu from UNR
Slides sets by Dr. K. Wayne from Princeton
which in turn have borrowed materials from other resources

3. Outline
What is algorithm: word origin, first algorithms, algorithms of today’s world
Pseudocode: convention and examples
Introduction to algorithm analysis: fibonacci seq calculation
counting number of “computer steps”
recursive formula for running time of recursive algorithm
math help: math. induction
Asymptotic notations
Algorithm running time classes: P, NP

4. What are Algorithms?

5. Algorithms Etymology
CS 477/677 - Lecture 1

6. Why study algorithms?
Internet: web search, packet routing, distributed file sharing
Biology: human genome project, protein folding, …
Computers: circuit layout, databases, caching,networking, compilers, ...
Computer graphics. Movies, video games, virtual reality, …
Security. Cell phones, e-commerce, voting machines,…
Multimedia. MP3, JPG, DivX, HDTV, face recognition, …
Social networks. Recommendations, news feeds, advertisements, ...
Physics. N-body simulation, particle collision simulation, ...

7. Learning Goals
Learn algorithms (sorting and searching, arithmetics, graph algorithms, linear programming algorithms), and practice implementing them in C++ (usage of C++ STL)
Algorithms analysis: correctness, efficiency (running time and space requirement)
Complexity analysis of problem itself:
Lower bound analysis: comparison based sorting cannot do better than nlogn
NP complete problem
Learn paradigms such as divide an conquer, greedy algorithms, randomization, dynamic programming and linear programming for design algorithmic solution to new problems
We will teach you how to “fish” for yourself!

8. Greedy Algorithms Similar to dynamic programming, but simpler approach
Also used for optimization problems
Idea: When we have a choice to make, make the one that looks best right now
Make a locally optimal choice in hope of getting a globally optimal solution
Greedy algorithms don’t always yield an optimal solution
Applications:
Activity selection, fractional knapsack, Huffman codes

9. Dynamic Programming
An algorithm design technique (like divide and conquer)
Richard Bellman, optimizing decision processes
Applicable to problems with overlapping subproblems
E.g.: Fibonacci numbers:
Recurrence: F(n) = F(n-1) + F(n-2)
Boundary conditions: F(1) = 0, F(2) = 1
Compute: F(5) = 3, F(3) = 1, F(4) = 2
Solution: store the solutions to subproblems in a table
Applications:
Assembly line scheduling, matrix chain multiplication, longest common sequence of two strings, 0-1 Knapsack problem

10. Decimal System & algorithms
Imagine adding two Roman numerals?
What is
Decimal system, invented in India around AD600
Uses only 10 symbols (0,1,…9)
write large number compactly
easy to perform arithmetic operations

11. Oldest Algorithms Al Khwarizmi laid out basic methods for
adding, multiplying and dividing numbers
extracting square roots
calculating digits of pi, …
These procedures were precise, unambiguous, mechanical, efficient, correct. i.e., they were algorithms, a term coined to honor Al Khwarizmi after decimal system was adopted in Europe many centuries later.

12. Algorithms that you’ve seen
Linear Search: search for an item with a matching key in an array (unsorted)
Binary Search
Bubble Sort, Insertion Sort, Selection Sort, Radix Sort
Search in a binary search tree: algorithm + data structure
Graph algorithms: traversal
Group Practice:
Idea (how does it work), correctness, efficiency

13. Example: Selection Sort
Input: a list of elements, L[1…n]
Output: rearrange elements in List, so that L[1]<=L[2]<=L[3]<…L[n]
Note that “list” is an ADT (could be implemented using array, linked list)
Ideas (in two sentences)
First, find location of smallest element in sub list L[1…n], and swap it with first element in the sublist
repeat the same procedure for sublist L[2…n], L[3…n], …, L[n-1…n]

14. Selection Sort (idea=>pseudocode)
for i=1 to n-1
// find location of smallest element in sub list L[i…n]
minIndex = i;
for k=i+1 to n
if L[k]<L[minIndex]: minIndex=k
//swap it with first element in the sublist
if (minIndex!=i)
swap (L[i], L[minIndex]);
// Correctness: L[i] is now the i-th smallest element

15. Overview of Lab1 Implement and test three basic sorting algorithms
Start from idea of the algorithms, keep refine it until it becomes code
Performance measurement: how long does it take for each sorting algorithms (functions) to sort an array of a given size
Basis of lab2 and lab3

16. Introduction to algorithm analysis
Consider calculation of Fibonacci sequence, in particular, the n-th number in sequence:
0, 1, 1, 2, 3, 5, 8, 13, 21, 34, …

17. Fibonacci Sequence 0, 1, 1, 2, 3, 5, 8, 13, 21, 34, … Formally,
Problem: How to calculate n-th term, e.g., what is F100, F200?

18. A recursive algorithm Three questions: Is it correct?
yes, as the code mirrors the definition…
How much time does it take?
Can we do better? (faster?)

19. In pursuit of better algorithms
We want to solve problems using less resource:
Space: how much memory is needed?
Time: how fast can we get the result?
Usually, the bigger input, the more memory it takes and longer it takes
it takes longer to calculate 200-th number in Fibonacci sequence than the 10th number
it takes longer to sort larger array
Efficient algorithms are critical for large input size/problem instance
Finding F100, Searching Web …
Two different approaches to evaluate efficiency of algorithms: Measurement vs. analysis

20. Experimental approach
Measure how much time elapses from algorithm starts to finishes
needs to implement, instrument and deploy
e.g., t1 = gettimeofday(&t1,NULL);
BubbleSort (a, size);
gettimeofday (&t2, NULL);
double timeInSeconds = (t2.tv_sec-t1.tv_sec) +
(t2.tv_usec-t2.tv_usec)/ ;
results are realistic, specific and random
specific to language, run time system (Java VM, OS), caching effect, other processes running
cannot shed light on: larger input size(not always possible to test all input size), faster CPU, …
Measurement is important for a “production” system/end product; but not informative for algorithm efficiency studies/comparison/prediction

21. Example (Fib1: recursive)
n T(n)ofFib1 F(n) e e e e e e e e e
Time (in seconds) n
Running time seems to grows
exponentially as n increases
Model fitting to find out T(n)?

22. Example (Fib2: iterative) e e e e e e …
e-06 …
Time (in seconds) n
Increase very slowly as n increases
Model fitting to find out T(n)?

23. Analytic approach
Is it possible to find out how running time grows when input size grows analytically?
Is running time a constant, increases linearly, logarithmically, quadratically, … exponentially?
Approach: analyze pseudocode, express total number of steps in terms of input size, and study its order of growth
results are general: not specific to language, run time system, caching effect, other processes sharing computer
shed light on effects of larger problem size, faster CPU, …
Analysis is appropriate for algorithm efficiency studies, comparison/prediction

24. Running time analysis
Given an algorithm in pseudocode or actual program
Express total number of computer steps (primitive operations) executed as a function of the size of input (n)
size of input: size of an array, polynomial degree, # of elements in a matrix, vertices and edges in a graph, or # of bits in the binary representation of input
Computer steps: arithmetic operations, data movement, control, decision making (if, while), comparison,…
each step take a constant amount of time, i.e., independent of input size
Ignore: overhead of function calls (call stack frame allocation, passing parameters, and return values)
Later: use estimated execution times of computer steps to estimate total “running time”

25. Case Studies: Fib1(n) Can you see that T(n) > Fn ? How big is T(n)?
Let T(n) be number of computer steps needed to compute fib1(n)
T(0)=1: when n=0, first step is executed
T(1)=2: when n=1, first two steps are executed
For n >1, T(n)=T(n-1)+T(n-2)+3: first two steps are executed, fib1(n- 1) is called (with T(n-1) steps), fib1(n-2) is called (T(n-2) steps), return values are added (1 step)
Can you see that T(n) > Fn ?
How big is T(n)?

26. Running Time analysis Analyze running time of recursive algorithm
Let T(n) be number of computer steps to compute fib1(n)
T(0)=1
T(1)=2
T(n)=T(n-1)+T(n-2)+3, n>1
Analyze running time of recursive algorithm
first, write a recursive formula for its running time
then, recursive formula => closed formula, asymptotic result
How fast does T(n) grow? Can you see that T(n) > Fn ?
How big is T(n)?

27. Mathematical Induction
F0=0, F1=1, Fn=Fn-1+Fn-2
We will show that Fn >= 20.5n, for n >=6 using strong mathematical induction technique
Intuition of basic mathematical induction
it’s like Domino effect
if one push 1st card, all cards fall because
1) 1st card is pushed down
2) every card is close to next card, so that when one card falls, next one falls

28. Mathematical Induction
Sometimes, we needs the multiple previous cards to knock down next card…
Intuition of strong mathematical induction
it’s like Domino effect: if one push first two cards, all cards fall because the weights of two cards falling down knock down the next card
Generalization: 2 => k

29. Fibonacci numbers F0=0, F1=1, Fn=Fn-1+Fn-2
We are going to show that Fn >= 20.5n, for n >=6 using strong mathematical induction technique
basis step: show it’s true when n=6, 7
inductive step: show if it’s true for k, k-1, then it’s true for k+1
if given Fk-1>=20.5(k-1), Fk>=20.5k, can we show Fk+1>=20.5(k+1) ?
In fact, Fn is approximately n

30. Exponential algorithms
Running time of Fib1: T(n)> n
Running time of Fib1 is exponential in n
calculate F200, it takes 2138 computer steps
on NEC Earth simulator, which executes 40 trillion (1012) steps per second, this takes at least 292 seconds.
Moore’s law (computer speeds double about every 18 months) only allows us to calculate one more term next year…
Algorithms with exponential running time are not efficient

31. Can we do better? Correctness?
Analyze running time of iterative (non-recursive) algorithm:
T(n)=1 // if n=0 return 0
+n // create an array of f[0…n]
+2 // f[0]=0, f[1]=1
+(n-1) // for loop: repeated for n-1 times
= 2n+2
T(n) is a linear function of n, or fib2(n) has linear running time

32. Alternatively… How long does it take for fib2(n) finish?
Estimation based upon CPU:
takes 1000us,
takes 200n us
each assignment takes 60us
addition and assignment takes 800us…
How long does it take for fib2(n) finish?
T(n)= n+2*60+(n-1)*800=1000n+320 // in unit of us
What about for loop itself?
We could take that into account by estimating its time too…
Again: T(n) is a linear function of n
Constants are not important: on different computers/CPU?
Complexity in systems (caching, OS scheduling) makes it pointless to do such fine-grained analysis anyway!

33. Time for exercises/examples
1. Reading algorithms in pseudocode
2. Writing algorithms in pseudocode
3. Analyzing algorithms

34. Algorithm Analysis: Example
What’s the running time of MIN?
Algorithm/Function.: MIN (a[1…n])
input: an array of numbers a[1…n]
output: the minimum number among a[1…n]
m = a[1]
for i=2 to n:
if a[i] < m: m = a[i]
return m
How do we measure the size of input for this algorithm?
How many computer steps when the input’s size is n?

35. Algorithm Analysis: bubble sort
Algorithm/Function.: bubblesort (a[1…n])
input: an array of numbers a[1…n]
output: a sorted version of this array
for i=1 to n-1:
for j=1 to n-i:
if a[j] > a[j+1]: swap (a[j], a[j+1])
return a
Does it work?
How do you choose to measure the size of input, i.e., array a?
How many computer steps when the input’s size is n?
To count steps in nested loop: when i==1, inner loop iterates for n-1 times; …

36. Algorithm Analysis: bubble sort
Algorithm/Function.: bubblesort (a[1…n])
input: an array of numbers a[1…n]
output: a sorted version of this array
for i=1 to n-1:
for j=1 to n-i:
if a[j] > a[j+1]: swap (a[j], a[j+1])
return a
i=1: inner loop (for j=1 to n-1) repeats for n-1 times
i=2: inner loop repeats for n-2 times
i=3: …
i=n-1: inner loop (for j=1 to n-(n-1)) repeats for 1 times
Total # of steps: T(n) = (n-1)+(n-2)+(n-3)+…+1

37. How big is T(n)? T(n) = (n-1)+(n-2)+(n-3)+…+1
Can you write big sigma notation for T(n)?
Can you write simplified formula for T(n)?
Can you prove the above using math. induction?
1) when n=2, left hand size =1, right hand size is
2) if the equation is true for n=k, then it’s also true for n=k+1

38. Algorithm Analysis: Binary Search
Algorithm/Function.: search (a[L…R], value)
input: an array of numbers a[L…R] sorted in ascending order, a number value
output: the index of value in array a (if value is in it), or -1 if not found
if (L>R): return -1
m = (L+R)/2
if (a[m]==value):
return m
else:
if (a[m]>value):
return search (a[L…m-1], value)
return search (a[m+1…R], value)
What’s the size of input in this algorithm?
Let T(n) be number of steps to search an array of size n (in worst case). Write recursive formula for T(n)
Observation: running time depends on where/whether value appears in the array
=> we usually focus on worst case or average case

39. mini-summary Running time analysis identifying input size n
counting, and recursive formula: number of steps in terms of n
Chapter 2: Solving recursive formula
Next:
More careful analysis: what is basic computer step, what is not basic step?
Asymptotic running time

40. A more careful analysis
Each computer step take a constant amount of time (independent from input size n)
Recall: Fn is about n, i.e., about 0.694n bits long
e.g., F100 is about 70 bits long, cannot fit in an int, or long int
Adding two arbitrarily large numbers takes longer time, not a constant time any more
How would you add 100 bits long integer?
Chapter 2: addition of two n-bit integers take time that’s roughly proportional n, i.e., cn, c is some constant

41. A more careful analysis
Given addition of two n-bit integers take time that’s roughly proportional n, i.e., cn, c is some constant
T(0)=1
T(1)=2
T(n)=T(n-1)+T(n-2) cn, n>1
We will learn Master Theorem (chapter 2) that allows us to obtain T(n) from above recursive formula…
fib1’s running time is proportional to nFn
Fn-1 is roughly 0.694n bits long
running time to add Fn-1+Fn-2 is

42. A more careful analysis
fib2’s running time T(n)
1: if n=0 return 0
n: create an array of f[0…n]
2: f[0]=0, f[1]=1
Sig summation term: for loop
fib2’s running time is proportional to n2

43. Remove Clutter constant factors are neither reliable nor important
different steps might take different time to execute
in older/newer computer, it can be bigger or smaller
lower order terms (n,3,…) is insignificant when n is big
To focus on how fast T(n) grows as n grows, we leave out
lower-order terms: n, 3 (which becomes insignificant as n becomes big)
constant coefficient of n2
Fib2(n) has a running time that is quadratic in n, proportional to n2

44. Order (Growth Rate) of functions
Growth rate of functions of n (from low to high):
log(n) < n < nlog(n) < n2 < n3 < n4 < ….< 2n

45. Running time in n We have:
Abstracts away low-order terms and constant factors
to focus on running time of an alg. as n grows to ∞
What does it mean?
Constants are still important, you still want to optimize your code to bring down constant!
lowering constants does not change growth rate of running time, or, scalability of algorithm
Constant factors do not affect “order of the function”

46. Algorithm Efficiency vs. Speed
E.g.: sorting n numbers
Friend’s computer = 109 instructions/second
Friend’s algorithm = 2n2 computer steps
Your computer = 107 instructions/second
Your algorithm = 50nlog(n) computer steps
To sort 106 numbers,
Your friend:
You:
Your algorithm finishes 20 times faster!
More importantly, the gap becomes larger with larger n!

47. What’s next? Compare asymptotic running time of algorithms
Asymptotically bound from upper and lower: Big Oh notation …

48. Big-O notations
Let f(n) and g(n) be two functions from positive integers to positive reals.
f=O(g) means: f grows no faster than g, g is asymptotic upper bound of
f = O(g) if there is a constant c>0 such that for all n, or
Most books use notations , where O(g) denotes the set of all functions T(n) for which there is a constant c>0, such that

49. Big-O notations: Example
f=O(g) if there is a constant c>0 such that for all n,
e.g., f(n)=100n2, g(n)=n3
so f(n)=O(g(n)), or 100n2=O(n3)
Exercise: show 100n2+8n=O(n2)
and nlog(n)=O(n2)

50. Exercises/Examples of Big-oh
2. Table of important Big-Oh sets
3. Warning (popular mistakes)

51. Big-O notations: rule of thumbs
f=O(g): f grows no faster than g, g is asymptotic upper bound of f), g dominates f
na dominates nb, if a>b
any exponential dominates any polynomials
any polynomial dominates any logarithm

52. Big-Ω notations
Let f(n) and g(n) be two functions from positive inters to positive reals.
f=Ω(g) means: f grows no slower than g, g is asymptotic lower bound of f
f = Ω(g) if and only if g=O(f)
or, if and only if there is a constant c, such that for all n,
What does it mean?

53. Big-Ω notations
f=Ω(g) means: f grows no slower than g, g is asymptotic lower bound of f
f = Ω(g) if and only if g=O(f)
or, if and only if there is a constant c, such that for all n,
e.g., f(n)=100n2, g(n)=n
so f(n)=Ω(g(n)), or 100n2=Ω(n)
Exercise: show 100n2+8n=Ω(n2)
and 2n=Ω(n8)

54. Big- notations
Let f(n) and g(n) be two functions from positive inters to positive reals.
Big- notation
f= (g) means: f grows no slower and no faster than g, f grows at same rate as g asymptotically
f= (g) if and only if = Ω(g) and f=O(g)
What does it mean?
Function f can be sandwiched between g by a constant factor …

55. Big- notations
f= (g) if and only if = Ω(g) and f=O(g)
e.g.,

56. mini-summary
in analyzing running time of algorithms, what’s important is scalability (perform well for large input)
therefore, constants are not important (counting if .. then … as 1 or 2 steps does not matter)
focus on higher order which dominates lower order parts
a three-level nested loop dominates a single-level loop
In algorithm implementation, constants matters!

57. Typical Running Time Functions
1 (constant running time):
Instructions are executed once or a few times
log(n) (logarithmic), e.g., binary search
A big problem is solved by cutting original problem in smaller sizes, by a constant fraction at each step
n (linear): linear search
A small amount of processing is done on each input element
n log(n): merge sort
A problem is solved by dividing it into smaller problems, solving them independently and combining the solution

58. Typical Running Time Functions
n2 (quadratic): bubble sort
Typical for algorithms that process all pairs of data items (double nested loops)
n3 (cubic)
Processing of triples of data (triple nested loops)
nK (polynomial)
2n (exponential): Fib1 (recursive function Fib1())
Few exponential algorithms are appropriate for practical use

59. NP-Completeness
An algorithm with polynomial running time is considered tractable (feasible to be solved efficiently)
Not all problems are tractable
Some problems cannot be solved by any computer no matter how much time is provided (Turing’s Halting problem) – such problems are called undecidable
Some problems can be solved in exponential time, but have no known polynomial time lag
Can we tell if a problem can be solved ?
NP, NP-complete, NP-hard

60. Summary This class focused on algorithm running time analysis
start with running time function, expressing number of computer steps in terms of input size
Focus on very large problem size, i.e., asymptotic running time
big-O notations => focus on dominating terms in running time function
Constant, linear, polynomial, exponential time algorithms …
NP, NP complete problem

61. Coming up? Algorithm analysis is only one aspect of the class
We will look at different algorithms design paradigm, using problems from a wide range of domain (number, encryption, sorting, searching, graph, …)

62. Readings
Chapter 1