analysis of algorithms Richard Kelley

welcome!  You’re in SEM 261.  This is analysis of algorithms  Please make sure you’re in the right place.

administrivia  We have to get through some boring stuff today.  Class Policies  course webpages  who I am  textbooks  homework  quizzes & exams  we’ll also do a little bit of fun stuff.  No heavy lifting today.

course webpage & syllabus  The main course page is  You’ll find the following there:  Syllabus  Schedule  Slides  Example Programs  Possibly other sites, as demand dictates  Code submission?  Blog with lecture notes?

meet the course staff  I’m Richard Kelley  call me Richard  PhD Student – don’t call me “Dr” or “Professor”  Social Artificial Intelligence & Machine Learning  Robotics  Multiagent Systems  I took this class as an undergrad at UW.  I’ve TA’ed for this class for a few years here.

textbooks  no required text  two (highly) recommended texts:  Algorithm Design by Kleinberg & Tardos  Introduction to Algorithms by Cormen, et. al.

instead of required textbooks  my slides  slides from previous semesters  links to online resources  videos  short notes  sample programs  all available from the course website.  when I use a resource, I will point you to it.

homework  regularly assigned  I’m aiming for at least one per week  some really easy  as in, “make sure you can compile this program.”  some more involved  programming  writing  English exposition  proofs  first (easy) one today... maybe.  Linux only 

quizzes and exams  necessary evil  regular quizzes to make sure you’re paying attention  these should be easy, count as participation  a midterm  about half way through the material  a final  comprehensive

grading  homework (55% of your grade)  some theory  some programming  some writing  midterm (15% of your grade)  final (25% of your grade)  attendance & participation (5% of your grade)  short quizzes

what’s this course about?  we want to write good programs.  most advanced courses you take cover topics  operating systems  networking  graphics  robotics  some courses are about the processes that lead to better code  human-computer interaction  software engineering  the goal is that at the end of this course, your programs will be better  regardless of topic  regardless of process

what is a good program?  I know it when I see it?  some common qualities  easy to read  easy to modify  lots of tests  correct!  fast  no memory leaks

what’s wrong with these measures?  nothing!  but we’re going to focus (mostly) on the last two.  we want to write programs that are fast.  we want to write programs that use little memory.  we’re going to take a slightly more general view, though.

not programs. algorithms.  it turns out that focusing on programs is a bad idea.  computers tend to get faster over time.  one way to make your program faster is to buy a better machine.  although faster machines are good, we can do better if we’re more careful in our design.  this means choosing better algorithms

what is an algorithm?  we can be really picky here  but we won’t be  for our purposes, an algorithm is  a step-by-step process for a well-defined problem  takes zero or more inputs  produces zero or more outputs  is finite  is clearly defined  we’ll use pseudocode to describe algorithms

what is a good algorithm?  most problems can be solved in many different ways, using many different algorithms.  some algorithms are better than others  faster or more space efficient  we want to choose the “best” algorithms  let’s focus on time and try to work out what this means.  (I’m following Kleinberg & Tardos in this process)

why time?  even though there are a few quantities we want to optimize our program for, we’re going to spend most of our energy on running time.  this is pretty standard.  many of the ideas here carry over to other types of analysis.

basic notation  we need to have notation for  the size of an input  the running time of the algorithm  we’ll use variables like n and m to talk about input sizes.  we focus on input sizes instead of inputs  running time often depends more on the size of the input than the actual value of the input.  we’ll use T( ) to talk about running times.  for example, T(n) is the running time of an algorithm on an input of size n.

attempt 1: good means fast  idea: implement the algorithm. run it on some inputs, record the times  this is acceptable in some cases  in the real world, profiling our programs is a common optimization.  this approach has fundamental limitations that we want to (and can easily) avoid.  at least one homework assignment will involve this kind of analysis though.

why does this fail?  problems:  sensitive to the computer we run on  sensitive to the quality of our implementation  sensitive to the inputs we choose  doesn’t tell us how running time scales with increasing problem sizes.  scale:  if I run two different algorithms on inputs of size one, and then I run them on inputs of size two, how much longer do things take?  rather than look at implementations, we need to think mathematically.

attempt 2: better than brute force  let’s start by making an agreement:  we will focus on worst-case performance.  we want to find a bound on the largest possible running time an algorithm could have, over all inputs of size n.  this has a nice property:  not sensitive to inputs we choose.  next, let’s define “good” as “better than a brute-force search through all possible solutions.”

brute-force search  most problems have a natural “solution space”  sorting  path finding in graphs  most optimization problems  with discrete (finite) problems, we can often just look at every possible solution to see if it’s what we want.  this is usually not tractable.  solution spaces tend to grow quickly  in sorting, brute force requires looking at n! possible arrangements.

what’s wrong with this definition?  “good” as “better than brute force” is certainly better than our first attempt, but it’s not perfect.  main problem  it’s too vague  we solve vagueness by making our definition more mathematical.

attempt 3: polynomial time  in brute-force search, when we increase the problem size by one, we increase the number of possibilities multiplicatively.  in sorting, we go from n! to (n+1)!  a good algorithm should scale better.  if we double the input size, we’d like the algorithm to slow down by only a constant factor.  we get this behavior by agreeing that good algorithms are efficient, and  an algorithm is efficient if its running time is bounded above by some fixed polynomial.

terminology  if an algorithm’s running time is bounded above by a fixed polynomial, we may say:  the algorithm has a polynomial running time.  the algorithm runs in polynomial time.  the algorithm is a polynomial-time algorithm.  if a problem has an algorithmic solution that runs in polynomial time, we might say:  the problem is in the complexity class P.  the problem is in P.  For instance, you probably know that sorting is in P.  bubble sort’s running time is bounded above by n^10.

our goal: revised  based on this discussion, we can be more precise about the main goals of this course:  we want to find polynomial-time algorithms for important problems.  we want to be able to prove that our algorithms run in polynomial time.  we also want to answer the questions  are there interesting problems (that are solvable) that are not in P?  for such problems, what do we do?

extras  this is a theory course  I’ll provide opportunities to delve into more theoretical items as the course progresses.  optional for undergrads, less optional for grads.  this is a practical course  you should be able to use this stuff every day.  most of what we’re learning is at the core of the C++ standard library & the boost library.  as the course progresses, we’re going to learn how to use the standard library to do a lot for us.  we’re also going to cover C++11 (the standard formerly known as C++0x)  algorithmic portions only – no variadic template craziness.

the schedule: main topics  basics  mathematical foundations  basic analysis  easy problems  graph algorithms  (discrete) optimization – dynamic programming & greedy  hard problems  intractability  randomness and approximation  concurrency

other stuff  we’ll cover a few topics that are not directly related to algorithms, but are useful to know.  unit testing – to make grading easier.  profiling – to figure out what parts of your program are taking so long.  static & dynamic analysis tools – to find bugs and general badness in your code.

extra #1: std::vector  assignment #0  full instructions online  download assignment0.cpp from the course webpage.  make sure you can compile it.  the program illustrates the use of std::vector, which you can think of as a dynamically-resizable array.  you should get in the habit of using std::vector whenever you need a contiguous block of storage.  basic usage is similar to arrays, and is illustrated in the program.  from now on in this class, no more raw arrays.

next time: math review  hopefully I’ve convinced you that we need to do some math.  the next few lectures will review the most important ideas and techniques that we have to apply to prove that algorithms run in polynomial time.  also, assignment 0 is due. Assuming we can get g++ to work in the ECC.  Questions?