1. Week 4 - Friday
CS322

2. Last time What did we talk about last time? Snow day! Before that:
Topological sort
Greedy algorithms
Interval scheduling
Interval partitioning
Minimizing lateness

3. Questions?

4. Assignment 3

5. Logical warmup
Two friends who live 36 miles apart decide to meet and start riding their bikes towards each other.
They plan to meet halfway.
Each is riding at 6mph.
One of them has a pet carrier pigeon who starts flying the instant the friends start traveling.
The pigeon flies back and forth at 18mph between the friends until the friends meet.
How many miles does the pigeon travel?

6. Scheduling to minimize lateness
Consider a problem with requests that are not fixed in time
Instead, each request i has a deadline di and requires time ti using the resources
If the finish time f(i) > di, its lateness li = f(i) – di
If the finish time f(i) ≤ di, its lateness li = 0
One goal we could have is to minimize the maximum lateness of any given job
We don't care about the sum of the lateness, just the single job that is the most overdue

7. Scheduling example Job 1: Job 2: Job 3: Solution: Lateness = 0
Deadline = 2
Deadline = 4
Deadline = 6

8. Designing the algorithm
Which request do we schedule next?
Shortest jobs first?
No, we could have short jobs with late deadlines
Jobs with the least slack time (di – ti)
No, consider Job 1 with t1 = 1 and d1 = 2 and Job 2 with t2 = 10 and d2 = 10
Sort by order of increasing deadlines?
Surprisingly, yes! Length of job doesn't matter

9. Minimizing lateness algorithm
Sort the jobs in increasing order of deadlines
For simplicity, relabel jobs and deadlines so that d1 ≤ d2 ≤ … dn
Set f = s
For i = 1, 2, …, n
Assign job i to the interval from s(i) = f to f(i) = f + ti
Set f = f + ti

10. Observations
Idle time is time when no jobs are scheduled but there are still jobs left (gaps).
There is an optimal schedule with no idle time.
We will use an exchange argument to transform the optimal schedule O into the schedule A we produce
We say that a schedule has an inversion if a job i with deadline di is scheduled before a job j with an earlier deadline dj < di
Our algorithm produces a schedule with no inversions

11. All schedules with no inversions and no idle time have the same maximum lateness
Proof:
If two different schedules have neither inversions nor idle time, they might not produce the same order of jobs, but they can only differ in the order in which jobs with identical deadlines are scheduled.
Consider such a deadline d. In both schedules, the jobs with deadline d are all scheduled consecutively (after all jobs with earlier deadlines and before all jobs with later deadlines). Among the jobs with deadline d, the last one has the greatest lateness, and this lateness does not depend on the order of the jobs. ∎

12. There is an optimal schedule with no inversions and no idle time
Proof:
We know there is an optimal schedule O with no idle time.
If O has an inversion, then there is a pair of jobs i and j such that j is scheduled right after i and has dj < di.
If there were no inversions, all of the deadlines would be in order, but the fact that there is an inversion means that some point will be reached where a job has an earlier deadline than the job before it. Let that job be i and the previous job be j.
If we swap i and j, we get a schedule with one fewer inversion.

13. Illustration of jobs i and j
Original schedule O (before swapping)
Schedule O' (after swapping)
Nothing other than i and j are affected
Job i
Job j
Job i
Job j

14. Proof continued
This new swapped schedule has a maximum lateness no larger than that of O.
Why?
Since there are no gaps, the only things affected are jobs i and j. (Nothing before or after changes.)
Job j is finishing earlier, so its lateness will not increase.
We will use a prime (') to differentiate quantities in the swapped schedule.
Now, f'(i) = f(j) and remember that di > dj
li' = f(j) – di < f(j) – dj = lj
The old maximum lateness L ≥ lj > li'
Thus, the new maximum lateness L' will not be greater ∎

15. Our greedy algorithm is optimal
Claim:
The schedule A produced by the greedy algorithm has optimal maximum lateness.
Proof:
The previous proof shows that we can construct a schedule without inversions as good as any optimal one with inversions.
The proof before that says that all schedules with no inversions and no idle time have the same maximum lateness.
Since our algorithm finds a schedule without inversions and no idle time, it must be optimal. ∎

16. Caching Caching is a common problem in computer science.
If you have a memory hierarchy, some memory is faster to access than others.
You copy data from slow memory (large, like a hard disk) into fast memory (small, like RAM).
Since cache space is limited, a cache maintenance algorithm is used to determine what data to remove from cache when new data is added.

17. Optimal caching We have n pieces of data stored in (slow) main memory
We have (fast) cache memory that can hold k < n pieces of data
We will access data items d1, d2, …, dm in order
In reality, we will not know what data items will be used ahead of time
It's still useful to study what the optimal could be
We must decide at all times which k items to keep in cache
If we try to access a piece of data not in the cache, it's called a cache miss
We want to minimize cache misses

18. Cache example We have data items a, b, and c Our cache size k = 2
Our cache initially contains a and b
The sequence of data accesses is:
a, b, c, b, c, a, b
What's the smallest number of cache misses possible?

19. Farthest in the future algorithm
When di needs to be brought into the cache
Evict the item that is needed farthest in the future
Apply this algorithm to the following problem
Data items: a, b, c, d, e
Cache size k = 3
Items a, b, c initially in the cache
Data access sequence:
a, b, c, d, a, d, e, a, d, b, c

20. Reduced schedules
The algorithm we're talking about only adds an item d into the cache in step i if there is a request for d in step i and d is not already in the cache
Such a schedule is called reduced
Consider a schedule S that is not necessarily reduced
We create a schedule S', the reduction of S:
In any step i where S brings in an item d that has not been requested, S' does not add d to the cache at that moment
Instead, it adds d to the cache at the next step j in which d is actually requested
For any reduced schedule the number of items brought in is exactly the number of misses

21. Optimality of Farthest-in-Future
Claim: Let S be a reduced schedule that makes the same evictions as SFF through the first j items in the sequence. Then, there is a reduced schedule S' that makes the same evictions as SFF for the first j + 1 items and incurs no more misses than S.

22. Equivalence of Farthest-in-Future
Proof:
Consider request j + 1 for item d = dj+1. Since S and SFF agree up to request j, they have the same cache contents.
If d is in the cache, no eviction is done, so S agrees with SFF, and S' = S.
If d is not in the cache but S and SFF both evict the same item, then again S agrees with SFF, and S' = S.

23. Proof continued
If S evicts item f but SFF evicts item e ≠ f, S and SFF don't agree through step j + 1.
First, S' must evict e instead of f. Now, we want to make S' as much as possible like S.
From request j + 2 on, S' does what S does until one of the following happens for the first time:
There is a request for g ≠ e, f that is not in the cache of S and S evicts e to make room. Since S' and S only differ on e and f, it must be that g is not in the cache of either. Now, S' evicts f, making the cache of S and S' the same. Then they can behave the same.

24. Proof continued
There is a request to f, and S evicts e'. If e' = e, then S' can already access f from the cache, and then the caches will be the same. If e' ≠ e, then S' evicts e' as well and brings in e from main memory. Then, S and S' have the same caches. However, S' is no longer reduced, so we need to transform S' to reduced schedule S''. Reduced schedule S'' doesn't increase the number of items brought in, and S'' agrees with SFF through step j + 1.

25. Proof continued
In both cases, we have a new schedule S' that agrees with SFF through the first j + 1 items and incurs no more misses than some reduced schedule S.
One of these two cases must arise before a reference to e because in step j + 1 Farther-in-Future evicted item e that would be needed farther in the future.
Before there could be a request to e, there would have been a request to f, invoke case 2. ∎

26. Optimality of Farthest-in-Future
Claim: SFF has no more misses than any other schedule S* and is thus optimal.
Proof: Consider optimal schedule S*. Starting with j = 0 (no item requests yet), construct a schedule S1 that agrees with SFF through the first step and has no more misses than S*. Inductively, we can then generate S2, S3, … Sm, where m is the total number of requests. Since Sm = SFF and Sm has no more misses than S*, SFF is optimal. ∎

27. Shortest Paths

28. Shortest path set up Directed graph G = (V, E) with start node s
Assume that there is a path from s to every other node (although that's not critical)
Every edge e has a length le ≥ 0
For a path P, length of P l(P) is the sum of the lengths of the edges on P
We want to find the shortest path from s to every other node in the graph
An undirected graph is an easy tweak

29. Designing the algorithm
Let's first look at the length of the paths, not the actual paths
We keep set S of vertices to which we have determined the true shortest-path distance
S is the explored part of the graph
Then, we try to find the shortest new path by traveling from any node in the explored part S to any node v outside
We update the distance to v and add v to S
Then, continue

30. Dijkstra's Algorithm Let S be the set of explored nodes
For each u ∈ S, we store a distance d(u)
Initially S = {s} and d(s) = 0
While S ≠ V
Select a node v ∉ S with at least one edge from S for which d'(v) = mine=(u,v):u∈S d(u) + le is as small as possible
Add v to S and define d(v) = d'(v)

31. Dijkstra's Algorithm ExampleB G E F C D 8 I H J 4 2 13 3 1 7 17 9 6

32. Consider S at any point during the algorithm
Claim: For each u ∈ S, the path Pu is a shortest s-u path
Proof by induction on the size of S:
Basis case: (n = 1)
|S| = 1 means S = {s} and d(s) = 0
Clearly, that's the best distance to s
Induction step: (n = k)
Assume that when |S| = k for k ≥ 1, S contains the shortest paths for everything in S

33. Proof continued
Suppose we are increasing S to size k + 1 by adding node v. Let (u, v) be the final edge on our s-v path Pv.
By the induction hypothesis, Pu is the shortest s-u path for all u ∈ S.
Consider any other s-v path P. Since v is not in S, P must leave S somewhere. Let y be the first node on P that is not in S, and let x ∈ S be the node just before y.

34. Proof continued
On step k + 1, we could have added y, but we didn't. Thus, there is no path from s to y through x that is shorter than Pv.
But the subpath of P up to y is such a path, so that subpath must be at least as long as Pv. Since edge lengths are nonnegative, P is at least as long as Pv.
Thus, Pv must be the shortest path to v. ∎

35. Reflections on Dijkstra's algorithm
You can think of Breadth-First Search as a pulse expanding, layer by layer, through a graph from some starting node
Dijkstra's algorithm is the same, except that the time it takes for the pulse to arrive is based not on the number of edges, but the lengths of the edges it has to pass through
Because Dijkstra's algorithm expands from the starting point to whatever is closer, it grows like a blob
There are algorithms that, under certain situations, can cleverly grow in the direction of the destination and will often take less time to find the path there

36. Running time for Dijkstra's algorithm
For n nodes, the While loop runs n – 1 times
In the worst case, we might need to look at all the edges to compute appropriate minima for each iteration
Yielding a running time of O(mn)
It turns out that using a priority queue allows us to get the time down to O(m log n)
Some deep data structure work has been done here, and I don't want to go into it
Read the literature if you want to implement the algorithm as fast as possible

37. Minimum Spanning Trees

38. Minimum spanning tree
We have a weighted, connected graph and we want to remove as many edges as possible such that:
The graph remains connected
The edges we keep have the smallest total weight
This is the minimum spanning tree (MST) problem
We can imagine pruning down a communication network so that it's still connected but only with the cheapest amount of wire total
MST algorithms are also used as subroutines in other graph problems

39. MST observations
Assuming positive edge weights, the resulting graph is obviously a tree
If the graph wasn't connected, it wouldn't be a solution to our problem
If there was a cycle, we could remove an edge, make it cheaper, and still have connectivity

40. Approaches
Kruskal's algorithm: Add edges to the MST in order of increasing cost unless it causes a cycle
Prim's algorithm: Grow outward from a node, always adding the cheapest edge to a node that is not yet in the MST
Backwards Kruskal's algorithm: Remove edges from the original graph in order of decreasing cost unless it disconnects the graph
All three algorithms work!

41. MST example 8 7 B A I D C 4 12 2 3 1 5 3 5 9 5 E 1 H 6 4 F G 2 11 5 10 4 9 1 J K L

42. Cut Property Assume all edge weights are distinct.
Let S be a subset of nodes that is neither empty nor equal to V.
Let edge e = (v,w) be the minimum-cost edge with one end in S and the other in V – S.
Every minimum spanning tree contains e.

43. Proof of Cut Property
Let T be a spanning tree that does not contain e. We will try to find an edge e' in T that is more expensive than e that we can swap with e to make a cheaper spanning tree.
The ends of e are v and w. Since T is a spanning tree, there must be a path P in T from v to w. Following P, we will eventually reach a node w' that is in V – S.
Let v' ∈ S be the node just before w' on P.
Let e' = (v', w') be the edge between v' and w'.

44. Proof continued
If we exchange e for e', we get edges T' = (T – {e'}) ∪ {e}.
T' is connected since T was connected and any path that used to cross (v', w') can follow the part of P from v' to v, the edge e, and then the part of P from w to w'.
T' is acyclic since the only cycle in T' ∪ {e'} is the one made up of e and path P, but it's gone since e' was deleted.
Both e and e' have one end in S and the other in V – S, but e is the cheapest edge with this property, so its weight is lower.
Thus, T' has lower cost than any spanning tree T that does not include e. ∎

45. Kruskal's algorithm produces an MST
Proof: Whenever we add an edge e = (v, w), let S be the set of nodes that v has a path to before adding e. Node v ∈ S. But w ∉ S, because e would otherwise create a cycle. Since e is the cheapest edge with one end in S and the other in V – S, the Cut Property says it must be part of every minimum spanning tree.
Thus, Kruskal's algorithm adds exactly those edges that must be part of every minimum spanning tree. ∎

46. Cycle Property
Assume that all edge costs are distinct. Let C be any cycle in G, and let edge e = (v,w) be the most expensive edge in C. Then e does not belong to any minimum spanning tree of G.

47. Proof of Cycle Property
Let T be a spanning tree that contains e. We can show that it doesn't have minimum cost.
If we delete e from T, it partitions nodes into two components, S, containing v, and V – S, containing w.
The edges of cycle C, with e removed, form a path P from v to w. There must be some edge e' on P that crosses from S to V – S.

48. Proof continued Consider the set of edges T' = (T – {e}) ∪ {e'}.
T' must be connected and have no cycles; thus, T' is a spanning tree.
Since e is the most expensive edge in C, e' is cheaper, and T' is cheaper than T. ∎

49. MST reflections
Using the Cut Property, it's easy to show the correctness of Prim's algorithm
Using the Cycle Property, it's easy to show the correctness of the Reverse Kruskal's algorithm
It turns out that any algorithm that follows the Cut Property to add edges to a spanning tree or any algorithm that follows the Cycle Property to remove edges from a graph (or any combination of the two) will find an MST

50. What about when some edges have the same cost?
In all MST algorithms, if there is a choice between edges with the same cost, either can be chosen
Provided that connectivity/cycle constraints are met
A way to demonstrate this is to add tiny random amounts to the weights of all edges, much smaller than the difference between any non-equal cost edges
These random changes serve as tie-breakers between edges of the same cost
However, they will not change the structure so that larger edges would have been chosen

51. Quiz

52. Upcoming

53. Next time…
Finish minimum spanning trees
Union find
Review

54. Reminders Finish Assignment 3 Read chapters 1 through 4
Due Monday before midnight
Read chapters 1 through 4
Exam 1 is next Wednesday
Review will be on Monday