1. Ｐａｒｔｉａｌ Ｓｏｌｕｔｉ ｏｎ ａｎｄ Ｅｎｔｒｏｐｙ Tadao Takaoka Department of Computer Science University of Canterbury Christchurch, New Zealand

2. Genera framework and motivation If the given problem is partially solved, how much more time is needed to solve the problem completely. AlgorithmInput data Saved data SaveRecover Output Time =Time spent + Time to be spent A possible scenario: Suppose computer is stopped by power cut and partially solved data are saved by battery. After a while power is back on, data are recovered, and computation resumes. How much more time ? Estimate from the partially solved data.

3. Definition of entropy Let X be the data set and X be decomposed like S(X)=(X 1, …, X k ). Each X i is solved. S(X) is a state of data, and abbreviated as S. Let p i =|X i |/|X|, and |X|=n. The entropy H(S) is defined by H(S) =  [i=1, k] |X i |log(|X|/|X i |) = -n  [i=1,k] p i log(p i )  p i = 1, 0  H(S)  nlog(k), maximum when all |X i | are equal to 1/k.

4. Amortized analysis The accounting equation at the i-th operation becomes a i = t i - ΔH(S i ), actual time – decrease of entropy where ΔH(S i ) = H(S i-1 ) - H(S i ). Let T and A be the actual total time and the amortized total time. Summing up a i for i=1,..., N, we have A = T + H(S N ) - H(S 0 ), or T = A + H(S 0 ) - H(S N ). For some applications, a i = t i - cΔH(S i ) for some costant c

5. Problems analyzed by entropy Minimal Mergesort : merging shortest ascending runs Shortest Path Problem: Solid parts solved. Make a single shortest path spanning tree Minimum Spanning Tree : solid part solved. Make a single tree souurce

6. Three examples with k=3 (1) Ｍｉｎｉｍａｌ mergesort Time =O(H(S)) Xi are ascending runs S(X) = (2 5 6 1 4 7 3 8 9) X1=(2 5 6), X2 =(1 4 7), X3=(3 8 9) (2) Shortest paths for nearly acyclic graphs Time=O(m+H(S)) G=(V, E) is a graph. S(V)=(V1, V2, V3) V i is an acyclic graph dominated by v i (3) Minimum spanning tree Time = O(m+H(S)) S(V)=(V1, V2, V3), subgraph Gi=(Vi, Ei) is the induced graph from Vi. We assume minimum spanning tree Ti for Gi is already obtained

7. Time complexities of the three problems Minimal mergesort O(H(S)) worst case O(nlog(n)) Single source shortest paths O(m+H(S)) worst case time O(m+nlog(n)) data structures: Fibonacci heap or 2-3 heap Minimum cost spanning trees O(m+H(S)) Presort of edges (mlog(n)) excluded worst case time O(m+nlog(n))

8. Minimal mergesort picture Metasort ML ．．.．．. W1W1 W2W2 W S(X) S’(X) merge...

9. Minimal Mergesort M  L : first list of L is moved to the last of M Meta-sort S(X) into S’(X) by length of X i Let L = S’(X); /* L : list of lists */ M=φ; M  L; /* M : list of lists */ for i=1 to k-1 do begin W 1  M; W 2  M; W=merge(W 1, W 2 ); While L  φ and |W|>first(L) do M  L M  L End

10. Lemma. L and M are meta-sorted. If W 2 is a merged list, I.e., not original X i, |W 1 | ≦ |W 2 | ≦ 2|W 1 | Lemma. Amortized time for i-th merge a i ≦ 0 Proof. Let |W 1 |=n 1 and |W 2 |=n 2 ΔH = n 1 log(n/n 1 )+n 2 log(n/n 2 )-(n 1 +n 2 )log(n/(n 1 +n 2 )  n 1 log(1+n 2 /n 1 )+n 2 log(1+n 1 /n 2 )  n 1 log2+n 2 (log(3/2)=log(3/2)(n 1 +n 2 ) a i = t i – cΔH ≦ n 1 +n 2 -1 - ΔH /log(3/2) ≦ 0

11. Main results in minimal mergesort Theorem. Minimal mergesort sorts sequence S(X) in O(H(S)) time Theorem. Any sorting algorithm takes  (H(S)) time if |X i |  2 for all i. If |X i |=1 for all I, S(X) is reverse-sorted, and it can be sorted in ascending order, in O(n) ttime.

12. Dijkstra’s algorithm for shortest paths with a priority queue d[s]=0 S={s}; F={w|(s,w) in out(s)} while |S|<n do delete v from F such that d[v] is minimum //delete-min add v to S O(log n) for w in out(v) do if w is not in S then if w is if F then d[w]=min{d[v], d[v]+c[v,w] //decrease- key O(1) else {d[w]=d[v]+c[v,w]; add w to F} //insert O(1) end do

13. Expansion of solution set S: solution set of vertices to which shortest distances are known F: frontier set of vertices connected from solution set by single edges v w w S : solution setF : frontier s Time = O(m + nlogn)

14. Efficient shortest path algorithm for nearly acyclic graphs d[s]=0 S={s}; Q={w|(s,w) in out(s)}; // F is organized as priority queue Q while |S|<n do if there is a vertex in F with no incoming edge from V-S then choose v // easy vertex else choose v from F such that d[v] is minimum //difficult add v to S Delete v from F for w in out(v) do if w is not in S then if w is in F then d[w]=min{d[v], d[v]+c[v,w] //decrease-key else {d[w]=d[v]+c[v,w]; add w to F} //insert O(1) end do

15. Easy vertices and difficult vertices v w ws difficult easy S F

16. Entropy analysis of the shortest path algorithm There are t difficult vertices u 1, …, u t. Each u i and the following easy vertices form an acyclic sub-graph. Let {v 1,…, v k }, where v 1 =u i for some i, be one of the above acyclic sub-graphs. Let the number of descendants of v i be n i. Delete v 1, …, v k. Time = log(n i )+…+log(n k ) ≦ O(klog(n/k). Let us index k by i for u i. Then the total time for deletes is O(k 1 log(n/k 1 )+…+k t log(n/k t ) = O(H(S)) where S(V)=(V 1, …, V t ). The rest of the time is O(m). Thus total time is O(m+O(H(S))

17. Nearly acyclic graph Acyclic components are regarded as solved There are three acyclic components in this graph. Vertices in the component V i can be deleted from the queue once the distance to the trigger v i is finalized ViVi vivi Acyclic graph topologically sorted 7 2 3 6 1

18. Algorithm 3 on the reduced graph ViVi vivi V1V1 After V 1 is finalized, the reduced graph becomes acyclic

19. Minimum spanning trees Blue fonts : vertices 2 3 4 2 3 4 1 2 5 5 4 8 6 9 T1 T2 T3 L=(1 2 2 2 3 3 4 4 4 5 5 6 7 8 9)  L=(4 5 5 6 7 8 9) name =(1 1 1 1 2 2 2 3 3 3 3)  (1 1 1 1 1 1 1 3 3 3 3) 1 2 3 4 5 6 7 8 9 10 11 7

20. Kruskal’s completion algorithm 1 Let the sorted edge list L be partially scanned 2 Minimum spanning trees for G 1,..., G k have been obtained 3 for i=1 to k do for v in V k do name[v]:=k 4 while k > 1 do begin 5 Remove the first edge (u, v) from L 6 if u and v belong to different sub-trees T1 and T2 7 then begin 8 Connect T1 and T2 by (u, v) 9 Change the names of the nodes in the smaller tree to that of the larger tree; 10 k:=k - 1; 11 end 12 end.

21. Ｅｎｔｒｏｐｙ ａｎａｌｙｓｉ ｓ for name changes the decrease of entropy is ΔH = n 1 log(n/n 1 ) + n 2 log(n/n 2 ) -(n 1 +n 2 )log(n/(n 1 +n 2 )) = n 1 log(1+n 2 /n 1 ) + n 2 log(1+n 1 /n 2 ) ≥ min{n 1, n 2 } Noting that t i <= min{n 1, n 2 }, amortized time becomes a i = t i - ΔH(S i ) ≦ 0 Scanning L takes O(m) time. Thus T=(m+H(S 0 )), where H(S 0 ) is the initial entropy.