1. En ｔｒｏｐｙ as Computational Complexity Computer Science Replugged Tadao Takaoka Department of Computer Science University of Canterbury Christchurch, New Zealand

2. Genera framework and motivation If the given problem is partially solved, how much more time is needed to solve the problem completely. AlgorithmInput data Saved data SaveRecover Output Time =Time spent + Time to be spent A possible scenario: Suppose computer is stopped by power cut and partially solved data are saved by battery. After a while power is back on, data are recovered, and computation resumes. How much more time ? Estimate from the partially solved data.

3. Entropy Thermodynamics klog e (Q), Q number of states in the closed system, k Boltzman constant Shannon’s information theory –  [i=1, n] p i log(p i ), where n=26 for English Our entropy, algorithmic entropy to describe computational complexity

4. Definition of entropy Let X be the data set and X be decomposed like S(X)=(X 1, …, X k ). Each X i is solved. S(X) is a state of data, and abbreviated as S. Let p i =|X i |/|X|, and |X|=n. The entropy H(S) is defined by H(S) =  [i=1, k] |X i |log(|X|/|X i |) = -n  [i=1,k] p i log(p i )  p i = 1, 0  H(S)  nlog(k), maximum when all |X i | are equal to 1/k.

5. Amortized analysis The accounting equation at the i-th operation becomes a i = t i - ΔH(S i ), actual time – decrease of entropy where ΔH(S i ) = H(S i-1 ) - H(S i ). Let T and A be the actual total time and the amortized total time. Summing up a i for i=1,..., N, we have A = T + H(S N ) - H(S 0 ), or T = A + H(S 0 ) - H(S N ). In many applications, A=0 and H(S N )=0, meaning T=H(S 0 ) For some applications, a i = t i - cΔH(S i ) for some constant c

6. Problems analyzed by entropy Minimal Mergesort : merging shortest ascending runs Shortest Path Problem: Solid parts solved. Make a single shortest path spanning tree Minimum Spanning Tree : solid part solved. Make a single tree souurce

7. Three examples with k=3 (1) Ｍｉｎｉｍａｌ mergesort Time =O(H(S)) Xi are ascending runs S(X) = (2 5 6 1 4 7 3 8 9) X1=(2 5 6), X2 =(1 4 7), X3=(3 8 9) (2) Shortest paths for nearly acyclic graphs Time=O(m+H(S)) G=(V, E) is a graph. S(V)=(V1, V2, V3) V i is an acyclic graph dominated by v i (3) Minimum spanning tree Time = O(m+H(S)) S(V)=(V1, V2, V3), subgraph Gi=(Vi, Ei) is the induced graph from Vi. We assume minimum spanning tree Ti for Gi is already obtained

8. Time complexities of the three problems Minimal mergesort O(H(S)) worst case O(nlog(n)) Single source shortest paths O(m+H(S)) worst case time O(m+nlog(n)) data structures: Fibonacci heap or 2-3 heap Minimum cost spanning trees O(m+H(S)) Presort of edges (mlog(n)) excluded worst case time O(m+nlog(n))

9. Which is more sorted? Entropy H(S) = nlog(k), k=4 Entropy H(S) = O(n), more sorted

10. Minimal mergesort picture Metasort ML ．．.．．. W1W1 W2W2 W S(X) S’(X) merge...

11. Minimal Mergesort M  L : first list of L is moved to the last of M Meta-sort S(X) into S’(X) by length of X i Let L = S’(X); /* L : list of lists */ M=φ; M  L; /* M : list of lists */ If L is not empty, M  L; for i=1 to k-1 do begin W 1  M; W 2  M; W=merge(W 1, W 2 ); While L  φ and |W|>first(L) do M  L M  L End

12. Merge algorithm for lists of lengths m and n (m ≦ n) in time O(mlog(1+n/m)) by Brown and Tarjan Lemma. Amortized time for i-th merge a i ≦ 0 Proof. Let |W 1 |=n 1 and |W 2 |=n 2 ΔH = n 1 log(n/n 1 )+n 2 log(n/n 2 )-(n 1 +n 2 )log(n/(n 1 +n 2 ) = n 1 log(1+n 2 /n 1 )+n 2 log(1+n 1 /n 2 ) t i ≦ O(n 1 log(1+n 2 /n 1 )+n 2 (log(1+n 1 /n 2 )) a i = t i – cΔH ≦ 0

13. Main results in minimal mergesort Theorem. Minimal mergesort sorts sequence S(X) in O(H(S)) time If pre-scanning is included, O(n+H(S)) Theorem. Any sorting algorithm takes  (H(S)) time if |X i |  2 for all i. If |X i |=1 for all i, S(X) is reverse-sorted, and it can be sorted in ascending order, in O(n) time.

14. Minimum spanning trees Blue fonts : vertices 2 3 4 2 3 4 1 2 5 5 4 8 6 9 T1 T2 T3 L=(1 2 2 2 3 3 4 4 4 5 5 6 7 8 9)  L=(4 5 5 6 7 8 9) name =(1 1 1 1 2 2 2 3 3 3 3)  (1 1 1 1 1 1 1 3 3 3 3) 1 2 3 4 5 6 7 8 9 10 11 7

15. Kruskal’s completion algorithm 1 Let the sorted edge list L be partially scanned 2 Minimum spanning trees for G 1,..., G k have been obtained 3 for i=1 to k do for v in V k do name[v]:=k 4 while k > 1 do begin 5 Remove the first edge (u, v) from L 6 if u and v belong to different sub-trees T1 and T2 7 then begin 8 Connect T1 and T2 by (u, v) 9 Change the names of the nodes in the smaller tree to that of the larger tree; 10 k:=k - 1; 11 end 12 end.

16. Ｅｎｔｒｏｐｙ ａｎａｌｙｓｉ ｓ for name changes the decrease of entropy is ΔH = n 1 log(n/n 1 ) + n 2 log(n/n 2 ) -(n 1 +n 2 )log(n/(n 1 +n 2 )) = n 1 log(1+n 2 /n 1 ) + n 2 log(1+n 1 /n 2 ) ≥ min{n 1, n 2 } Noting that t i <= min{n 1, n 2 }, amortized time becomes a i = t i - ΔH(S i ) ≦ 0 Scanning L takes O(m) time. Thus T=(m+H(S 0 )), where H(S 0 ) is the initial entropy.

17. Single source shortest paths Expansion of solution set S: solution set of vertices to which shortest distances are known F: frontier set of vertices connected from solution set by single edges v w w S : solution setF : frontier s Time = O(m + nlogn)

18. Priority queue F is maintained in Fibonacci or 2-3 heap Delete-min O(log n) Decrease key O(1) Insert O(1)

19. Dijkstra’s algorithm for shortest paths with a priority queue, heap with time= O(m+nlog(n)) d[s]=0; S={s}; F={w|(s,w) in out(s)}; d[v]=c[s,v] for all v in F; while |S|<n do delete v from F such that d[v] is minimum // delete-min add v to S O(log n) for w in out(v) do if w is not in S then if w is if F then d[w]=min{d[v], d[v]+c[v,w] // decrease-key O(1) else {d[w]=d[v]+c[v,w]; add w to F} // insert O(1) end do

20. Sweeping algorithm Let v 1, …, v n be topologically sorted d[v 1 ]=0; for i=2 to n do d[v i ]=  for i=1 to n do do for w in out(v i ) do d[w]=min{d[v i ], d[v i ]+c[v i,w]} Time = O(m)

21. Efficient shortest path algorithm for nearly acyclic graphs d[s]=0 S={s}; F={w|(s,w) in out(s)}; // F is organized as priority queue while |S|<n do if there is a vertex in F with no incoming edge from V-S then choose v // easy vertex O(1) else choose v from F such that d[v] is minimum //difficult O(log n) add v to S find-min Delete v from F // delete for w in out(v) do if w is not in S then if w is in F then d[w]=min{d[v], d[v]+c[v,w] //decrease-key else {d[w]=d[v]+c[v,w]; add w to F} //insert O(1) end do

22. Easy vertices and difficult vertices v w ws difficult easy S F

23. Nearly acyclic graph Acyclic components are regarded as solved There are three acyclic components in this graph. Vertices in the component V i can be deleted from the queue once the distance to the trigger v i is finalized ViVi uiui Acyclic graph topologically sorted 7 2 3 8 1

24. Entropy analysis of the shortest path algorithm Generalization delete-min = (find-min, delete) to (find-min, delete, …, delete) There are t difficult vertices u 1, …, u t. Each u i and the following easy vertices form an acyclic sub-graph. Let {v 1,…, v k }, where v 1 =u i for some i, be one of the roots of the above acyclic sub-graphs. Let the number of descendants of v i in the heap be n i. Delete v 1, …, v k. Time = log(n 1 )+…+log(n k ) ≦ O(klog(n/k). Let us index k by i for u i. Then the total time for deletes is O(k 1 log(n/k 1 )+…+k t log(n/k t ) = O(H(S)) where S(V)=(V 1, …, V t ). Time O(tlog n) for t find-mins is absorbed in O(H(S)) The rest of the time is O(m). Thus total time is O(m+O(H(S))