1. Design and Implementation of the Recursive Virtual Address Space Model for Small Scale Multiprocessor Systems Diploma Thesis Marcus Völp Supervisor: Volkmar Uhlig Universität Karlsruhe

2. September 2002 Marcus Völp Universität Karlsruhe2 recursive virtual address space model microkernel Policy S 1 Policy S 2 Policy S 0 Policy S 1a Policy S 1b Policy S 2a Policy S 2b

3. September 2002 Marcus Völp Universität Karlsruhe3 recursive virtual address space model map

4. September 2002 Marcus Völp Universität Karlsruhe4 recursive virtual address space model unmap

5. September 2002 Marcus Völp Universität Karlsruhe5 recursive virtual address space model grant

6. September 2002 Marcus Völp Universität Karlsruhe6 Policy S 1 Policy S 2 Policy S 0 Policy S 1a Policy S 1b Policy S 2a Policy S 2b recursive virtual address space model microkernel Region Mapper Pager

7. September 2002 Marcus Völp Universität Karlsruhe7 recursive virtual address space model mapping database keep track of mappings to implement unmap No long interrupt latencies No unbounded priority inversion No helping 00 Hazelnut + Pistachio unmap with interrupts disabled

8. September 2002 Marcus Völp Universität Karlsruhe8 recursive virtual address space model mapping database keep track of mappings to implement unmap No long interrupt latencies No unbounded priority inversion No helping 00 unbounded priority inversion Hazelnut + Pistachio (if interrupts enabled) Fiasco (if helping disabled)

9. September 2002 Marcus Völp Universität Karlsruhe9 recursive virtual address space model mapping database keep track of mappings to implement unmap No long interrupt latencies No unbounded priority inversion No helping 00 Helping problematic for SMP - xcpu time donation - migration to helper cpu

10. September 2002 Marcus Völp Universität Karlsruhe10 Outline Motivation Problem analysis Unbouned priority inversion inherent to pre-order tree traversal (if not helped out) Unbounded priority inversion inherent to non-post-order tree traversal Mapping database Data Structures Locking Scheme Restart Point Tracking Performance Conclusion

11. September 2002 Marcus Völp Universität Karlsruhe11 Problem Analysis Pre-Order tree traversal  b in  b unmap ( b ) remove node c preemption  a unmap ( a ) Priority:  a >  b Solutions roll forward unmap (b) help  b keep node c (last thread removes it) a D b c

12. September 2002 Marcus Völp Universität Karlsruhe12 Problem Analysis Unbound priority inversion is inherent to non post-order tree traversal General traversal algorithm: i child nodes before parent k child nodes after parent Post order traversal: k = 0 for all nodes D b …… 1i1k a

13. September 2002 Marcus Völp Universität Karlsruhe13 Conclusion from problem analysis Post order tree traversal Find leaf node to start with Collapse tree from leaf to root of subtree

14. September 2002 Marcus Völp Universität Karlsruhe14 Mapping Database Datastructure fast insertion (map) find start-leafnode post order traversal Synchronization SMP safe avoid Priority inversion avoid Starvation => Restart Point Tracking

15. September 2002 Marcus Völp Universität Karlsruhe15 Data Structure LL Number of Pointers: 3 + depth Find first leaf: O (N) a bc e d gf hik 0 1 2 3 Post order sorted doubly linked list (page reference info+ overmapping)

16. September 2002 Marcus Völp Universität Karlsruhe16 Data Structure Tree a bc e d gf hik (left child, right sibling)

17. September 2002 Marcus Völp Universität Karlsruhe17 Data Structure LL Tree Number of Pointers: 5 Find first leaf: O (D) a bc e d gf hik

18. September 2002 Marcus Völp Universität Karlsruhe18 Data Structure LL O1 Number of Pointers: 6 Find first leaf: O (1) best O (D) worst a bc e d gf hik

19. September 2002 Marcus Völp Universität Karlsruhe19 Data Structure LL O1 Number of Pointers: 6 Find first leaf: O (1) best O (D) worst a bc e d gf hik

20. September 2002 Marcus Völp Universität Karlsruhe20 Data Structure RepresentationNumber of Pointers Find first leaf LL3 + depthO (N) LL Tree5O (D) LL O16O (1) best O (D)

21. September 2002 Marcus Völp Universität Karlsruhe21 Synchonization Datastructure consistency Atomic modifiaction No Priority Inversion => Scheduler Conscious Locks Scalability Maximal possible parallelism => Frame granular locking

22. September 2002 Marcus Völp Universität Karlsruhe22 Synchronization acquire Lockrelease Lock unpreemptable preemption pending? => yield Roll forward to Preemption point backoff code

23. September 2002 Marcus Völp Universität Karlsruhe23 Synchronization Frame granular locking (the real world) a bc de 4MB f g ij h kl 1MB …

24. September 2002 Marcus Völp Universität Karlsruhe24 Restart Point Tracking (rw) d (r) a bc 11 (rw) d (r) e (rw) a bc 11 Why Restart Point Tracking ? Required Properties Low overhead on map / unmap Guarentee fwd progress to avoid starvation

25. September 2002 Marcus Völp Universität Karlsruhe25 Restart Point Tracking Why Restart Point Tracking ? Required Properties Low overhead on map / unmap Guarentee fwd progress to avoid starvation (rw) d (r) a bc 11 (rw) a bc 11

26. September 2002 Marcus Völp Universität Karlsruhe26 Restart Point Tracking Update restart point Find restart point information Update restart point information On Wakeup Find node to proceed ! Guarantee progress of find !

27. September 2002 Marcus Völp Universität Karlsruhe27 Restart Point Tracking 423 32 Token Based Preempted Thread List

28. September 2002 Marcus Völp Universität Karlsruhe28 Token Based Preempted Thread List Restart Point Tracking 423 12

29. September 2002 Marcus Völp Universität Karlsruhe29 Token Based Preempted Thread List Restart Point Tracking 423 12

30. September 2002 Marcus Völp Universität Karlsruhe30 Restart Point Tracking Rootnode overrun detection 423 12 check overrun flag in mdb node

31. September 2002 Marcus Völp Universität Karlsruhe31 Summary Mapping Database Post order traversal Data Structures LL LL-Tree LL-O1 Synchronization Scheduler Conscious Locks Roll forward to preemption point Token based preempted thread list Guarantee progress of preempted operations Root overrun detection sizesearch time for first leaf

32. September 2002 Marcus Völp Universität Karlsruhe32 Performance Evaluation Comparable performance for map systemcall (prepare IPC, map page, resume faulting instruction) Repre- sentation # pointerSearch time leaf node Mapping node size LL3 + depthO(N)32 byte (24 byte) LL-Tree5O(D)32 byte LL-O16O(1) best O(D) worst 40 byte (32 byte) on PIII 500 MHz, 256 KB L2 Cache

33. September 2002 Marcus Völp Universität Karlsruhe33 Performance Evaluation Unmap wide mapping tree Performance of unmap systemcall … 12 n

34. September 2002 Marcus Völp Universität Karlsruhe34 Overhead to Interrupt Latencies Map: insert mapping update PTE Unmap: 3 Preemption Points 1 step: search start point remove mapping update PTE finish unmap  Mapping Database responsible for 13ms interrupt latency (largest distance between preemption points)

35. September 2002 Marcus Völp Universität Karlsruhe35 Conclusion Search time optimization for leaf node does not pay Good preemptability without helping No unbounded priority inversion No need for prevention !!! approx. 12.2% - 26% loss of performance for unmap Open: explaination of Fiasco performance bounded tree-size N for real-time unmap

36. September 2002 Marcus Völp Universität Karlsruhe36 End of slide show, click to exit

37. September 2002 Marcus Völp Universität Karlsruhe37 Page Reference Information / Overmapping [rw] [r] [rw] Increase rights omitting prior flush: same source address space same virtual source address same virtual destination address same physical page parent link

38. September 2002 Marcus Völp Universität Karlsruhe38 Page Reference Information / Overmapping 0 a cb d p a [rw] p b [rw] p d [w] p c [w] 0 a cb d p a [rw] p b [r] p d [r] 0 a cb d p a [rw] p b [r] {R} {RW}

39. September 2002 Marcus Völp Universität Karlsruhe39 THE REAL END

40. September 2002 Marcus Völp Universität Karlsruhe40 Motivation Problems in Existing Implementations Long Interrupt Latencies Orangepip, Hazelnut Unbounded Priority Inversion Calypso, Hazelnut (two phase lock) Complex Helping or Timeslice Donation Fiasco, L4 Alpha

41. September 2002 Marcus Völp Universität Karlsruhe41 Guaranteeing Forward Progress Why Restart Point Tracking ? a bc e d gf hik

42. September 2002 Marcus Völp Universität Karlsruhe42 Guaranteeing Forward Progress Why Restart Point Tracking ? a bc 11 (rw) d (r) e (rw) a bc 11 d (r) a bc 11 (rw)

43. September 2002 Marcus Völp Universität Karlsruhe43 Guaranteeing Forward Progress Restart Point Tracking

44. September 2002 Marcus Völp Universität Karlsruhe44 Guaranteeing Forward Progress Restart Point Tracking

45. September 2002 Marcus Völp Universität Karlsruhe45 Guaranteeing Forward Progress Restart Point Tracking

46. September 2002 Marcus Völp Universität Karlsruhe46 Guaranteeing Forward Progress 423 32 Restart Point Tracking Token Based Preempted Thread List

47. September 2002 Marcus Völp Universität Karlsruhe47 Experimental Results Performance: Map 4KB to handle Pagefault Unmap Deep + Wide subtree Interrupt Latencies Time OPs are rolled forward Intel Pentium III 500 MHz, 512 KB L2, 2x 16 KB L1, 64 / 32 entry 4KB TLB (data / code), 8 / 2 entry 4MB TLB

48. September 2002 Marcus Völp Universität Karlsruhe48 Experimental Results

49. September 2002 Marcus Völp Universität Karlsruhe49 Experimental Results

50. September 2002 Marcus Völp Universität Karlsruhe50 Experimental Results

51. September 2002 Marcus Völp Universität Karlsruhe51 Experimental Results Worst case interrupt latency find first leaf

52. September 2002 Marcus Völp Universität Karlsruhe52 Experimental Results

53. September 2002 Marcus Völp Universität Karlsruhe53 Interrim Measurements

54. September 2002 Marcus Völp Universität Karlsruhe54 Interrim Measurements