1. In-Line Interrupt Handling for Software Managed TLBs Aamer Jaleel and Bruce Jacob Electrical and Computer Engineering University of Maryland at College Park { ajaleel, blj } @ eng.umd.edu International Conference on Computer Design (ICCD) 2001 September 24 – September 26 Austin, Texas

2. Outline Reorder buffers Interrupt handling Traditional method In-lined method ( novel solution ) Performance of in-lining TLB interrupts Conclusions

3. Reorder Buffer (ROB) Hardware data structure (queue) “Holding tank” for instructions & their pipeline state New instructions are queued at the tail and retired from the head Allows for interrupts to be handled in-order ROB 0 ROB 1 ROB 2 ROB 3 ROB 4 ROB 5 ROB 6 ROB 7 ROB 8 ROB 9 ROB 10 ROB 11 ROB 12 ROB 13 ROB 14 ROB 15 HEAD TAIL Reorder Buffer (ROB) A Hardware Data Structure Queue new instructions Retire instructions Instruction type Pipeline State Exception flags etc Empty Slots

4. Handling an Interrupt ( Traditional Method ) Interrupts handled at retire stage If ROB[ head ].has_exception = true Save state & exceptional PC Flush ROB Set PC to appropriate handler Handle exception with privileges enabled Restore exceptional PC and continue executing user code

5. A Novel Approach – In-lining Hardware knows length of handler If ROB[ head ].has_exception = true If empty slots in ROB >= L HC & enough resources Save current tail pointer & nextPC ( PC of next instruction to fetch ) Set mode to inline and reset head and tail pointers Fetch handler, once done unset INLINE mode, tail pointer, continue fetching user code When TLB updated, undo all TLB misses in ROB Else, handle interrupt by traditional method

6. Done Executing Empty ROB slot Ready To Execute Privilege Bit Set Instruction had exception Interrupt In-lining… (An Example) TLB Interrupt! CAN I INLINE? Space available = 9 >= L HC CAN INLINE! LW SHIFT SUB MISC - - - - - - - - ADDI LW MUL ADDI LW SHIFT SUB MISC - - - - - - - - - LW MUL ADDI Saved Tail Ptr CYCLE 1: TLB miss detected INLINE mode set, tail pointer & nextPC saved, reset head & tail pointers, PC = handler code L HC Reset Pointers LW SHIFT SUB MISC - - - Handler 1 Handler 2 - - - - LW MUL ADDI CYCLE 2 & 3: Handler Fetched User & Handler Execute (ROB 2,7,8) LW SHIFT SUB MISC - - - Handler 1 Handler 2 Handler 3 Handler 4 - - LW MUL ADDI Tail Ptr Head Ptr Assuming length of handler L HC = 6, Instruction Fetch/Retire Per Cycle = 2

7. Interrupt In-lining… (An Example) LW SHIFT SUB MISC - - - Handler 1 Handler 2 - - - - LW MUL ADDI CYCLE 2 & 3: Handler Fetched User & Handler Execute (ROB 2,7,8) LW SHIFT SUB MISC - - - Handler 1 Handler 2 Handler 3 Handler 4 - - LW MUL ADDI DONE FETCHING HANDLER Restore Tail Ptr LW SHIFT SUB MISC - - - - - Handler 3 Handler 4 Handler 5 TLB WR LW MUL ADDI CYCLE 4: Done Fetching Handler INLINE mode unset, tail pointer & nextPC restored. Resume fetching user code LW SHIFT SUB MISC - - - Handler 1 Handler 2 Handler 3 Handler 4 Handler 5 TLB WR LW MUL ADDI Done Executing Empty ROB slot Ready To Execute Privilege Bit Set Instruction had exception

8. Interrupt In-lining… (An Example) CYCLE 3: User & Handler Execute LW SHIFT SUB MISC - - - Handler 1 Handler 2 Handler 3 Handler 4 - - LW MUL ADDI Restore Tail Ptr LW SHIFT SUB MISC - - - - - Handler 3 Handler 4 Handler 5 TLB WR LW MUL ADDI CYCLE 4: Done Fetching Handler INLINE mode unset, tail pointer & nextPC restored. Resume fetching user code LW SHIFT SUB MISC - - - Handler 1 Handler 2 Handler 3 Handler 4 Handler 5 TLB WR LW MUL ADDI LW SHIFT SUB MISC ADDI - - - Handler 3 Handler 4 Handler 5 TLB WR LW MUL ADDI CYCLE 5: Fetch user code from where it last stopped Done Executing Empty ROB slot Ready To Execute Privilege Bit Set Instruction had exception

9. Interrupt In-lining… (An Example) Restore Tail Ptr LW SHIFT SUB MISC - - - - - Handler 3 Handler 4 Handler 5 TLB WR LW MUL ADDI CYCLE 4: Done Fetching Handler INLINE mode unset, tail pointer & nextPC restored. Resume fetching user code LW SHIFT SUB MISC - - - Handler 1 Handler 2 Handler 3 Handler 4 Handler 5 TLB WR LW MUL ADDI LW SHIFT SUB MISC ADDI - - - Handler 3 Handler 4 Handler 5 TLB WR LW MUL ADDI CYCLE 5: Fetch user code from where it last stopped LW SHIFT SUB MISC ADDI SW - - - Handler 5 TLB WR LW MUL ADDI CYCLE 6: Fetch & execute more user code Done Executing Empty ROB slot Ready To Execute Privilege Bit Set Instruction had exception

10. Interrupt In-lining… (An Example) LW SHIFT SUB MISC ADDI - - - Handler 3 Handler 4 Handler 5 TLB WR LW MUL ADDI CYCLE 5: Fetch user code from where it last stopped LW SHIFT SUB MISC ADDI SW - - - Handler 5 TLB WR LW MUL ADDI CYCLE 6: Fetch & execute more user code TLB UPDATED UNDO ALL TLB INTERRUPTS LW SHIFT SUB MISC ADDI SW - - - Handler 5 TLB WR LW MUL ADDI CYCLE 7: TLB refilled in execute stage Undo All TLB misses LW SHIFT SUB MISC ADDI SW - - - Handler 5 TLB WR LW MUL ADDI Done Executing Empty ROB slot Ready To Execute Privilege Bit Set Instruction had exception

11. Interrupt In-lining… (An Example) LW SHIFT SUB MISC ADDI SW - - - Handler 5 TLB WR LW MUL ADDI CYCLE 6: Fetch & execute more user code LW SHIFT SUB MISC ADDI SW - - - Handler 5 TLB WR LW MUL ADDI CYCLE 7: TLB refilled in execute stage Undo All TLB misses LW SHIFT SUB MISC ADDI SW - - - Handler 5 TLB WR LW MUL ADDI CYCLE 8: Handler “Vanishes” Re-access TLB LW SHIFT SUB MISC ADDI SW ADD - - - LW MUL ADDI Done Executing Empty ROB slot Ready To Execute Privilege Bit Set Instruction had exception

12. Issues With Interrupt In-lining Hardware knows handler length There should be a privilege bit per ROB entry When done fetching handler, fetch nextPC Save nextPC NOT exceptionalPC ( Add MUX ) When done updating TLB Undo all instructions w/TLB miss and set them as “ready to execute” Branch mispredictions must be handled If mispredict occurs while in-lining, replace nextPC

13. Experimental Methodology Simulation Tool: Alpha 21264 4-way OOO 80 instructions in flight FA I/D TLBs w/NMRU policy, 128-entry 8 KB page size 150 renaming registers 22 instruction D-TLB handler Benchmarks : Small scientific kernels Red Black Jacobi Matrix Multiply Quicksort

14. Why not SPEC2000? TLB miss rates are not realistic Real Life Apps Spec FP 2000 * * * * * * * * * * * * * * * * * Our benchmarks

15. Results - In-lining Limitations Benefit from in-lining: 80-90% of TLB miss interrupts Can not in-line because: Not enough space in ROB ( < 2% ) Pipeline already stalled due to lack of resources ( free registers )

16. # of User Instructions Flushed Reorder buffer 50– 55% full when interrupt occurs In-lining reduces # instr flushed by 40– 80%

17. Interrupt Overhead Cost of re-fetching and executing flushed instr In-lining reduces cost of TLB miss by 10– 40%

18. Performance of Benchmarks Execution time  by 5 – 25% for same size TLB Can get the performance of a traditional TLB with an in-lined TLB of ¼ size

19. Speedup Vs Miss Rate As TLB management  benefit from in- lining  Applications that will benefit from in-lining are those that need it the most

20. Conclusions In-lining can be used for ALL types of software handled “transparent interrupts” Avoids unnecessary flushing of pipeline In-line interrupt handling for TLB misses Cuts # of instr flushed by 55-80% Reduces overhead by 10-40% Improves performance by 5-25%

21. Future Work Speculative in-lining No need to check for ROB space, check for deadlock Energy savings by not re-fetching and re-executing instructions

22. Related Work Save entire internal state of entire pipeline and restore after completion of handler (Cyber 200 for VM interrupts) Save instruction window (ROB) as part of machine state, restore when done (Torng & Day) A new thread fetches handler code while existing thread continues fetching user code (Zilles, Emer, & Sohi)

23. Miscellaneous Slides If Time Permitting

24. Interrupts Interrupt: exceptional condition Perform behind the scenes work Transparent to the user application e.g. unaligned memory access, instruction emulation, TLB miss handling, etc Two types Software handled ( privileged code ) Hardware handled ( special hardware )

25. Precise Interrupts Precise Interrupt Everything before excepted instruction has finished execution and has committed Everything after excepted instruction has NOT committed The excepted instruction may or may not have finished execution

26. Software-Managed TLBs Vs Hardware-Managed TLBs Hardware managed TLBs outperform software managed TLBs Software managed used because of flexibility Software managed TLBs E.g. MIPS, Alpha, SPARC, PA-RISC Hardware managed TLBs E.g. IA-32, PowerPC

27. Disadvantages… Two sources of performance loss: No user code executes while exception is handled Instructions are re-fetched and re-executed Solution: avoid pipeline & ROB flushes Why is the pipeline flushed? Ensure privileges? Attach a privilege bit to ROB entry ( Henry ) Have enough space for interrupt handler?

28. Interrupt In-lining – An Example Head Pointer Tail Pointer Old Tail PointerEmpty ROB slots Waiting to be issued Had TLB Interrupt Finished execution Had MISC Interrupt Assuming length of handler L HC = 6, Instruction Fetch/Retire Per Cycle = 2 LW SHIFT SUB MISC ADDI SW ROB 8 ROB 9 ROB 10 ROB 11 ROB 12 LW MUL ADDI Privilege Bit Set

29. In-lining TLB Interrupts First level handler length is short TLB miss handlers are most commonly executed OS primitives TLB miss handling account for more than 40% of total run time and 80% of the kernels computation time TLB miss interrupts occur once every 100 – 1000 instructions in applications ranging from databases to engineering workloads

30. Issues With Interrupt In-lining In-lined instructions shouldn’t affect state of user registers Problem w/conventional method of register renaming First handler instruction should receive a mapping of the current state of register file A user instruction should receive mapping of the previous user instruction mapped

31. Alpha 21264 Pipeline Fetch Decode & Map Execute Write back Retire