1. Copyright © 2010 K Computing Use of RT-Preempt Kevin Dankwardt, Ph.D. K Computing www.kcomputing.com k@kcomputing.com

2. Copyright © 2010 K Computing Overview This talk will guide developers interested in making use of the rt- preempt effort. We will cover such things as: kernel configuration comparison of preemption features discussion of the use of priority inheritance discussion of ISRs and priority scheduling discussion of gotchas of real-time applications use of mlockall() - and issues gathering performance info, including page faults with proc files timer resolution After this talk developers should be able to more quickly make effective use of the rt-preempt work.

3. Copyright © 2010 K Computing What is real-time? There is lots of debate on the definition of real-time Let's try … hard real-time - deadlines can not be missed without causing failure of the system. For example, an airplane crashes firm real-time - may have significant jitter, but, the maximum response time is within an acceptable range soft real-time - indefinite postponement is not possible, jitter is high, average may be fine, worst case means missing some deadlines but won't cause irreparable harm. Note there are two perspectives - a real-time O/S and a real-time application. Thus, you can have a hard real-time operating system and a hard real-time application.

4. Copyright © 2010 K Computing What's Preemption All About User Mode while () {... } Kernel Mode Process Context ISR while () {... spin_lock()... spin_unlock() } while () {... } InterruptProcess Resumes Time

5. Copyright © 2010 K Computing Preemptibility Improves latency by allowing a newly awoken higher priority process to more quickly preempt a lower priority process in the midst of a system call By improving the kernel applications may run more responsively by merely switching out a standard kernel for the improved one. For example, DVD players may run more reliably on an improved kernel without needing to be aware that the kernel they are running on has been improved. With about Linux kernel release 2.2 the issue of kernel preemptibility began to get quite a lot of attention. Ingo Molnar (of RedHat) and Andrew Morton (then of The University of Wollongong) both produced patch sets that provided preemption within particularly long sections in the kernel. Improves latency by allowing a newly awoken higher priority process to more quickly preempt a lower priority process in the midst of a system call By improving the kernel applications may run more responsively by merely switching out a standard kernel for the improved one. For example, DVD players may run more reliably on an improved kernel without needing to be aware that the kernel they are running on has been improved.

6. Copyright © 2010 K Computing Low Latency via schedule() Insert additional scheduling calls in the system call code so that the scheduler reviews the queue to see if there is now a higher priority process waiting. An alternative to the preemptible kernel. On 2.6.32.8, cscope -d -k -L -3 cond_resched | wc -l produces: 410

7. Copyright © 2010 K Computing More About Preemption Points static inline int should_resched(void) { return need_resched() && !(preempt_count() & PREEMPT_ACTIVE); } static void __cond_resched(void) { add_preempt_count(PREEMPT_ACTIVE); schedule(); sub_preempt_count(PREEMPT_ACTIVE); } int __sched _cond_resched(void) { if (should_resched()) { __cond_resched(); return 1; } return 0; }

8. Copyright © 2010 K Computing Preemptible Kernels A preemptible kernel allows preemption in any place that it is not specifically prohibited - in contrast to a non- preemptible kernel that allows preemption only when explicitly asked for The preemptible kernel work takes advantage of the work in the kernel need to support multiprocessing With multiprocessing, sections of the kernel need to be protected from concurrent access.

9. Copyright © 2010 K Computing Making the kernel preemptible 2.6 kernels have preemption options that can be configured. Preemption is tied to SMP locks. Essentially, locking an SMP lock means also "don't preempt" Requires a 2.4 or newer kernel The approach makes code, that is otherwise unaware of it, preemptible. – driver writers need do nothing special to have their driver preemptible. Note that just because one’s driver does not request a lock, kernel code calling it may. Recent kernels are including more real-time preemption features. 2.6 kernels have preemption options that can be configured. Preemption is tied to SMP locks. Essentially, locking an SMP lock means also "don't preempt" The approach makes code, that is otherwise unaware of it, preemptible. – driver writers need do nothing special to have their driver preemptible. Note that just because one’s driver, for example, does not request a lock, kernel code calling it may.

10. Copyright © 2010 K Computing Comparing Preemptible Solutions Older kernels implement preemption through a counter. – When a spin-lock is acquired the counter is incremented. – When a high priority process awakens the scheduler checks to see whether the preemption counter indicates, by having the value zero, that preemption is allowed. – By employing a counter the mechanism works when locks are nested. – With this mechanism, however, any spin-lock held critical section prevents preemption, even if the lock is for an unrelated resource. Newer kernels can employ a priority inheritance mutex. – With this mechanism a high priority process can preempt a low priority process that holds a mutex for a different resource. – In addition, since they employ priority inheritance low priority processes holding a mutex can not indefinitely postpone a higher priority process waiting on the mutex. – This solves the Priority Inversion Problem. – Interrupts are also prioritized – Read-write locks are single threaded Regular preemption implements preemption through a counter. – When a spin-lock is acquired the counter is incremented. – When a high priority process awakens the scheduler checks to see whether the preemption counter indicates, by having the value zero, that preemption is allowed. – By employing a counter the mechanism works when locks are nested. – With this mechanism, however, any spin-lock held critical section prevents preemption, even if the lock is for an unrelated resource. RT-Preempt employ a priority inheritance mutex. – With this mechanism a high priority process can preempt a low priority process that holds a mutex for a different resource. – In addition, since they employ priority inheritance low priority processes holding a mutex can not indefinitely postpone a higher priority process waiting on the mutex. – This solves the Priority Inversion Problem. – Interrupts are also prioritized – Read-write locks are single threaded

11. Copyright © 2010 K Computing Beware code not being preemptible A test with the earlier preemptible kernel showed: – the functions read() and write() of a dynamically loaded driver were preempted just fine – the functions init_module(), open(), and close() were not preempted because the kernel held a lock Thus, it is still possible that a section of kernel code can hold a lock for a period longer than acceptable for one’s application.

12. Copyright © 2010 K Computing RT-Preempt Patch Emerging collection of kernel improvements. Patch kernel. - CONFIG_PREEMPT_RT=y - CONFIG_HIGH_RES_TIMERS=y preemptible rtmutexes instead of spinlocks/rwlocks. Priority inheritance for in-kernel spinlocks and semaphores. Interrupt handlers are preemptible kernel threads, thus scheduled like processes A real-time app can therefore have a priority higher than ISRs. Userspace POSIX timers with high resolution. Not ALL interrupts are handled this way. Look at output of ps to see a thread per IRQ Use chrt to change priority

13. Copyright © 2010 K Computing Usage Of Preempt Macros excludes defconfig, Makefile, Documentation For 2.6.31.12 with rt-preempt applied – w/o rt-preempt “DESKTOP” is just “PREEMPT” – w/ rt-preempt “PREEMPT = DESKTOP || RT”

14. Copyright © 2010 K Computing Priority Inheritance Priority inversion is when a lower priority process holds a resource, a semaphore, say, that a higher priority process needs. The higher priority process will not get the semaphore until the lower priority process gives it up. Now, what if a third process, one with an intermediate priority, is CPU bound? The low priority process may never get the CPU so that it can complete the critical section and give up the semaphore. Thus, the high priority process is indefinitely postponed. Priority inheritance solutions give the low priority task, the priority of the highest priority task waiting for the resource so that the formerly low priority task is not preempted by intermediate priority tasks

15. Copyright © 2010 K Computing Real-time Applications Avoid page faults – mlockall() – do not create threads/processes after initial (non- realtime) phase of program. – Don't grow/shrink (manage your own pool of memory if needed). Also, grow your stack to its maximum needed size. – Don't open files. – Give your process a real-time priority The Big Kernel Lock – in other apps. – don't use file locking – don't use regular ioctls (use unlocked_ioctl) http://rt.wiki.kernel.org/index.php/HOWTO:_Build_an_RT-application

16. Copyright © 2010 K Computing Page Fault Info getrusage() watch -n 1 grep pgmajfault /proc/vmstat watch -n 1 'cut -f12 -d” “ /proc/ /stat' watch -n 1 “egrep 'VmRSS|VmSize' /proc/ /status”

17. Copyright © 2010 K Computing Timers test with cyclictest.c from rt.wiki.kernel.org clock_gettime() timer_create()

18. Copyright © 2010 K Computing Comparing Preemption Choices

19. Copyright © 2010 K Computing Timing Data

20. Copyright © 2010 K Computing Timer Resolution We want to wake up ASAP when it is time for a timer to expire. Or in other words, we don't want to wait extra long because the resolution of time is too coarse. grep resolution in /proc/timer_list –.resolution: 1 nsecs – happy days

21. Copyright © 2010 K Computing Conclusion Cheers and Good Luck The Rules: 1. Don't do bad things in your application. 2. Don't let other tasks do bad things. 3. Don't allow your system to have bad things done to it.