1. ASQF Fachgruppe Automation/ Automation Day 1 Realtime capable Linux: Requirements (Standard)LinuxRealtime LinuxRealtime kernel One (physical) address space with n-tasks Application program 100% source compatibility better binary compatibility API Response time: < 10 µsResponse time: ~1ms Response time: < 10 µs same homogeneity One process address space with n-threads All tasks have one common API implemented by one common library All threads of a process have one common API implemented by one common library

2. ASQF Fachgruppe Automation/ Automation Day 2 Realtime capable Linux: The important questions Is the POSIX-API suitable? –Only a subset is relevant for realtime applications –The mechanism for the notification of asynchronous events determine widespread usage Can it provide for competitive performance? –Ideally it should be as good as specialized realtime kernels –Ultimately cycle times of 31,25 µs should be possible What about additional changes to the Linux kernel ? –Ideally none –Only minimal changes allow for long time support

3. ASQF Fachgruppe Automation/ Automation Day 3 Thread management –Creation,Termination, Scheduling Synchronization –Mutex –Spinlock  Communication –POSIX Message Queue –Semaphore –Shared Memory  Timer – types (periodic, one shot, absolute, relative) – resolution – clock tick – notification The POSIX-API: Minimal set for realtime tasks needs improvements

4. ASQF Fachgruppe Automation/ Automation Day 4 The POSIX Notification-API: The Timer-API Timer sigevent object SIGEV_THREAD CLOCK_REALTIME SIGEV_SIGNAL SIGEV_NONE timer_create (clockid_t clockid, timer_t *timerid, struct sigevent *evp); Problems: SIGEV_THREAD: The implementation within the glibc is not realtime capable  another implementation fixes it (deterministic, performance improvement by a factor of 10) SIGEV_SIGNAL: Sending a signal to the entire process is not deterministic. Potentially the entire thread list of a process must be scanned.  minimal overhead and deterministic behavior require sending to a thread (Linux can handle it)  no response based on signal handlers (signal handlers are for synchronous exceptions only)  having explicit wait only allows for extremely fast notification.

5. ASQF Fachgruppe Automation/ Automation Day 5 SIGEV_THREAD: The realtime capable implementation sigwaitinfo() handler carrier thread signal to thread realtime implementation Solution: creation of carrier thread at timer creation Trade off between memory consumption and independent handlers controllable same pthread_attr object with same values  share an existing carrier thread different pthread_attr object or the same but with different values  create new carrier thread C ompletely deterministic Performance improvement more than a factor of 10 sigwaitinfo() signal to thread pthread_exit() handler pthread_create() carrier thread managing thread current glibc implementation pthread_exit() handler sigwaitinfo() handler carrier thread signal to thread

6. ASQF Fachgruppe Automation/ Automation Day 6 The POSIX Notification-API: Extensions(1) t timer fires sigwait() Scenario a sigwait() Scenario b timer overrun sigwait() Scenario c Can be queried This situation must get signalled An additional call sigevent_set_notification() makes sending the signal to a thread available at the API level. This call maps the threadid of the API to the tid of the kernel allows via an additional encoding of the sigev_notify field for supervision of timeliness implementation with best possible deterministic behavior The existing sigevent structure needs no changes. timer fires POSIX definition (not very helpful)

7. ASQF Fachgruppe Automation/ Automation Day 7 The POSIX Notification-API: Extensions(2) A device must be a possible source of a timer tick (clock) Example: A Profinet Realtime Ethernet Controller ERTEC provides for a 250 µsec tick, not necessarily synchronous with CLOCK_REALTIME  additional call register_clock() A firing timer is one special type of an event arbitrary asynchronous IO events should get signalled the same way  additional call event_create() Timer sigevent object SIGEV_THREAD (realtime capable implementation of the library module) CLOCK_REALTIME SIGEV_SIGNAL (inhibited) direct IO-Event create_event() Timer Device Driver clock register_clock() sigevent_set_notification() Signal to thread (optional with supervision) Extensions to the POSIX API

8. ASQF Fachgruppe Automation/ Automation Day 8 Realtime capable Linux: Base concept Linux kernel realtime kernel threads controlled by realtime scheduler threads controlled by Linux-Scheduler system call entry exactly the same for both kernels Ipipe (ADEOS) Common Library Linux Process POSIX API HW- Interrupts Soft- Interrupts static priorities (POSIX-counting) 99 0 0 realtime kernel Linux Linux + Realtime boundary implementation defined Linux

9. ASQF Fachgruppe Automation/ Automation Day 9 Typical partitioning of a realtime thread setup phase time critical response path Much functionality needed but neither realtime requirements nor high performance requirements For all calls not implemented, the thread temporarily runs under the control of the Linux scheduler Wait for event event Realtime capable Linux: Functionality of the realtime kernel minimal functionality needed minimal realtime kernel -timer functions and its notification -communication and synchronization -scheduling

10. ASQF Fachgruppe Automation/ Automation Day 10 Realtime capable Linux: Limits of homogeneity Linux Kernel Realtime Kernel threads managed by the realtime scheduler threads managed by the Linux scheduler Ipipe Linux process HW- Interrupts Soft- Interrupts Signal source eg. timer Signal source eg. IO event A signal can only get sent to that domain which created the signal source This restriction is important for best possible performance (allows for separation of implementation) acceptable and manageable (violations are reported)

11. ASQF Fachgruppe Automation/ Automation Day 11 Realtime capable Linux: Implementation Kernel Two additional loadable modules: A device driver –functional extensions of the API (API-calls  ioctl() system calls, additional internal interfaces for realtime capable device drivers) –Interface for initialization/configuration of the realtime module eg. registration of a process as realtime process The realtime module (20K for x86) –Functionality of the realtime kernel –Communication between realtime and Linux kernel Three modified kernel files: –futex.c (hook), posix_timers.c (clock registration), mqueue.c Ipipe modifications: –Systemcall handling (additional hook at system call exit + optimization) –Interrupt handling (optimization, support for truly preemptive realtime kernel)

12. ASQF Fachgruppe Automation/ Automation Day 12 Realtime capable Linux: Implementation glibc Modified functions: Initialization –Registration of the realtime process Spinlock function also for static priorities Realtime capable implementation for SIGEV_THREAD Additional functions Device driver functions via ioctls –Registration of a timer clocks –Registration of an (IO-)event Thread specific sigevent notification (the kernel is able to do it) fast message queues

13. ASQF Fachgruppe Automation/ Automation Day 13 Testscenario: Measurement with getuid(), since getpid() is handled inside glibc Measurement with time stamp counter (94 clocks resp. 12/5 clocks) glibc-2.3.6 generated with either int 80 or sysenter system entry (…/…) Cel. 2,8 GHz 1000 clocks = 357 ns Athlon 2 GHz 1000 clocks = 500 ns 343/147 Linux 2.6.15.4 original ipipe original 1124/609 1397/914 Linux Domain Realtime Domain Linux + Ipipe patch + AuD patch + AuD optimized + AuD patch + AuD optimized 1260/767 987/494 289/199 383/257 Realtime capable Linux: System call overhead in clocks 317/255 466/338 987/494 1155/641 /185

14. ASQF Fachgruppe Automation/ Automation Day 14 high priority thread sem_post()/mq_send() sem_wait()/mq_receive() up helper thread of lowest priority down Low priority thread must have a chance to proceed to sem_wait() / mq_receive() sem_wait()/mq_receive() sem_post()/mq_send() low priority thread Realtime capable Linux: Performance inter-thread communication Testscenario: Two threads communicating either via semaphores or through POSIX message queues Note: Times for down are always longer than for up, since an additional sem_wait() / mq_receive()-call is included.

15. ASQF Fachgruppe Automation/ Automation Day 15 Realtime Domain Linux Domain up down 2667 2982 5061 4652 3140 3675 3844 4211 5292 6437 12254 13766 850 1463 1147 1625 2123 4628 up down up Realtime capable Linux: Interthread communication in clocks Celeron 2,8 GHz 1000 clocks = 357 ns Athlon 64 2,0 GHz 1000 clocks = 500 ns up down up down 2083 2852 2699 3323 4240 6235 up down Semaphore Message Queue Semaphore Message Queue 8422 10522 8453 21494 6322 8978 Times for Realtime  Linux first switch to interrupted Linux thread afterwards switch to woken up Linux thread (message queue) semaphores (futex) need an additional proxy thread for futex_wake optimized 947 1715 1401 5150 1789 2460 2676 3658 3346 7373 3964 4993 9377 12286 8663 20027 7120 9650 System call entry: int 80

16. ASQF Fachgruppe Automation/ Automation Day 16 Realtime Domain Linux Domain up down 2268 2615 4547 3980 3801 3361 3539 3780 4715 5377 15992 13304 744 1279 1071 1519 2075 4374 up down up Realtime capable Linux: Interthread communication in clocks Celeron 2,8 GHz 1000 clocks = 357 ns Athlon 64 2,0 GHz 1000 clocks = 500 ns up down up down 1963 2509 2530 3241 3912 6174 up down Semaphore Message Queue Semaphore Message Queue 6721 8390 7266 19825 4715 7004 optimized 836 1417 1297 4943 1647 2215 2050 2771 2827 6666 3335 4197 7655 10364 7130 18575 5534 8106 System call entry: sysenter Times for Realtime  Linux first switch to interrupted Linux thread afterwards switch to woken up Linux thread (message queue) semaphores (futex) need an additional proxy thread for futex_wake

17. ASQF Fachgruppe Automation/ Automation Day 17 ISR response thread response min max 525 1785 2688 284 3287 16149 17441 3276 min max 1024 1010 101 9446 Athlon64 2,0 GHz 1000 clocks = 500 ns Celeron 2,8 GHz 1000 clocks = 357 ns Realtime capable Linux: Interrupt response time in clocks Test scenario: An application thread issues via a driver an ipipe soft interrupt The ISR resumes via send_event another application thread ( waiting with sigtimedwait()) ISR frequency varied between 100 µsec and 1 sec System load generated via GUI (Eclipse-Start, file transfer, etc.) The minimal figures are of interest in several respect: for high interrupt frequencies representative for the average and therefore the load typical for path length, and therefore useful for scalability projections What can be achieved with less powerful HW (e.g. SOC with mediocre memory interface )? for a dedicated processor of a multicore system (private TLB, L1-Cache, L2-Cache) the guaranteed worst case time comes closer to the minimal figure. 2241382 1264 9240

18. ASQF Fachgruppe Automation/ Automation Day 18 Realtime capable Linux: Positioning to RT_PREEMPT The kernel implementation is transparent to application programs –An application binary for a dual domain system runs without any changes on a Linux only system and therefore also on a RT_PREEMPT system Possible coexistence scenarios with RT_PREEMPT –RT_PREEMPT as an alternative if no extremely strong realtime requirements –An additional realtime domain for extremely strong requirements for response time and thread communication, if demonstration is required (proof) tiny kernel, short paths  large complex kernel, lengthy paths, nested PI mutexes, RCU locks Commonalities with RT_PREEMPT – The same modifications of the glibc are needed. – The same kernel enhancements (registering a clock, realtime capable POSIX message queues) are needed

19. ASQF Fachgruppe Automation/ Automation Day 19 Realtime capable Linux: Conclusion The POSIX-Realtime-Extensions need some fine tuning –Minimal extensions to the API definition improve usability and allow for best possible performance Realtime under Linux can be provided transparently –The performance figures are comparable to specialized realtime kernels Ipipe is a sustainable base –Small optimizations improve system call handling and interrupt response