Real-Time Linux Pacific Northwest National Labs Rick Lindsley 4/6/09

Tux on vacation during the 2.6.29 release Tuz is taking his place http://www.phoronix.com/scan.php?page=news_item&px=NzE1MA

Agenda A Brief History of the RT Project IBM, distros, RT community resources What is Real-Time Mainline Linux Real-Time Features Real-Time Kernel Characteristics priority-driven, deterministic Real-Time Kernel Features scheduling, threaded interrupts, locking, time infrastructure New stuff to administer RT Some Test Results

Linux and Real-Time - A History IBM - DDG 1000 Project Requirements 2005 - Raytheon Defense application with specific requirements Needed Java too Mainline real-time support not where we needed IBM worked with the mainline Linux kernel community time, locking, scheduling, hrtimers, interrupt threads, preemptible RCU, ..., ongoing work IBM had to roll our own -> RHEL4.U2 + real-time patchset release kernel, glibc patches, userspace bits, BIOS (SMI Remediation)

SMI Remediation 50ms spikes in latency! Yikes! (periodic observed in e326m) LTC worked with System X BIOS/Firmware folks System Management Interface (SMI) working modified IBM Premium Real-Time Mode OS now handles non-fatal SMIs (user space logging of messages) no CPU thermal throttling Real-Time does not like devices probing/polling CPU for long! SMIs addressed in LS21, HS21XM (SMIs only on errors) ServerProven testing an issue Advantage over systems with sys mgt procs - incl. Thinkpads Need SMI – free consistent option in future platforms

Real-Time Community and Distro Resources Ingo Molnar --- > Stephen Rostedt (RT patchset maintainer) www.kernel.org.pub/linux/kernel/projects/rt linux-rt-users@vger.kernel.org OFTC - #linux-rt Distros Novell's SuSE Linux Enterprise Real Time (SLERT/ SLES 10 SP2 based) Red Hat's Enterprise MRG Realtime Linux 1.0 (RHEL5.1+ based) Both above supported on IBM Blades (HS21XM, LS21 - ServerProven) Tests / Wiki Linux Test Project (LTP) - realtime.sh (realtime tests, microbenchmarks) We maintain external community wiki: http://rt.wiki.kernel.org

What is Real Time?

Normal Non-Real-Time Compute Model Excellent throughput but with wide variability Average times may vary No prioritisation Favors raw throughput, latencies uncontrolled Acceptable non-real-time behavior: 100,000 transactions in a minute 99,999 at 5ms and 1 at 2 seconds Unacceptable for real-time systems

What Do We Mean by Real-Time Computing? Like driving in the HOV lane High priority: predictable, deterministic Its not about going faster. In fact in computing it could be somewhat slower ... but with bounded response times 50,000 transactions in a minute All 50,000 in less than 6ms As opposed to average near 5ms for non-real-time example Determinism = consistent response time = low latency When “close enough” is not “good enough” When life and limb matters or when a SLA failure could cost a lot of money

It used to be... Real-Time systems used dedicated HW (often embedded) Real-Time systems used custom OSs Real-Time systems had a stripped down environment Real-Time applications were written in specific languages Real-Time systems were not general purpose platforms

Enterprise Real-Time Want low latency Want priority-driven environment Want fast response times, but not fatal Tolerate some jitter Want general-purpose environment LAMP stack Standard programming languages (Java, ...) Usual set of tools Run on general purpose HW (servers, ...) shared environment Financial trading, industrial, defense, ...

Real-Time Features in mainline and the patchset POSIX.4 Timers Asynchronous I/O POSIX.4 Real-Time Scheduler CFS is same in RT and mainline hrtimers (high resolution timers) chrt Manipulate real-time attributes of a process (class, priority) CONFIG_PREEMPT Real-Time Patchset CONFIG_PREEMPT_RT locking, interrupt handling

Sources of Jitter Jitter - variation in execution times lack of determinism Sources of jitter for application threads: Scheduling - CPU contention Lock contention Interrupts - Hardware, Softirq (periodic, asynchronous) Variance in memory allocation times, etc... Device/Network/external factors system mgt events, throttling, ...

Real-Time Kernel Characteristics - Priority Driven Traditional OSs are optimized for performance, overall throughput BUT on RT we: Always execute the N highest priority threads on an N-CPU box SCHED_FIFO not limited by timeslices (will not relinquish the CPU!) if a higher priority thread becomes runnable, lower priority thread is migrated or suspended this can cause severe cache thrashing inefficient overall (might yield almost immediately, ...) potential deadlock, need care while programming, administering Sacrifice system throughput for real-time priority ordered hierarchy Trade-off benefits high priority tasks at the expense of low priority tasks

Real-Time Kernel Goals - Deterministic maximum latency bound want to guarantee operations will take “at most 30 seconds” no matter what happens (all paths) Better to have 300 operations that take 20ms rather than 299 operations that take 15ms and 1 operation that takes 60ms. Real-Time applications need bounded latencies Sacrifice throughput, performance for determinism Trade-off benefits high priority tasks at the expense of low priority tasks

Real-Time Kernel Features - Scheduling CFS Scheduler - default scheduling algorithm SCHEDULING POLICY SCHED_RR, SCHED_FIFO, SCHED_OTHER SCHED_FIFO not limited by timeslices (will not relinquish the CPU!) SCHED_RR has priority but is timesliced SCHED_OTHER does not have a real-time priority (time-sliced, priority 0) SCHEDULING PRIORITY SCHED_FIFO, SCHED_RR: 1 - 100 Java : 1 - 89 Critical Kernel Threads : 90 +

Real-Time Kernel Features - Threaded Interrupts HW and SW Interrupts are SCHED_FIFO kernel threads Work deferred from HW interrupt to softirq (now a kernel thread) except timer interrupt Priorities are distributed across priority of 30 - 99 can be tricky if large number of high priority threads manipulating these priorities is HIGH ART need to ensure critical kernel activities not blocked Applications used to get interrupts which ran till they completed caused a lot of variance in the time it took to run now kernel activity competes for time with high priority real-time processes allows high priority real-time tasks to run with fewer interrupts

Real-Time Kernel Features - cont'd Can't spin on RT (need to be preempted) Some spin locks are now mutexes futexes - priority inheritance supporting userspace mutexes Preemptive RCU allows rcu locks to be preempted High resolution timers Disable CPU throttling for thermal events (fatal range cause shutdown) Recommendations: turn off irqbalanced disable all other services classify real-time tasks

What can users do? On mainline Linux (non-RT) real-time priority scheduling chrt to set scheduling class and priority taskset to pin interrupts, threads Run on RT (tuning, priorities) Run on RT with application code changes e.g. mlockall, etc to pin memory

New stuff to administer RT - Red Hat MRG /usr/sbin/rtctl /etc/rtgroups manage groups of processes/threads at a time Iinitialize / set real-time attributes (policy, priority) realtime group pam perms to set real-time priority, policy (non-root) /etc/init.d/ibm-prtm - SMI handling daemon /usr/sbin/ibm-prtmd ftrace Tuna - configuring, monitoring, latency tracing tool debugging & tracing kernels

New stuff to administer RT - SLERT /etc/init.d/ibmconfig /etc/init.d/set_kthread_prio set priority / policy of kernel threads /etc/set_kthread_prio.conf /sbin/set_kthread_prio /etc/init.d/ibm-prtm - SMI handling daemon /usr/sbin/ibm-prtmd realtime group pam perms to set real-time priority, policy (non-root) Night Hawk tools (Novell looking to adopt TUNA)

Mainline idle/loaded - async-handler Scatterplot

RT idle/loaded - async-handler Scatterplot

Questions?

IBM Real-Time Linux Releases To Raytheon DDG 1000 Project First alpha drop (Jan 31, 2006; RHEL 4.2) SuSE Enterprise Real Time (SLES10; Sept 06; Concurrent ) R1 (Oct 31, 2006; RHEL 4.2) R1-SR1 (Jan 31, 2007; RHEL 4.2) R1-SR2 (Sept 28, 2007; RHEL 4.2, 2.6.16-12 based RT patchset) R1-SR3 (Jan 30, 2008; RHEL 5.1, 2.6.21-4 based RT patchset) SuSE Linux Enterprise Real Time (SLES10 SP2; May 28, 08) 2.6.22.19 Red Hat Enterprise Real Time Linux (RHEL5 MRG; June 18, 08) R2 (Q3, 2008; RHEL5.2-MRG, 2.6.24-7 based RT patchset) To Raytheon Themis Project: R1-SR3 (Q3, ported to Themis HW)