1. Real-Time Linux Pacific Northwest National Labs Rick Lindsley 4/6/09

2. Tux on vacation during the 2.6.29 release Tuz is taking his place

3. Agenda A Brief History of the RT Project
IBM, distros, RT community resources
What is Real-Time
Mainline Linux Real-Time Features
Real-Time Kernel Characteristics
priority-driven, deterministic
Real-Time Kernel Features
scheduling, threaded interrupts, locking, time infrastructure
New stuff to administer RT
Some Test Results

4. Linux and Real-Time - A History
IBM - DDG 1000 Project Requirements
Raytheon
Defense application with specific requirements
Needed Java too
Mainline real-time support not where we needed
IBM worked with the mainline Linux kernel community
time, locking, scheduling, hrtimers, interrupt threads, preemptible RCU, ..., ongoing work
IBM had to roll our own -> RHEL4.U2 + real-time patchset release
kernel, glibc patches, userspace bits, BIOS (SMI Remediation)

5. SMI Remediation
50ms spikes in latency! Yikes! (periodic observed in e326m)
LTC worked with System X BIOS/Firmware folks
System Management Interface (SMI) working modified
IBM Premium Real-Time Mode
OS now handles non-fatal SMIs (user space logging of messages)
no CPU thermal throttling
Real-Time does not like devices probing/polling CPU for long!
SMIs addressed in LS21, HS21XM (SMIs only on errors)
ServerProven testing an issue
Advantage over systems with sys mgt procs - incl. Thinkpads
Need SMI – free consistent option in future platforms

6. Real-Time Community and Distro Resources
Ingo Molnar --- > Stephen Rostedt (RT patchset maintainer)
OFTC - #linux-rt
Distros
Novell's SuSE Linux Enterprise Real Time (SLERT/ SLES 10 SP2 based)
Red Hat's Enterprise MRG Realtime Linux 1.0 (RHEL5.1+ based)
Both above supported on IBM Blades (HS21XM, LS21 - ServerProven)
Tests / Wiki
Linux Test Project (LTP) - realtime.sh (realtime tests, microbenchmarks)
We maintain external community wiki:

7. What is Real Time?

8. Normal Non-Real-Time Compute Model
Excellent throughput but with wide variability
Average times may vary
No prioritisation
Favors raw throughput, latencies uncontrolled
Acceptable non-real-time behavior:
100,000 transactions in a minute
99,999 at 5ms and 1 at 2 seconds
Unacceptable for real-time systems

9. What Do We Mean by Real-Time Computing?
Like driving in the HOV lane
High priority: predictable, deterministic
Its not about going faster. In fact in computing it could be somewhat slower ... but with bounded response times
50,000 transactions in a minute
All 50,000 in less than 6ms
As opposed to average near 5ms for non-real-time example
Determinism = consistent response time = low latency
When “close enough” is not “good enough”
When life and limb matters or when a SLA failure could cost a lot of money

10. It used to be... Real-Time systems used dedicated HW (often embedded)
Real-Time systems used custom OSs
Real-Time systems had a stripped down environment
Real-Time applications were written in specific languages
Real-Time systems were not general purpose platforms

11. Enterprise Real-Time Want low latency Want priority-driven environment
Want fast response times, but not fatal
Tolerate some jitter
Want general-purpose environment
LAMP stack
Standard programming languages (Java, ...)
Usual set of tools
Run on general purpose HW (servers, ...)
shared environment
Financial trading, industrial, defense, ...

12. Real-Time Features in mainline and the patchset
POSIX.4 Timers
Asynchronous I/O
POSIX.4 Real-Time Scheduler
CFS is same in RT and mainline
hrtimers (high resolution timers)
chrt
Manipulate real-time attributes of a process (class, priority)
CONFIG_PREEMPT
Real-Time Patchset
CONFIG_PREEMPT_RT
locking, interrupt handling

13. Sources of Jitter Jitter - variation in execution times
lack of determinism
Sources of jitter for application threads:
Scheduling - CPU contention
Lock contention
Interrupts - Hardware, Softirq (periodic, asynchronous)
Variance in memory allocation times, etc...
Device/Network/external factors
system mgt events, throttling, ...

14. Real-Time Kernel Characteristics - Priority Driven
Traditional OSs are optimized for performance, overall throughput
BUT on RT we:
Always execute the N highest priority threads on an N-CPU box
SCHED_FIFO not limited by timeslices (will not relinquish the CPU!)
if a higher priority thread becomes runnable, lower priority thread is migrated or suspended
this can cause severe cache thrashing
inefficient overall (might yield almost immediately, ...)
potential deadlock, need care while programming, administering
Sacrifice system throughput for real-time priority ordered hierarchy
Trade-off benefits high priority tasks at the expense of low priority tasks

15. Real-Time Kernel Goals - Deterministic
maximum latency bound
want to guarantee operations will take “at most 30 seconds”
no matter what happens (all paths)
Better to have 300 operations that take 20ms rather than 299 operations that take 15ms and 1 operation that takes 60ms.
Real-Time applications need bounded latencies
Sacrifice throughput, performance for determinism
Trade-off benefits high priority tasks at the expense of low priority tasks

16. Real-Time Kernel Features - Scheduling
CFS Scheduler - default scheduling algorithm
SCHEDULING POLICY
SCHED_RR, SCHED_FIFO, SCHED_OTHER
SCHED_FIFO not limited by timeslices (will not relinquish the CPU!)
SCHED_RR has priority but is timesliced
SCHED_OTHER does not have a real-time priority (time-sliced, priority 0)
SCHEDULING PRIORITY
SCHED_FIFO, SCHED_RR:
Java :
Critical Kernel Threads : 90 +

17. Real-Time Kernel Features - Threaded Interrupts
HW and SW Interrupts are SCHED_FIFO kernel threads
Work deferred from HW interrupt to softirq (now a kernel thread)
except timer interrupt
Priorities are distributed across priority of
can be tricky if large number of high priority threads
manipulating these priorities is HIGH ART
need to ensure critical kernel activities not blocked
Applications used to get interrupts which ran till they completed
caused a lot of variance in the time it took to run
now kernel activity competes for time with high priority real-time processes
allows high priority real-time tasks to run with fewer interrupts

18. Real-Time Kernel Features - cont'd
Can't spin on RT (need to be preempted)
Some spin locks are now mutexes
futexes - priority inheritance supporting userspace mutexes
Preemptive RCU
allows rcu locks to be preempted
High resolution timers
Disable CPU throttling for thermal events (fatal range cause shutdown)
Recommendations:
turn off irqbalanced
disable all other services
classify real-time tasks

19. What can users do? On mainline Linux (non-RT)
real-time priority scheduling
chrt to set scheduling class and priority
taskset to pin interrupts, threads
Run on RT (tuning, priorities)
Run on RT with application code changes
e.g. mlockall, etc to pin memory

20. New stuff to administer RT - Red Hat MRG
/usr/sbin/rtctl
/etc/rtgroups
manage groups of processes/threads at a time
Iinitialize / set real-time attributes (policy, priority)
realtime group
pam perms to set real-time priority, policy (non-root)
/etc/init.d/ibm-prtm - SMI handling daemon /usr/sbin/ibm-prtmd
ftrace
Tuna - configuring, monitoring, latency tracing tool
debugging & tracing kernels

21. New stuff to administer RT - SLERT
/etc/init.d/ibmconfig
/etc/init.d/set_kthread_prio
set priority / policy of kernel threads
/etc/set_kthread_prio.conf
/sbin/set_kthread_prio
/etc/init.d/ibm-prtm - SMI handling daemon /usr/sbin/ibm-prtmd
realtime group
pam perms to set real-time priority, policy (non-root)
Night Hawk tools (Novell looking to adopt TUNA)

22. Mainline idle/loaded - async-handler
Scatterplot

23. RT idle/loaded - async-handler
Scatterplot

24. Questions?

25. IBM Real-Time Linux Releases
To Raytheon DDG 1000 Project
First alpha drop (Jan 31, 2006; RHEL 4.2)
SuSE Enterprise Real Time (SLES10; Sept 06; Concurrent )
R1 (Oct 31, 2006; RHEL 4.2)
R1-SR1 (Jan 31, 2007; RHEL 4.2)
R1-SR2 (Sept 28, 2007; RHEL 4.2, based RT patchset)
R1-SR3 (Jan 30, 2008; RHEL 5.1, based RT patchset)
SuSE Linux Enterprise Real Time (SLES10 SP2; May 28, 08)
Red Hat Enterprise Real Time Linux (RHEL5 MRG; June 18, 08)
R2 (Q3, 2008; RHEL5.2-MRG, based RT patchset)
To Raytheon Themis Project: R1-SR3 (Q3, ported to Themis HW)