1. Real Time Linux Who Needs It? Presented by: Steven Rostedt srostedt@redhat.com

2. What is Real Time? ● Real Time Clock? ● Real Time Video? ● Real Time Delivery? ● Real Time Operating System? ● Real Time Presentation Translation?

3. What is Real Time? ● Real Time Clock? ● Real Time Video? ● Real Time Delivery? ● Real Time Operating System? ● Real Time Presentation Translation?

4. What is a Real Time Operating System? ● Faster? ● Real Time does not mean fast! NO ! ● What does a RTOS give us? – Deterministic

5. What does being Deterministic Give Us? ● Repeatability ● Reliable Results ● Known Worst Cases ● Known Reaction Times

6. Real Time Hard vs Soft ● Hard Real Time – Mathematically proven ● Given a set of inputs, always gives the same output at the same time – Must be small (~5000 lines of code) – Bounded Latencies ● Soft Real Time – Can deal with outliers – Tries to be reliable – Unbounded Latencies

7. Examples of Users of Hard Real Time ● Airplane Engine Controls ● Nuclear Power Plants ● Mars Lander ● Shuttle

8. Examples of Soft Real Time Users ● Video Systems ● Video Games ● Some Communication Systems Vanilla Linux is a Soft Real Time System

9. Real Time Linux ● Is it a Soft Real Time System? – Does not allow for outliers ● Those few cases that happen outside the Maximum – Has Bounded Latency ● Is it a Hard Real Time System? – Not a small system – Can not be mathematically proven ● What is Real Time Linux then? – Hard Real Time Designed

10. Real Time Linux ● Can not be mathematically proven – It is too big ● Tries very hard to bound all latencies – Any time an unexpected latency is found, it is considered a bug ● The Design of RT Linux follows the same as a Hard Real Time OS design

11. Who Uses Real Time Linux? ● Financial Industries – code already too big for true Hard Real Time – Can not miss deadlines ● Audio Recordings – A latency will ruin a good performance ● Can not fail but... – Nobody dies if it does

12. What RT Linux Gave to Vanilla Linux ● High Resolution Timers ● Generic Interrupt Design – Cleaned up the code ● Preemptible RCU Locks ● Real Time Scheduler ● Priority Inheritance (on user space locks) ● Ftrace - Linux Kernel Tracer

13. What Is Left? ● Interrupts as Threads – Helps against Interrupt Inversion Latencies ● Spin locks to sleeping mutexes – Interrupts do not need to be disabled – Helps against reaction time Latencies ● Task wake up ● Timer response ● Priority Inheritance on kernel locks – Helps against Priority Inversion Latencies

14. What is Latency? ● The time between when an event is expected to happen, and the time it happens ● Causes of Latency – Interrupts disabled – Interrupt executing – Slow scheduler

15. Interrupts

16. Interrupts Disabled

17. Interrupt Inversion

18. Threaded Interrupts

19. # ps ax |grep IRQ 42 ? S< 0:00 [IRQ-9] 56 ? S< 0:00 [IRQ-12] 57 ? S< 0:00 [IRQ-1] 59 ? S< 0:00 [IRQ-8] 67 ? S< 0:00 [IRQ-19] 74 ? S< 0:00 [IRQ-16] 75 ? S< 0:00 [IRQ-17] 76 ? S< 0:00 [IRQ-18] 138 ? S< 0:00 [IRQ-28] 447 ? S< 0:00 [IRQ-3] 920 ? S< 0:00 [IRQ-14] 922 ? S< 0:00 [IRQ-15] 1935 ? S< 0:00 [IRQ-29] 1937 ? S< 0:00 [IRQ-30]

20. Latency ● Latency always happens – It takes code to execute ● Bounded latency – We know the worst case (the biggest latency) ● Unbounded Latency – No idea of how long it will last – Classic case ● Priority Inversion

21. Priority Inversion

22. Priority Inheritance

23. Vanilla Linux 2.6.29.5 # cyclictest -t1 -p95 -n -i1000 -o10 -v |./oscilloscope-cmd.py & # hackbench 50

24. RT Linux 2.6.29.5-rt22 # cyclictest -t1 -p95 -n -i1000 -o10 -v |./oscilloscope-cmd.py & # hackbench 50

25. Latency Tracer Vanilla 2.6.29.5 # tracer: irqsoff # irqsoff latency trace v1.1.5 on 2.6.29.5-vanilla -------------------------------------------------------------------- latency: 644 us, #3/3, CPU#0 | (M:preempt VP:0, KP:0, SP:0 HP:0 #P:2) ----------------- | task: sh-4075 (uid:500 nice:0 policy:0 rt_prio:0) ----------------- # _------=> CPU# # / _-----=> irqs-off # | / _----=> need-resched # || / _---=> hardirq/softirq # ||| / _--=> preempt-depth # |||| / # ||||| delay # cmd pid ||||| time | caller # \ / ||||| \ | / sh-4075 0d... 1us!: _write_lock_irq (release_task) sh-4075 0d..1 644us : _write_unlock_irq (release_task) sh-4075 0d..1 645us : trace_hardirqs_on (release_task)

26. Latency Tracer Vanilla 2.6.29.5-rt22 # tracer: irqsoff # # irqsoff latency trace v1.1.5 on 2.6.29.5-rt22 # -------------------------------------------------------------------- # latency: 41 us, #3/3, CPU#1 | (M:preempt VP:0, KP:0, SP:0 HP:0 #P:2) # ----------------- # | task: swapper-0 (uid:0 nice:0 policy:0 rt_prio:0) # ----------------- # # _------=> CPU# # / _-----=> irqs-off # | / _----=> need-resched # || / _---=> hardirq/softirq # ||| / _--=> preempt-depth # |||| / # ||||| delay # cmd pid ||||| time | caller # \ / ||||| \ | / -0 1d..1 1us+: acpi_idle_enter_bm <-cpuidle_idle_call -0 1d..1 41us : acpi_idle_enter_bm <-cpuidle_idle_call -0 1d..1 42us : stop_critical_timings <-cpuidle_idle_call

27. Hardware Does Matter! ● Cache and TLB misses can cause unpredictable results ● SMI - System Management Interrupt – Controlled by the BIOS – The Operating System has no control and no idea they happen ● The hardware must also be deterministic

28. More info http://rt.wiki.kernel.org/index.php/Main_Page This slide: http://people.redhat.com/srostedt/ rt-linux-who-needs-it.odp

29. Ftrace A little demo on a development kernel (not a Real Time kernel)