Real Time Linux Who Needs It? Presented by: Steven Rostedt

What is Real Time? ● Real Time Clock? ● Real Time Video? ● Real Time Delivery? ● Real Time Operating System? ● Real Time Presentation Translation?

What is Real Time? ● Real Time Clock? ● Real Time Video? ● Real Time Delivery? ● Real Time Operating System? ● Real Time Presentation Translation?

What is a Real Time Operating System? ● Faster? ● Real Time does not mean fast! NO ! ● What does a RTOS give us? – Deterministic

What does being Deterministic Give Us? ● Repeatability ● Reliable Results ● Known Worst Cases ● Known Reaction Times

Real Time Hard vs Soft ● Hard Real Time – Mathematically proven ● Given a set of inputs, always gives the same output at the same time – Must be small (~5000 lines of code) – Bounded Latencies ● Soft Real Time – Can deal with outliers – Tries to be reliable – Unbounded Latencies

Examples of Users of Hard Real Time ● Airplane Engine Controls ● Nuclear Power Plants ● Mars Lander ● Shuttle

Examples of Soft Real Time Users ● Video Systems ● Video Games ● Some Communication Systems Vanilla Linux is a Soft Real Time System

Real Time Linux ● Is it a Soft Real Time System? – Does not allow for outliers ● Those few cases that happen outside the Maximum – Has Bounded Latency ● Is it a Hard Real Time System? – Not a small system – Can not be mathematically proven ● What is Real Time Linux then? – Hard Real Time Designed

Real Time Linux ● Can not be mathematically proven – It is too big ● Tries very hard to bound all latencies – Any time an unexpected latency is found, it is considered a bug ● The Design of RT Linux follows the same as a Hard Real Time OS design

Who Uses Real Time Linux? ● Financial Industries – code already too big for true Hard Real Time – Can not miss deadlines ● Audio Recordings – A latency will ruin a good performance ● Can not fail but... – Nobody dies if it does

What RT Linux Gave to Vanilla Linux ● High Resolution Timers ● Generic Interrupt Design – Cleaned up the code ● Preemptible RCU Locks ● Real Time Scheduler ● Priority Inheritance (on user space locks) ● Ftrace - Linux Kernel Tracer

What Is Left? ● Interrupts as Threads – Helps against Interrupt Inversion Latencies ● Spin locks to sleeping mutexes – Interrupts do not need to be disabled – Helps against reaction time Latencies ● Task wake up ● Timer response ● Priority Inheritance on kernel locks – Helps against Priority Inversion Latencies

What is Latency? ● The time between when an event is expected to happen, and the time it happens ● Causes of Latency – Interrupts disabled – Interrupt executing – Slow scheduler

Interrupts

Interrupts Disabled

Interrupt Inversion

Threaded Interrupts

# ps ax |grep IRQ 42 ? S< 0:00 [IRQ-9] 56 ? S< 0:00 [IRQ-12] 57 ? S< 0:00 [IRQ-1] 59 ? S< 0:00 [IRQ-8] 67 ? S< 0:00 [IRQ-19] 74 ? S< 0:00 [IRQ-16] 75 ? S< 0:00 [IRQ-17] 76 ? S< 0:00 [IRQ-18] 138 ? S< 0:00 [IRQ-28] 447 ? S< 0:00 [IRQ-3] 920 ? S< 0:00 [IRQ-14] 922 ? S< 0:00 [IRQ-15] 1935 ? S< 0:00 [IRQ-29] 1937 ? S< 0:00 [IRQ-30]

Latency ● Latency always happens – It takes code to execute ● Bounded latency – We know the worst case (the biggest latency) ● Unbounded Latency – No idea of how long it will last – Classic case ● Priority Inversion

Priority Inversion

Priority Inheritance

Vanilla Linux # cyclictest -t1 -p95 -n -i1000 -o10 -v |./oscilloscope-cmd.py & # hackbench 50

RT Linux rt22 # cyclictest -t1 -p95 -n -i1000 -o10 -v |./oscilloscope-cmd.py & # hackbench 50

Latency Tracer Vanilla # tracer: irqsoff # irqsoff latency trace v1.1.5 on vanilla latency: 644 us, #3/3, CPU#0 | (M:preempt VP:0, KP:0, SP:0 HP:0 #P:2) | task: sh-4075 (uid:500 nice:0 policy:0 rt_prio:0) # _------=> CPU# # / _-----=> irqs-off # | / _----=> need-resched # || / _---=> hardirq/softirq # ||| / _--=> preempt-depth # |||| / # ||||| delay # cmd pid ||||| time | caller # \ / ||||| \ | / sh d... 1us!: _write_lock_irq (release_task) sh d us : _write_unlock_irq (release_task) sh d us : trace_hardirqs_on (release_task)

Latency Tracer Vanilla rt22 # tracer: irqsoff # # irqsoff latency trace v1.1.5 on rt22 # # latency: 41 us, #3/3, CPU#1 | (M:preempt VP:0, KP:0, SP:0 HP:0 #P:2) # # | task: swapper-0 (uid:0 nice:0 policy:0 rt_prio:0) # # # _------=> CPU# # / _-----=> irqs-off # | / _----=> need-resched # || / _---=> hardirq/softirq # ||| / _--=> preempt-depth # |||| / # ||||| delay # cmd pid ||||| time | caller # \ / ||||| \ | / -0 1d..1 1us+: acpi_idle_enter_bm <-cpuidle_idle_call -0 1d..1 41us : acpi_idle_enter_bm <-cpuidle_idle_call -0 1d..1 42us : stop_critical_timings <-cpuidle_idle_call

Hardware Does Matter! ● Cache and TLB misses can cause unpredictable results ● SMI - System Management Interrupt – Controlled by the BIOS – The Operating System has no control and no idea they happen ● The hardware must also be deterministic

More info This slide: rt-linux-who-needs-it.odp

Ftrace A little demo on a development kernel (not a Real Time kernel)