Fall 2013 SILICON VALLEY UNIVERSITY CONFIDENTIAL 1 Introduction to Embedded Systems Dr. Jerry Shiao, Silicon Valley University

SILICON VALLEY UNIVERSITY CONFIDENTIAL 2 Spring 2014 Section 14 Real-Time Patch Linux 2.6 Real-Time Modifications  Linux Community wants to convert the Linux Kernel into true Real-Time Operation System (RTOS) without a microkernel.  Linux RTOS Needs: Interrupt Service Routines must NOT unconditionally preempt any process running on the CPU. Critical Sections as needed by processes. Unbounded Priority Inversion not allowed.  Ingo Molnar, approached the Linux RTOS problem by changing the Linux system with Real-Time features that would improve user’s experience. High-Resolution timers. Kernel lock validation. Generic interrupts for all architectures. Robust futexes. Priority Inheritance.

SILICON VALLEY UNIVERSITY CONFIDENTIAL 3 Spring 2014 Section 14 Real-Time Patch Linux 2.6 Real-Time Modifications  Interrupt as Threads  Interrupt Service Routine (ISR) has highest priority events in the Operating System and only another ISR can interrupt an ISR, but only if interrupts are enable.  ISR pushes work to Kernel Thread, Tasklet, or Softirq. Kernel Thread: Work Queue, queues up work from ISR that will be executed by worker kernel thread, keventd. Works in a FIFO order. Softirq: Service routine performed after ISR returns and before interrupted process is resumed. Kernel Thread, ksoftirqd, executes queued softirqs.  Same softirq run simultaneously on different CPUs.  Reentrant: Need to use locks to protect against concurrent accesses. Tasklet: Similar to softirq, performed after ISR returns and before resuming interrupted process.  Same tasklet cannot run simultaneously on different CPUs.  Not reentrant.  Implemented as a softirq

SILICON VALLEY UNIVERSITY CONFIDENTIAL 4 Spring 2014 Section 14 Real-Time Patch Linux 2.6 Real-Time Modifications  Interrupt Signal Will Always Preempts CPU and executes the ISR.  Interrupt Inversion: ISR takes time away from higher priority process to handle lower priority process.  Lower Interrupt Latency and Interrupt Inversion: Transform all ISRs, softirqs, and tasklets into Kernel Threads.  Converts Interrupt Handlers into Kernel Threads. Masking of the Interrupt Line. Wake Interrupt Service Thread (one for each type of interrupt). Interrupt Service Thread registered by the Device Driver.  Original Process Higher Priority, than Interrupt Service Thread will not Preempt.

SILICON VALLEY UNIVERSITY CONFIDENTIAL 5 Spring 2014 Section 14 Real-Time Patch Linux 2.6 Real-Time Modifications Interrupt Inversion: Interrupt Service Routine takes time away from a high priority process in order to perform work that is of lower priority. The ISR is lower priority than the process that is currently running. Interrupt Latency: Time before the high priority process gets the CPU back.

SILICON VALLEY UNIVERSITY CONFIDENTIAL 6 Spring 2014 Section 14 Real-Time Patch Linux 2.6 Real-Time Modifications Real Time Patch: ISR wakes up a Kernel Thread that will run the function registered by the Device Driver. Interrupt Latency: Shortens the time of Interrupt Inversion to a bare minimum.

SILICON VALLEY UNIVERSITY CONFIDENTIAL 7 Spring 2014 Section 14 Real-Time Patch Linux 2.6 Real-Time Modifications Real Time Patch: The Threads that are created to service the interrupts are named “IRQ_n”, where “n” is the number of the Interrupt Vector. Interrupt Thread for Interrupt Vector 14, the IDE disk driver.

SILICON VALLEY UNIVERSITY CONFIDENTIAL 8 Spring 2014 Section 14 Real-Time Patch Linux 2.6 Real-Time Modifications ps command shows the pid = 1722 has priority set to 80. chrt command sets the Real- Time scheduling attributes of the process ( pid ). -p: Existing pid = 1722, set the priority = 80. -f: Set scheduling policy to SCHED_FIFO.

SILICON VALLEY UNIVERSITY CONFIDENTIAL 9 Spring 2014 Section 14 Real-Time Patch Linux 2.6 Real-Time Modifications  Interrupts and CPU Affinities. Each thread kept on the CPU where the thread started execution in order to prevent expensive cache flushes.  IRQ Thread determined by CPU Affinity of Interrupt. # cat /proc/interrupts | grep ide0 14: IO-APIC-edge ide0 # ps ax | grep IRQ ? S< 0:00 [IRQ-14] # cat /proc/irq/14/smp_affinity 1 # echo 2 > /proc/irq/14/smp_affinity # cat /proc/irq/14/smp_affinity 2 Get the Interrupt Vector for the IDE 0 Controller ( 14 ). Get the Process ID of ISR Thread 14. Get the CPU Affinity of IRQ Interrupt Vector 14 ( CPU 0 ). Set the CPU Affinity of IRQ Interrupt Vector 14 ( CPU 1 ).

SILICON VALLEY UNIVERSITY CONFIDENTIAL 10 Spring 2014 Section 14 Real-Time Patch Linux 2.6 Real-Time Modifications  IRQ Thread determined by CPU Affinity of Interrupt. # taskset -p 790 pid 790's current affinity mask: 2 # echo 1 > /proc/irq/14/smp_affinity # taskset -p 790 pid 790's current affinity mask: 2 # ls -lR / > /dev/null # taskset -p 790 pid 790's current affinity mask: 1 Get IRQ Thread’s CPU Affinity. The IRQ Thread’s CPU Affinity does not change until the interrupt is triggered. Trigger some IDE interrupts. After the interrupts, the IRQ Thread’s Affinity follows the IRQ Affinity. Set the CPU Affinity of IRQ Interrupt Vector 14 ( CPU 0 ).

SILICON VALLEY UNIVERSITY CONFIDENTIAL 11 Spring 2014 Section 14 Real-Time Patch Linux 2.6 Real-Time Modifications Softirqs As Threads. Softirq runs with interrupts enabled. Possible Preempt higher priority Kernel Threads. Make ALL softirqs run under ksoftirqd Thread.  Problem: ALL softirqs grouped into one priority (the ksoftirqd Thread).  Solution: Softirq Thread is created for each softirq routine. Softirq Thread bound to a CPU. Softirq Thread has name: softirq-name/n “n” is the CPU where Softirq Thread is set to run.

SILICON VALLEY UNIVERSITY CONFIDENTIAL 12 Spring 2014 Section 14 Real-Time Patch Linux 2.6 Real-Time Modifications Softirqs As Threads. $ ps -e -o pid,rtprio,comm | grep sirq 5 50 sirq-high/ sirq-timer/ sirq-net-tx/ 8 50 sirq-net-rx/ 9 50 sirq-block/ sirq-tasklet sirq-sched/ sirq-hrtimer sirq-rcu/ sirq-high/ sirq-timer/ sirq-net-tx/ sirq-net-rx/ sirq-block/ sirq-tasklet sirq-sched/ sirq-hrtimer sirq-rcu/1 Softirq Thread has name: softirq-name/n “n” is the CPU where Softirq Thread is set to run. Priority of the softirqs are set at 50 by default.

SILICON VALLEY UNIVERSITY CONFIDENTIAL 13 Spring 2014 Section 14 Real-Time Patch Linux 2.6 Real-Time Modifications Softirq Timer Thread (softirq-hrtimer) Control Timing Events:  Crucial to be executed on time. timer_create POSIX System Call:  Registers function to be called by hrtimer subsystem. High-priority process MUST be lower priority than softirq-hrtimer thread.  If not, high-priority process will NOT release CPU and softirq-hrtimer thread will starve. Dynamic softirq-hrtimer thread: Automatically change its priority to the high-priority process that requested the timer.

SILICON VALLEY UNIVERSITY CONFIDENTIAL 14 Spring 2014 Section 14 Real-Time Patch Linux 2.6 Real-Time Modifications  Priority Inheritance: Low-priority task has priority raised until completes critical section (releases shared resource). Nested resource locks can lead to deadlock.  Avoid deadlock by allowing each task to own one shared resource.  Do not allow nested locks. Overhead raising priority and then lowering priority. Task M Task L Normal Execution Critical Section Execution -Task M request resource. -Task L priority raised. Executes critical section. -Task L Completes critical section. Priority lowered. -Task M receives resource. Executes critical section.

SILICON VALLEY UNIVERSITY CONFIDENTIAL 15 Spring 2014 Section 14 Real-Time Patch Linux 2.6 Real-Time Modifications Linux Configuration  make menuconfig  Preemption Mode  No Forced Preemption (Server)  Linux Kernel runs in 2.4 Kernel Mode. No Preemption of the Kernel when CPU is in Kernel Mode. High-Priority Process MUST wait until current process context switch back to User Mode or blocks on I/O.  Low Overhead: Low Context Switching Less scheduling, more time for process to run on CPU. Cache is valid. Translation Lookaside Buffer (TLB) is valid. Large batch processes, need throughput, and run serially.  Disadvantages: Low reaction time. Negative affect on application requiring low latencies and user interaction.

SILICON VALLEY UNIVERSITY CONFIDENTIAL 16 Spring 2014 Section 14 Real-Time Patch Linux 2.6 Real-Time Modifications Linux Configuration  Kernel code has points of voluntary preemption, where preemption is safe.  make menuconfig  Preemption Mode  Voluntary Kernel Preemption Kernel cannot call schedule( ) when interrupts are disabled. “might_sleep” function checks for interrupts are disabled or spin lock is held. Place “might_sleep” function in Kernel process handling system call. The Kernel process will call schedule( ) when condition is valid and allow higher priority process to get the CPU.

SILICON VALLEY UNIVERSITY CONFIDENTIAL 17 Spring 2014 Section 14 Real-Time Patch Linux 2.6 Real-Time Modifications  Linux originally designed around fairness, sharing resources among its uses and to perform under sustained load conditions. General purpose complexity too unpredictable for real-time.  Preemption Linux 2.6 system call was preemptable. A process forced to release CPU.  Real Time Scheduler Linux 2.6 scheduler determined time slice for each process and which process to run based on runqueue per priority level. Active and Expired priority arrays.  High Resolution POSIX Timers. Timer hardware removed timer update dependency on Jiffies.  Minimize execution of interrupt critical sections (interrupt latency).

SILICON VALLEY UNIVERSITY CONFIDENTIAL Spring 2014 Section 14 Real-Time Patch Real-Time Patch To Native Linux Kernel  Linux Kernel in Real-Time Environment: Preemption Improvements.  Reduce time in non-preemptible code (interrupt context) to minimize interrupt latency and scheduling latency.  Low latency patches in Kernel, add points where Kernel thread relinquishes non-preemption. Unbounded Priority Inversion avoided Normal Execution Critical Section Execution Unbounded Priority Inversion Task H Task M Task L Priority

SILICON VALLEY UNIVERSITY CONFIDENTIAL 19 Spring 2014 Section 14 Real-Time Patch Real-Time Patch To Native Linux Kernel  Priority Ceiling: Shared resource has predefined priority ceiling. Task acquires shared resource, task priority temporarily raised to priority ceiling. Prevents deadlocks by stopping nested locks. Static analysis to determine priority ceiling of each shared resource. Every task must be known. Task H Task M Task L Normal Execution Critical Section Execution -Task L priority raised to priority ceiling. -Executes critical section. -Task H request resource. Priority raised to priority ceiling. -Executes critical section.

SILICON VALLEY UNIVERSITY CONFIDENTIAL 20 Spring 2014 Section 14 Real-Time Patch Real-Time Patch To Native Linux Kernel  Linux Kernel in Real-Time Environment: Handle IRQs as Threads.  Lowers latency caused by softirqs and tasklets.  Softirqs preempts all threads, including kernel.  Default priority IRQ threads (PID = IRQ_n, n is interrupt vector) is 50.  chrt utility used to modify the priority of the ISR thread.  Interrupts can be prioritized, even though hardware does not support prioritized interrupts.  User process can be prioritized higher than IRQ thread.  CPU affinity of IRQ thread based on CPU affinity of the interrupt. Prevents expensive cache flushes.

SILICON VALLEY UNIVERSITY CONFIDENTIAL 21 Spring 2014 Section 14 Real-Time Patch Real-Time Patch To Native Linux Kernel  Linux Kernel in Real-Time Environment: Real-Time Patch has infrastructure for most hard real-time.  Size of the code is too large and general purpose design makes it difficult to meet all RT Patch needs. Only necessary drivers are loaded (bulk of Linux are drivers).  Drivers must be audited to not disable interrupts for long periods. Kernel configuration options must be optimized. Only enable options that are needed.