CSCI1600: Embedded and Real Time Software Lecture 27: Real Time Linux Steven Reiss, Fall 2015

Conventional Operating Systems  Memory model  Independent address spaces  Allocation and deallocation  Stack management  Device support  Abstraction based on descriptors  Clock and other basic devices built in  High-level I/O support (e.g. networking)  Thread / process model (with scheduling)  IPC support (pipes, sockets, shared memory)  Synchronization primitives

Conventional OS Problems  Size (for embedded)  Micro-kernel based architectures help here  Unpredictable (and poor) latency  Slow interrupt handlers  Unpreemptible kernel code  Latency might be built into hardware  System management interrupts on x86  DMA bus mastering; USB handling; VGA console; Page faults  Unpredictable Scheduling  Desktop/server heuristics  Scheduling overhead grows with workload  Poor support for precise timing  Timing accuracy related to tick overhead

Are the Problems Fixable?  Want to retain the bulk of prior work  Hundreds of device drivers  Many abstractions make programming easier  TCP/IP, File systems, IPC, Memory  How can we do this?  Two approaches to this are used  Fixing the problems  Sidestepping the problems

RT-Linux  Assume most of the program is non-real time  There is a small hard (soft) real-time component  Attempt to side step the problem  Run the OS as a real time task  Allow other RT tasks (outside the OS)  Provide the benefits of an OS and still have a real time environment  Except that Linux tasks are now sporadic

Linux on a RTOS  We’ve seen this before  Scheduling sporadic tasks  Bandwidth servers (Total and Constant)  Group all the sporadic tasks together  Provide a real-time task (with fixed bandwidth)  This executes the various sporadic tasks  Using their priorities  This is basically RT-Linux  Linux is “ported” to use RTOS primitives instead of hardware  Like running on a virtual machine

RT Linux

RT-Linux Details  RTLinux core  Non-preemptible  Hard real time scheduler  Basic OS functionality  Deterministic interrupt-latency ISRs run in core  Others are transferred to Linux via a FIFO  Primitive tasks  Only statically allocated memory, no virtual memory, fixed priority  Run as real time threads

RT-Linux Details  Real time tasks  No address space protection  Run with disabled interrupts  FIFO (message queue)  Connects real time tasks with LINUX  Shared memory synchronization  Non-real time tasks  Run as Linux processes

RT-Linux Pros and Cons  Pros  Hard real-time guarantees (low tens of microseconds)  OS code is virtually unchanged  Applies equally well to other OS’s  Supports BSD as well as Linux  Cons  Real time task lose the convenience of a mature well- understood API  Communication with Linux processes is ad-hoc  Message queues – but user needs to define the messages

Fixing the Problems  Problems to tackle  Scheduling, latency, timing  Techniques  Priority scheduling and priority inheritance  Limit un-preemptible code  Offer high-resolution timers

Scheduling  Real time schedulers are actually simpler  Support for fixed (absolute) priority  Ability to bind tasks to specific processors (cores)  Create new classes of processes (RR, FIFO)  Round robin runs for a fixed time slice  FIFO is never guaranteed  Enhance mutexes to support priority (PIP,PCP)  Problem: scheduling latency  Support fixed priorities  But scheduling decisions might be deferred  Interrupt handlers cannot be preempted  Spin-locked code can not be preempted

Minimizing Latency  Simple Solution:  Threaded interrupt handling  Move interrupt handlers into kernel threads; true ISR is now very short  Handling can be preempted  Preemptible spin locks  Turn spin locks into mutexes  This almost works  Timer interrupts need to be handled directly  Individual handlers are still atomic  Spinlocks not always equivalent to mutexes  Programming devices, modifying page tables, in the scheduler  Read-copy-update synchronization in Linux make non-preemption assumptions  New type: raw_spinlock_t

Other Improvements  Dynamic tick  Ask for timer IRQ as needed, not periodically  High resolution timers  O(1) scheduler  Robust mutexes  O(1) kernel memory allocation  Ability to lock pages into memory (avoid page faults)  Most problems can be minimized  But retrofitting them to an OS make hard real time unlikely

Solaris  Attempt to fix the problems  Starting with the initial OS design  Not retrofitted  Features of Solaris for real time  Interrupts are handled by threads  Kernel is fully preemptible and multithreaded  Real time scheduling is supported  Deterministic with real time priorities  Fixed priority scheduling supported

Solaris Real Time Threads  Separate classes of threads  Fixed-priority (non-varying)  But lower priority than Operating System  Real-Time (fixed priority with fixed quantum)  Higher priority than operating system  Real time threadss  Lock their pages in memory  Can call OS (but then run at lower priority)

Solaris Features  Priority inheriting synchronization primitives  Processor sets  The ability to isolate one or more CPUs for scheduling  Ability to lock tasks to a particular core  Full support for the Posix real time standard  Real time threads  High-precision timers and clocks  Fast TCP/IP (zero copy)  Memory locking  Early binding of shared libraries

Solaris  Has been used as a hard real time environment  But not as an embedded environment  Too large

Embedded Linux  Linux is a common tool for embedded systems  Linux itself is relatively small  Linux is fairly modular  It can run in 1M (or less) of memory

Requirements on RT/Embedded  Safety Requirements  The car won’t crash  Trains won’t crash  Both traffic lights won’t be green at the same time  You won’t blow a fuse  The system won’t deadlock  The heat and air conditioning won’t be on at the same time

More Requirements: Timing  Forms  X will occur within k ms of event Y  X will occur every 10 +/- 1 ms  Examples  The solenoid will trigger within 10 ms of the bumper being hit  The switches will be sensed every 20 ms  The heat will come on within 5 minutes of detecting the room is too cool

More Requirements: Fairness  Forms  Eventually X will occur  If X occurs, then eventually Y will occur  X will occur infinitely often  As long as Y is true, X will occur infinitely often  Examples  Eventually you will add K to the score when you hit Y  As long as the heating system is on, the room will eventually be comfortable

Requirements  Are often mandatory (hard)  Failure is not a viable option  Car or plan crashes  Broken devices  Heart stops pumping  Water is supplied unfiltered  The nuclear plant does not shut down  This is what hard real time means

What Are Your Requirements?

Problems  How do you show the system is safe  How do you get confidence in the system’s safety  When can you reasonably deploy the system  Can the system be used in a medical device  Can peoples lives depend on the system

Exercise  Suppose you wanted to sell your tic-tac-toe box  What would you want as requirements  How would you check these?  Take the ankle-mounted obstacle finder example  What would be the requirements  How would you check these?

Helpful Techniques: SE  Good requirements analysis  Understanding all the possible problems and solutions  Good specifications  Accurate modeling  Showing the models are correct  Petri net and FSA validation  Design for security  Defensive coding  Building monitors as part of the code  Enter a safe state if the monitor detects anything wrong  Enter a safe state if the monitor detects anything unusual  Checking tools: lint, compiler, modeling analysis

Homework  Read chapter 13