CSCI1600: Embedded and Real Time Software Lecture 28: Real Time Operating Systems Steven Reiss, Fall 2017

What is a RTOS It’s an operating system Similar to UNIX, Windows, Linux, Mac OS, Solaris Provides all the low level services one expects So they don’t have to be rewritten for each application Shares the CPU and other hardware resources Except it’s different in some ways Lecture 29: Real time OS 11/10/2018

RTOS Differences More control over scheduling More “embeddability” Real time tasks Real priorities (not hints) Simpler scheduling techniques Easier to reason about Provable properties More “embeddability” Application might remain in control RTOS is a library Footprint/size is a concern Compile time configurations, only basis services Lecture 29: Real time OS 11/10/2018

RTOS Basic Functions Kernel (scheduler, resource management) Error handling System clock functions Device drivers, network stack send/receive IPC Initiate and start Interrupts Memory management Lecture 29: Real time OS 11/10/2018

Device Drivers Goal Motivation Provide a standard interface to any I/O device Open, close, read, write, ioctl Motivation Lets OS control the device and its resources Lets programs be easily written for new devices Lets new devices be attached to the system Lecture 29: Real time OS 11/10/2018

RTOS Scheduling Tasks Valid state transitions State: RUNNING, READY/RUNNABLE, SUSPENDED/BLOCKED Blocked: TIMER, INTERRUPTS, LOCK Priority Context (stack pointer, registers) for preemption Valid state transitions RUNNING -> READY, READY -> RUNNING sched() : makes proper task running RUNNING -> SUSPENDED suspend() : moves current job to waiting state SUSPENDED -> READY wakeup() called when job gets an interrupt Lecture 29: Real time OS 11/10/2018

RTOS Task Interface Task interface to scheduler Locks I/O Timing yield() [RUNNING->READY], sleep() [RUNNING->TIMERS] acquire(), release(), enable/disable interrupts Locks Each lock maintains a queue and a state Handle acquire and release appropriately I/O Each device maintains a queue read(), write() Timing Maintain a timer queue (ordered by wake up time) Processed on timer interrupts Lecture 29: Real time OS 11/10/2018

RTOS Tasks What information does a scheduler need about a real time (periodic) task? For RM/DM scheduling For EDF scheduling How might that information be conveyed? Lecture 29: Real time OS 11/10/2018

Task Priorities Suppose each task has a priority Based on its rate (period) Smaller rates have higher priority Aperiodic tasks have lower priority What happens then? Assume that periodic tasks use less than 60% of time What can you say about them meeting deadlines Lecture 29: Real time OS 11/10/2018

RTOS Scheduling Real time tasks and real time priorities Other tasks: simple method Round robin, FIFO, priority based; fair share In slack time from real time tasks Lecture 29: Real time OS 11/10/2018

RTOS Examples: Arduino Operating system is a set of libraries Provide access to I/O lines (analog/digital) PWM output, analog input, pull-up Read/write from SD or memory Provide access to timers, interrupts Provide network access Initialization/Loop No scheduling Lecture 29: Real time OS 11/10/2018

RTOS: μCOS (MUCOS, FreeRTOS) Library of C routines implementing OS functions User stays in control Highly protable, ROM-able, scalable, preemptive, deterministic Up to 64 tasks (56 user) Connectivity to GUI and file system interfaces (drivers) 8 to 64 bit processors supported Functions Task management, memory management, time management, IPC Lecture 29: Real time OS 11/10/2018

μCOS Task Management Rate monotonic scheduling States Critical regions All tasks are periodic Tasks have priority (based on period) Same priority tasks are handled via round robin Assumes true periodic load is < 69% States New, Ready, Waiting, Running, Terminated Critical regions OS_ENTER_CRITICAL, OS_EXIT_CRITICAL Task switching OS_TASK_SW switches task Saves registers, restores registers, returns to new task Lecture 29: Real time OS 11/10/2018

μCOS Functionality Memory Management Time Management IPC Done in terms of partitions (fixed size blocks) Allocation is constant time and deterministic Multiple partitions provide blocks of different sizes Time Management Clock tick every 10-100ms increments a counter Delay and resume based on the counter IPC Semaphores, mailboxes, message queues ISR routines can interact using wait/signal Lecture 29: Real time OS 11/10/2018

μCOS Features Add-ons (additional libraries) Embeddability Bitmapped graphics FAT file system Embeddability Individually enable or disable features Semaphores, mailboxes, message queues Task manipulations (delete, change priority) Number of tasks, events, queue sizes Small footprint (15k with everything enabled) Lecture 29: Real time OS 11/10/2018

Conventional OS RT Problems Size (for embedded) Micro-kernel based architectures help here Unpredictable (and poor) latency Slow interrupt handlers Unpreemptible kernel code Latency might be built into hardware System management interrupts on x86 DMA bus mastering; USB handling; VGA console; Page faults Unpredictable Scheduling Desktop/server fairness heuristics Scheduling overhead grows with workload Poor support for precise timing Timing accuracy related to tick overhead Lecture 29: Real time OS 11/10/2018

Are the Problems Fixable? Want to retain the bulk of prior work Hundreds of device drivers Many abstractions make programming easier TCP/IP, File systems, IPC, Memory How can we do this? Two approaches to this are used Fixing the problems Sidestepping the problems Lecture 29: Real time OS 11/10/2018

RT-Linux Assume most of the program is non-real time There is a small hard (soft) real-time component Attempt to side step the problems Run the OS as a real time task Allow other RT tasks (outside the OS) Provide the benefits of an OS and still have a real time environment Except that Linux tasks are now aperiodic

Linux on a RTOS We’ve seen this before This is basically RT-Linux Scheduling sporadic tasks Bandwidth servers Group all the sporadic tasks together Provide a real-time task (with fixed bandwidth) This executes the various sporadic tasks Using their priorities This is basically RT-Linux Linux is “ported” to use RTOS primitives instead of hardware Like running on a virtual machine

RT-Linux Details RT-Linux core Non-preemptable Hard real time scheduler Basic OS functionality Deterministic interrupt-latency ISRs run in core Others are transferred to Linux via a FIFO Primitive real-time tasks Only statically allocated memory, no virtual memory Fixed priority Run as real time threads

RT-Linux Details Real time tasks FIFO (message queue) No address space protection Run with disabled interrupts FIFO (message queue) Connects real time tasks with LINUX Shared memory synchronization Non-real time tasks Run as Linux processes

RT-Linux Pros and Cons Pros Cons Hard real-time guarantees (low tens of microseconds) OS code is virtually unchanged Applies equally well to other OS’s Supports BSD as well as Linux Cons Real time tasks are hard to code No mature well-understood API No memory protection Communication with Linux processes is ad-hoc Message queues – but user needs to define the messages

Fixing the Problems Problems to tackle Techniques Scheduling, latency, timing Techniques Priority scheduling and priority inheritance Limit non-preemptable code Offer high-resolution timers

Scheduling Real time schedulers are actually simpler Support for fixed (absolute) priority Ability to bind tasks to specific processors (cores) Create new classes of processes (RR, FIFO) Round robin runs for a fixed time slice FIFO is never guaranteed Enhance mutexes to support priority (PIP,PCP) Problem: scheduling latency Support fixed priorities But scheduling decisions might be deferred Interrupt handlers cannot be preempted : can be fixed Spin-locked code can not be preempted : can be fixed

Other Improvements Dynamic tick O(1) scheduler Robust mutexes Ask for timer IRQ as needed, not periodically High resolution timers O(1) scheduler Robust mutexes O(1) kernel memory allocation Ability to lock pages into memory (avoid page faults) Most problems can be minimized But retrofitting them to an OS make hard real time unlikely Soft real time on Linux, Windows

Solaris Attempt to fix the problems Features of Solaris for real time Starting with the initial OS design Not retrofitted Features of Solaris for real time Interrupts are handled by threads Kernel is fully preemptible and multithreaded High-res timers; PIP/PCP support;fast TCP; memory locking; locking threads to cores Real time scheduling is supported Deterministic with real time priorities

Embedded Linux Linux is a common tool for embedded systems Linux itself is relatively small Linux is fairly modular It can run in 1M (or less) of memory

Next Time Verification Lecture 29: Real time OS 11/10/2018

Homework Read chapter 13 Exercises 13.1, 13.2, 13.4

OS & Device Drivers User/kernel boundary Kernel takes care of Supported as part of the CPU Privileged vs non-privileged instructions I/O instructions are privileged Interrupts are privileged Memory mapping instructions are privileged Kernel takes care of Memory management Process scheduling All I/O Lecture 29: Real time OS 11/10/2018

I/O Devices Can be broken down by category (flavor) CHAR : device accessed like a stream Serial I/O, tty, mouse, sound Also simple devices (parallel I/O) BLOCK : device accessed in bulk Often DMA access NETWORK : device is a network device Devices of the same flavor have much in common Lecture 29: Real time OS 11/10/2018

Device Numbers Devices Identified by a device number Number: (major,minor) Major number: driver ID Minor number: used by the driver Allocated dynamically in modern OS /dev/xxx -> identifies major and minor numbers Lecture 29: Real time OS 11/10/2018

Kernel Modules Modern kernels allow you to add/remove drivers Dynamically Without rebuild, reboot, recompile Kernel code is restricted programming No general libraries available Kernel functions are callable Some (limited) memory allocation, delay, synch, logging Lecture 29: Real time OS 11/10/2018

User Access to Devices Uses the file interface (device = file) open() : provides access to the device Can be exclusive (write) or nonexclusive (read) close() : releases the device Standard access calls read, write seek ioctl, fcntl What do these operations mean? Whatever the driver says they mean Lecture 29: Real time OS 11/10/2018

Device Driver Interface module_init, module_exit Routines called on installation and removal Init Registers the device Sets the device to a known state Gets the IRQ, addresses of I/O Exit Releases the memory addresses Lecture 29: Real time OS 11/10/2018

Device Driver Interface Driver provides other methods open(), read(), write(), llseek(), ioctl(), fasync(), release() These can be omitted is not relevant All calls provided with a common kernel structure (filp) That can hold local information about the device open/release – fill in the data structure read/write provided with pointers to user space fasync – indicates asynchronous I/O desired Lecture 29: Real time OS 11/10/2018

Device Driver Interface ioctl(int device,int command,{data}) command is device specific (header file defined) data might be a reference to user address space Commands can be used to control the device Pinball: RESET, READREG, WRITEREG, DATA, DISPLAY, INTERR Lecture 29: Real time OS 11/10/2018

RTOS Task Coding What does a task look like in a RTOS? Consider outputting a sound sample Loop while samples exist Output current sample Get time until next sample Sleep Note that waiting here is okay (why?) Consider checking switches in tic-tac-toe Lecture 29: Real time OS 11/10/2018

RTOS Scheduling Real time tasks and real time priorities Other tasks: simple method Round robin, FIFO, priority based; fair share Priority approach using dynamic priority (UNIX) Priority increases if suspended voluntarily CPU-bound processes are penalized Low-priority “snubbed” processes are increased Limit to priority fluctuation This doesn’t work for real time tasks Lecture 29: Real time OS 11/10/2018

Conventional Operating Systems Memory model Independent address spaces Allocation and deallocation Stack management Device support Abstraction based on descriptors Clock and other basic devices built in High-level I/O support (e.g. networking) Thread / process model (with scheduling) IPC support (pipes, sockets, shared memory) Synchronization primitives Lecture 29: Real time OS 11/10/2018

RT Linux

Minimizing Latency Simple Solution: This almost works Threaded interrupt handling Move interrupt handlers into kernel threads; true ISR is now very short Handling can be preempted Preemptible spin locks Turn spin locks into mutexes This almost works Timer interrupts need to be handled directly Individual handlers are still atomic Spinlocks not always equivalent to mutexes Programming devices, modifying page tables, in the scheduler Read-copy-update synchronization in Linux make non-preemption assumptions New type: raw_spinlock_t

Solaris Real Time Threads Separate classes of threads Fixed-priority (non-varying) But lower priority than Operating System Real-Time (fixed priority with fixed quantum) Higher priority than operating system Real time threadss Lock their pages in memory Can call OS (but then run at lower priority)

Solaris Features Priority inheriting synchronization primitives Processor sets The ability to isolate one or more CPUs for scheduling Ability to lock tasks to a particular core Full support for the Posix real time standard Real time threads High-precision timers and clocks Fast TCP/IP (zero copy) Memory locking Early binding of shared libraries

Solaris Has been used as a hard real time environment But not as an embedded environment Too large

Requirements on RT/Embedded Safety Requirements The car won’t crash Trains won’t crash Both traffic lights won’t be green at the same time You won’t blow a fuse The system won’t deadlock The heat and air conditioning won’t be on at the same time

More Requirements: Timing Forms X will occur within k ms of event Y X will occur every 10 +/- 1 ms Examples The solenoid will trigger within 10 ms of the bumper being hit The switches will be sensed every 20 ms The heat will come on within 5 minutes of detecting the room is too cool

More Requirements: Fairness Forms Eventually X will occur If X occurs, then eventually Y will occur X will occur infinitely often As long as Y is true, X will occur infinitely often Examples Eventually you will add K to the score when you hit Y As long as the heating system is on, the room will eventually be comfortable

Requirements Are often mandatory (hard) Failure is not a viable option Car or plan crashes Broken devices Heart stops pumping Water is supplied unfiltered The nuclear plant does not shut down This is what hard real time means

What Are Your Requirements?

Problems How do you show the system is safe How do you get confidence in the system’s safety When can you reasonably deploy the system Can the system be used in a medical device Can peoples lives depend on the system

Exercise Suppose you wanted to sell your tic-tac-toe box What would you want as requirements How would you check these? Take the ankle-mounted obstacle finder example What would be the requirements

Helpful Techniques: SE Good requirements analysis Understanding all the possible problems and solutions Good specifications Accurate modeling Showing the models are correct Petri net and FSA validation Design for security Defensive coding Building monitors as part of the code Enter a safe state if the monitor detects anything wrong Enter a safe state if the monitor detects anything unusual Checking tools: lint, compiler, modeling analysis