1. CSCI1600: Embedded and Real Time Software
Lecture 28: Real Time Operating Systems
Steven Reiss, Fall 2017

2. What is a RTOS It’s an operating system
Similar to UNIX, Windows, Linux, Mac OS, Solaris
Provides all the low level services one expects
So they don’t have to be rewritten for each application
Shares the CPU and other hardware resources
Except it’s different in some ways
Lecture 29: Real time OS
11/10/2018

3. RTOS Differences More control over scheduling More “embeddability”
Real time tasks
Real priorities (not hints)
Simpler scheduling techniques
Easier to reason about
Provable properties
More “embeddability”
Application might remain in control
RTOS is a library
Footprint/size is a concern
Compile time configurations, only basis services
Lecture 29: Real time OS
11/10/2018

4. RTOS Basic Functions Kernel (scheduler, resource management)
Error handling
System clock functions
Device drivers, network stack send/receive
IPC
Initiate and start
Interrupts
Memory management
Lecture 29: Real time OS
11/10/2018

5. Device Drivers Goal Motivation
Provide a standard interface to any I/O device
Open, close, read, write, ioctl
Motivation
Lets OS control the device and its resources
Lets programs be easily written for new devices
Lets new devices be attached to the system
Lecture 29: Real time OS
11/10/2018

6. RTOS Scheduling Tasks Valid state transitions
State: RUNNING, READY/RUNNABLE, SUSPENDED/BLOCKED
Blocked: TIMER, INTERRUPTS, LOCK
Priority
Context (stack pointer, registers) for preemption
Valid state transitions
RUNNING -> READY, READY -> RUNNING
sched() : makes proper task running
RUNNING -> SUSPENDED
suspend() : moves current job to waiting state
SUSPENDED -> READY
wakeup() called when job gets an interrupt
Lecture 29: Real time OS
11/10/2018

7. RTOS Task Interface Task interface to scheduler Locks I/O Timing
yield() [RUNNING->READY], sleep() [RUNNING->TIMERS]
acquire(), release(), enable/disable interrupts
Locks
Each lock maintains a queue and a state
Handle acquire and release appropriately
I/O
Each device maintains a queue
read(), write()
Timing
Maintain a timer queue (ordered by wake up time)
Processed on timer interrupts
Lecture 29: Real time OS
11/10/2018

8. RTOS Tasks
What information does a scheduler need about a real time (periodic) task?
For RM/DM scheduling
For EDF scheduling
How might that information be conveyed?
Lecture 29: Real time OS
11/10/2018

9. Task Priorities Suppose each task has a priority
Based on its rate (period)
Smaller rates have higher priority
Aperiodic tasks have lower priority
What happens then?
Assume that periodic tasks use less than 60% of time
What can you say about them meeting deadlines
Lecture 29: Real time OS
11/10/2018

10. RTOS Scheduling Real time tasks and real time priorities
Other tasks: simple method
Round robin, FIFO, priority based; fair share
In slack time from real time tasks
Lecture 29: Real time OS
11/10/2018

11. RTOS Examples: Arduino
Operating system is a set of libraries
Provide access to I/O lines (analog/digital)
PWM output, analog input, pull-up
Read/write from SD or memory
Provide access to timers, interrupts
Provide network access
Initialization/Loop
No scheduling
Lecture 29: Real time OS
11/10/2018

12. RTOS: μCOS (MUCOS, FreeRTOS)
Library of C routines implementing OS functions
User stays in control
Highly protable, ROM-able, scalable, preemptive, deterministic
Up to 64 tasks (56 user)
Connectivity to GUI and file system interfaces (drivers)
8 to 64 bit processors supported
Functions
Task management, memory management, time management, IPC
Lecture 29: Real time OS
11/10/2018

13. μCOS Task Management Rate monotonic scheduling States Critical regions
All tasks are periodic
Tasks have priority (based on period)
Same priority tasks are handled via round robin
Assumes true periodic load is < 69%
States
New, Ready, Waiting, Running, Terminated
Critical regions
OS_ENTER_CRITICAL, OS_EXIT_CRITICAL
Task switching
OS_TASK_SW switches task
Saves registers, restores registers, returns to new task
Lecture 29: Real time OS
11/10/2018

14. μCOS Functionality Memory Management Time Management IPC
Done in terms of partitions (fixed size blocks)
Allocation is constant time and deterministic
Multiple partitions provide blocks of different sizes
Time Management
Clock tick every ms increments a counter
Delay and resume based on the counter
IPC
Semaphores, mailboxes, message queues
ISR routines can interact using wait/signal
Lecture 29: Real time OS
11/10/2018

15. μCOS Features Add-ons (additional libraries) Embeddability
Bitmapped graphics
FAT file system
Embeddability
Individually enable or disable features
Semaphores, mailboxes, message queues
Task manipulations (delete, change priority)
Number of tasks, events, queue sizes
Small footprint (15k with everything enabled)
Lecture 29: Real time OS
11/10/2018

16. Conventional OS RT Problems
Size (for embedded)
Micro-kernel based architectures help here
Unpredictable (and poor) latency
Slow interrupt handlers
Unpreemptible kernel code
Latency might be built into hardware
System management interrupts on x86
DMA bus mastering; USB handling; VGA console; Page faults
Unpredictable Scheduling
Desktop/server fairness heuristics
Scheduling overhead grows with workload
Poor support for precise timing
Timing accuracy related to tick overhead
Lecture 29: Real time OS
11/10/2018

17. Are the Problems Fixable?
Want to retain the bulk of prior work
Hundreds of device drivers
Many abstractions make programming easier
TCP/IP, File systems, IPC, Memory
How can we do this?
Two approaches to this are used
Fixing the problems
Sidestepping the problems
Lecture 29: Real time OS
11/10/2018

18. RT-Linux Assume most of the program is non-real time
There is a small hard (soft) real-time component
Attempt to side step the problems
Run the OS as a real time task
Allow other RT tasks (outside the OS)
Provide the benefits of an OS and still have a real time environment
Except that Linux tasks are now aperiodic

19. Linux on a RTOS We’ve seen this before This is basically RT-Linux
Scheduling sporadic tasks
Bandwidth servers
Group all the sporadic tasks together
Provide a real-time task (with fixed bandwidth)
This executes the various sporadic tasks
Using their priorities
This is basically RT-Linux
Linux is “ported” to use RTOS primitives instead of hardware
Like running on a virtual machine

20. RT-Linux Details RT-Linux core
Non-preemptable
Hard real time scheduler
Basic OS functionality
Deterministic interrupt-latency ISRs run in core
Others are transferred to Linux via a FIFO
Primitive real-time tasks
Only statically allocated memory, no virtual memory
Fixed priority
Run as real time threads

21. RT-Linux Details Real time tasks FIFO (message queue)
No address space protection
Run with disabled interrupts
FIFO (message queue)
Connects real time tasks with LINUX
Shared memory synchronization
Non-real time tasks
Run as Linux processes

22. RT-Linux Pros and Cons Pros Cons
Hard real-time guarantees (low tens of microseconds)
OS code is virtually unchanged
Applies equally well to other OS’s
Supports BSD as well as Linux
Cons
Real time tasks are hard to code
No mature well-understood API
No memory protection
Communication with Linux processes is ad-hoc
Message queues – but user needs to define the messages

23. Fixing the Problems Problems to tackle Techniques
Scheduling, latency, timing
Techniques
Priority scheduling and priority inheritance
Limit non-preemptable code
Offer high-resolution timers

24. Scheduling Real time schedulers are actually simpler
Support for fixed (absolute) priority
Ability to bind tasks to specific processors (cores)
Create new classes of processes (RR, FIFO)
Round robin runs for a fixed time slice
FIFO is never guaranteed
Enhance mutexes to support priority (PIP,PCP)
Problem: scheduling latency
Support fixed priorities
But scheduling decisions might be deferred
Interrupt handlers cannot be preempted : can be fixed
Spin-locked code can not be preempted : can be fixed

25. Other Improvements Dynamic tick O(1) scheduler Robust mutexes
Ask for timer IRQ as needed, not periodically
High resolution timers
O(1) scheduler
Robust mutexes
O(1) kernel memory allocation
Ability to lock pages into memory (avoid page faults)
Most problems can be minimized
But retrofitting them to an OS make hard real time unlikely
Soft real time on Linux, Windows

26. Solaris Attempt to fix the problems Features of Solaris for real time
Starting with the initial OS design
Not retrofitted
Features of Solaris for real time
Interrupts are handled by threads
Kernel is fully preemptible and multithreaded
High-res timers; PIP/PCP support;fast TCP; memory locking; locking threads to cores
Real time scheduling is supported
Deterministic with real time priorities

27. Embedded Linux Linux is a common tool for embedded systems
Linux itself is relatively small
Linux is fairly modular
It can run in 1M (or less) of memory

28. Next Time
Verification
Lecture 29: Real time OS
11/10/2018

29. Homework
Read chapter 13
Exercises 13.1, 13.2, 13.4

30. OS & Device Drivers User/kernel boundary Kernel takes care of
Supported as part of the CPU
Privileged vs non-privileged instructions
I/O instructions are privileged
Interrupts are privileged
Memory mapping instructions are privileged
Kernel takes care of
Memory management
Process scheduling
All I/O
Lecture 29: Real time OS
11/10/2018

31. I/O Devices Can be broken down by category (flavor)
CHAR : device accessed like a stream
Serial I/O, tty, mouse, sound
Also simple devices (parallel I/O)
BLOCK : device accessed in bulk
Often DMA access
NETWORK : device is a network device
Devices of the same flavor have much in common
Lecture 29: Real time OS
11/10/2018

32. Device Numbers Devices Identified by a device number
Number: (major,minor)
Major number: driver ID
Minor number: used by the driver
Allocated dynamically in modern OS
/dev/xxx -> identifies major and minor numbers
Lecture 29: Real time OS
11/10/2018

33. Kernel Modules Modern kernels allow you to add/remove drivers
Dynamically
Without rebuild, reboot, recompile
Kernel code is restricted programming
No general libraries available
Kernel functions are callable
Some (limited) memory allocation, delay, synch, logging
Lecture 29: Real time OS
11/10/2018

34. User Access to Devices Uses the file interface (device = file)
open() : provides access to the device
Can be exclusive (write) or nonexclusive (read)
close() : releases the device
Standard access calls
read, write
seek
ioctl, fcntl
What do these operations mean?
Whatever the driver says they mean
Lecture 29: Real time OS
11/10/2018

35. Device Driver Interface
module_init, module_exit
Routines called on installation and removal
Init
Registers the device
Sets the device to a known state
Gets the IRQ, addresses of I/O
Exit
Releases the memory addresses
Lecture 29: Real time OS
11/10/2018

36. Device Driver Interface
Driver provides other methods
open(), read(), write(), llseek(), ioctl(), fasync(), release()
These can be omitted is not relevant
All calls provided with a common kernel structure (filp)
That can hold local information about the device
open/release – fill in the data structure
read/write provided with pointers to user space
fasync – indicates asynchronous I/O desired
Lecture 29: Real time OS
11/10/2018

37. Device Driver Interface
ioctl(int device,int command,{data})
command is device specific (header file defined)
data might be a reference to user address space
Commands can be used to control the device
Pinball: RESET, READREG, WRITEREG, DATA, DISPLAY, INTERR
Lecture 29: Real time OS
11/10/2018

38. RTOS Task Coding What does a task look like in a RTOS?
Consider outputting a sound sample
Loop while samples exist
Output current sample
Get time until next sample
Sleep
Note that waiting here is okay (why?)
Consider checking switches in tic-tac-toe
Lecture 29: Real time OS
11/10/2018

39. RTOS Scheduling Real time tasks and real time priorities
Other tasks: simple method
Round robin, FIFO, priority based; fair share
Priority approach using dynamic priority (UNIX)
Priority increases if suspended voluntarily
CPU-bound processes are penalized
Low-priority “snubbed” processes are increased
Limit to priority fluctuation
This doesn’t work for real time tasks
Lecture 29: Real time OS
11/10/2018

40. Conventional Operating Systems
Memory model
Independent address spaces
Allocation and deallocation
Stack management
Device support
Abstraction based on descriptors
Clock and other basic devices built in
High-level I/O support (e.g. networking)
Thread / process model (with scheduling)
IPC support (pipes, sockets, shared memory)
Synchronization primitives
Lecture 29: Real time OS
11/10/2018

41. RT Linux

42. Minimizing Latency Simple Solution: This almost works
Threaded interrupt handling
Move interrupt handlers into kernel threads; true ISR is now very short
Handling can be preempted
Preemptible spin locks
Turn spin locks into mutexes
This almost works
Timer interrupts need to be handled directly
Individual handlers are still atomic
Spinlocks not always equivalent to mutexes
Programming devices, modifying page tables, in the scheduler
Read-copy-update synchronization in Linux make non-preemption assumptions
New type: raw_spinlock_t

43. Solaris Real Time Threads
Separate classes of threads
Fixed-priority (non-varying)
But lower priority than Operating System
Real-Time (fixed priority with fixed quantum)
Higher priority than operating system
Real time threadss
Lock their pages in memory
Can call OS (but then run at lower priority)

44. Solaris Features Priority inheriting synchronization primitives
Processor sets
The ability to isolate one or more CPUs for scheduling
Ability to lock tasks to a particular core
Full support for the Posix real time standard
Real time threads
High-precision timers and clocks
Fast TCP/IP (zero copy)
Memory locking
Early binding of shared libraries

45. Solaris Has been used as a hard real time environment
But not as an embedded environment
Too large

46. Requirements on RT/Embedded
Safety Requirements
The car won’t crash
Trains won’t crash
Both traffic lights won’t be green at the same time
You won’t blow a fuse
The system won’t deadlock
The heat and air conditioning won’t be on at the same time

47. More Requirements: Timing
Forms
X will occur within k ms of event Y
X will occur every 10 +/- 1 ms
Examples
The solenoid will trigger within 10 ms of the bumper being hit
The switches will be sensed every 20 ms
The heat will come on within 5 minutes of detecting the room is too cool

48. More Requirements: Fairness
Forms
Eventually X will occur
If X occurs, then eventually Y will occur
X will occur infinitely often
As long as Y is true, X will occur infinitely often
Examples
Eventually you will add K to the score when you hit Y
As long as the heating system is on, the room will eventually be comfortable

49. Requirements Are often mandatory (hard) Failure is not a viable option
Car or plan crashes
Broken devices
Heart stops pumping
Water is supplied unfiltered
The nuclear plant does not shut down
This is what hard real time means

50. What Are Your Requirements?

51. Problems How do you show the system is safe
How do you get confidence in the system’s safety
When can you reasonably deploy the system
Can the system be used in a medical device
Can peoples lives depend on the system

52. Exercise Suppose you wanted to sell your tic-tac-toe box
What would you want as requirements
How would you check these?
Take the ankle-mounted obstacle finder example
What would be the requirements

53. Helpful Techniques: SE
Good requirements analysis
Understanding all the possible problems and solutions
Good specifications
Accurate modeling
Showing the models are correct
Petri net and FSA validation
Design for security
Defensive coding
Building monitors as part of the code
Enter a safe state if the monitor detects anything wrong
Enter a safe state if the monitor detects anything unusual
Checking tools: lint, compiler, modeling analysis