Performance Evaluation of Real-Time Operating Systems RTEMS, RTLinux, eCos

Real-Time Systems Systems that interact predictably with events in the outside world Examples: Flight control systems Collision avoidance systems Satellite guidance systems Patient Monitoring System Control Systems for synchrotrons

Real-Time Operating Systems Free the applications programmer from writing code for task scheduling and dispatching. Manage time-sharing of a processor for a number of different tasks and interrupt sources, while adhering to strict time constraints. Widely used in all kinds of applications both on PCs and in embedded systems. Generally operate unattended -- consequences of failure can be catastrophic. The function of a real-time operating system is to manage time-sharing of a processor for a number of different tasks and interrupt sources, while adhering to strict time constraints. Real-time operating systems are widely used in all kinds of applications both on PCs and in embedded systems where the processor is completely encapsulated within the hardware it is controlling. Real-time systems must interact predictably with devices in the outside world. “They generally operate unattended, and the consequences of any kind of failure range from severe to unthinkable.”

What is available? Many RTOSs have been developed: Commercial Open source RTOS vendors and developers publish performance metrics to showcase their products. However, their evaluations are: not performed on a common platform not comprehensive Many commercial RTOSs are available. Several open source, royalty-free packages are also available or under development. RTOS vendors and developers publish performance metrics to show the superiority of their products. However, these metrics are not performed on a common platform and so are like comparing apples to oranges. They are also not necessarily comprehensive.

Evaluation of an RTOS How would you choose an appropriate RTOS for your application? Need for an impartial evaluation. Several such projects have been conducted. Only evaluated some performance characteristics Only looked at a few commercial operating systems Many are now out of date ‘Real-Time Consult’ introduced a more comprehensive project. Developed a thorough test methodology Evaluated several commercial RTOSs Have not evaluated open source RTOSs An extensive project was started about three years ago, by a group called “Real-Time Consult”, to qualify and compare real-time operating systems. they have developed a methodology and started producing evaluation reports for several commercially available RTOSs. However, we have not found any suitable comparison for open source RTOSs. There are many benefits to using an open source RTOS, such as: the cost benefit (royalty-free) and, more importantly, the performance can be more thoroughly tested than if the source code is not available code can be modified or modules written for your own particular application.

Research Objective Performance evaluation, on a common platform, of three open source real-time operating systems: RTEMS RTLinux eCos I will get each RTOS running on a PC, then do performance tests on each one.

Real-Time Operating Systems Overview RTOS Task Scheduling Task Dispatching Task Synchronization The function of a real-time operating system is to manage time-sharing of a processor for a number of different tasks and interrupt sources, while adhering to strict time constraints.

Task Scheduling Time Slicing Last Arrival Order First Time Task 3

Task Scheduling Preemptive Priority High Priority Low Time Task 3 The function of a real-time operating system is to manage time-sharing of a processor for a number of different tasks and interrupt sources, while adhering to strict time constraints. Task 1 Task 1

Task Dispatching Task-control blocks is the most popular method of identifying and managing tasks. Task-Control Block (TCB) contains: a context (e.g. program counter and register contents) an identification string a status, such as ready, executing or blocked a priority (if applicable) A Typical Task Control Block Program Counter Task Status Task ID Number Pointer to next TCB Other Context Priority : Register Contents

Task Synchronization Events Messages Semaphores Mutexes

Desired Performance Characteristics Tasks should run with: minimal event latency (the time between the triggering of a task and its start) minimal jitter (the variation in running times of a task that is supposed to run at a fixed period).

Performance Evaluation Throughput – speed at which the system executes instructions Responsiveness – how fast it starts to handle in interrupt request Determinism – how it reacts under load how long does it take to finish what it is doing and start handling the interrupt request

Performance Metrics Low Level Tests Context switching Interrupt latency Exclusion objects Semaphores Mutexes Synchronization events

Context Switch Time Task A Start Task B : : Context Switch Time Switch to Task B Save context of Task A Load context of Task B Task A Start Task B PC Register 0 : Other Context PC Register 0 : Other Context Time Context Switch Time

Other Considerations Not an isolated event Affected by number of tasks pending Depends on priorities of pending tasks Amount of context to be saved

Interrupt Latency Interrupt and Interrupt Dispatch Latencies Task Task ISR First Instruction in Interrupt Service Routine (ISR) is executed Interrupt Occurs Task resumes ISR ends Task Time Interrupt Latency Interrupt Dispatch Latency

Multiple Interrupt Latencies Interrupt-to-Task Run Interrupt Interrupt Dispatch Time Interrupt Service Routine Other Interrupt Pre-emption Disabled More than one interrupt is pending: response time for second interrupt depends on the length of the first interrupt. You have to finish serving the first interrupt which means you have to execute the Interrupt Service Routine. This includes: ISR must, at a minimum, send a synchronization event to a task. Other processing code might also be included in the ISR and must also finish. This is why it is recommended that this kind of processing be done outside the ISR if at all possible. Return from interrupt and enable next interrupt Scheduling Context Switch Return from System Call Task Run

Interrupt Task Response Time Real response to an interrupt often occurs in a task synchronized by, but outside of, the actual ISR. Task response also depends on whether the kernel is preemptable. If not, then kernel call must be completed before ISR. More than one interrupt is pending: response time for second interrupt depends on the length of the first interrupt. You have to finish serving the first interrupt which means you have to execute the Interrupt Service Routine. This includes: ISR must, at a minimum, send a synchronization event to a task. Other processing code might also be included in the ISR and must also finish. This is why it is recommended that this kind of processing be done outside the ISR if at all possible. Return from interrupt and enable next interrupt

Performance Metrics High Level Tests Network throughput Stress tests

Real-Time Consult Project Evaluation and comparison of RTOS performance Commercial systems only, to date Evaluations available for: Intime 1.20 RTX 4.2 Hyperkernel 4.3 VxWorks/x86 5.3.1 pSOSystem/x86 2.2.6 QNX 6.1 CE 3.0

Real-Time Consult Project Test Method: timing is measured using an external PCI bus analyzer. Two types of tests: Performance Tests Stress Tests Qualitative Evaluation (API richness, etc.)

Real-Time Consult Project Performance Tests (context switch & latencies): Thread switch latency time required to preempt current thread Interrupt latency preempt current thread and start interrupt handler Interrupt dispatch latency switch from interrupt context to context of interrupted thread or next thread in queue Thread creation and deletion

Task Switch Latency Test Create tasks TSK1, …, TSKN Task start TSK0 Delete tasks Write trace on PCI Bus TRC_TSK0_TS0 Yield Processor Yes No End of Test? Task end Each Task Write trace on PCI Bus TRC_TSKN_TSN Yield Processor Task start TSKN

Real-Time Consult Project Performance Tests (objects, file, network): Synchronization objects time to create and delete a synchronization object Exclusion objects time to create and delete an exclusion object File system operations creating and deleting files, reading from files and writing to files in synchronous mode Network stack performance of TCP/IP stack bandwidth for various packet sizes, and CPU usage

Real-Time Consult Project Stress Tests: Two simultaneous interrupts Interrupt nesting maximum sustainable interrupt frequency maximum number of objects memory leaks

Focus of my Research Evaluation of three Open Source Real-Time Operating Systems: RTEMS Real Time Executive for Multiprocessor Systems RTLinux Real Time Linux eCos Embedded Configurable Operating System There are many benefits to using an open source RTOS, such as: the cost benefit (royalty-free) and, more importantly, the performance and reliability can be more thoroughly tested than if the source code is not available code can be modified or modules written for your own particular application.

Application Dependent Software Standard Application Components RTEMS Developed by the U.S. military as alternative to using commercial RTOS Small, easy to port High level of user configurability Kernel is preemptible Application Dependent Software Standard Application Components Device Drivers R T E M S Target Hardware

RTLinux Abstraction layer between the hardware and the standard Linux kernel Appears as actual hardware to standard Linux kernel Lowest priority is assigned to standard Linux kernel, which then runs as a independent task. RTLinux executive is nonpreemptible Leaves Linux kernel essentially untouched so it doesn’t hinder future Linux development

ECOS Targeted at high-volume applications: Configurable: Portable consumer electronics, telecommunications, automotive other deeply embedded applications. Configurable: lets developer configure a system that best matches the needs of the application. Typical configuration options: type of scheduler number of priority levels. current release of the system has over 200 options Portable Fully preemptable

Typical Application Control system for the Canadian Light Source (CLS) uses an open source real-time operating system and control software: RTOS  RTEMS Control Software  EPICS (Experimental Physics and Industrial Control System) Such distributed control systems typically comprise tens or even hundreds of computers, networked together to allow communication between them and to provide control and feedback of the various parts of the device from a central control room, or even remotely over the internet. Commonly used in distributed control systems to operate devices such as: Particle Accelerators Large Experiments Major Telescopes.