ITTC IAB 4/7/05 1 Dr. Douglas Niehaus © 2005 Information and Telecommunication Technology Center Industrial Advisory Board Meeting April 7, 2005 Douglas Niehaus LibeRTOS: A Flexibly Configurable and Highly Capable Platform for Industrial Automation

ITTC IAB 4/7/05 2 Dr. Douglas Niehaus © 2005 Overview Linux for real-time and embedded systems KURT-Linux Origins Time keeping Event scheduling Performance Evaluation LibeRTOS Unified computation control Configurability Tool support Conclusions

ITTC IAB 4/7/05 3 Dr. Douglas Niehaus © 2005 Real-Time and Embedded Linux Linux is strongly increasing in popularity for real-time and embedded systems for several reasons Open source ensures all parts of the system are accessible to the system developers Local archive of entire system Long term stability Simple licensing restrictions: GPL and LGPL Application software unrestricted Low cost Mature programming tools: GCC, GDB, binutils, busybox

ITTC IAB 4/7/05 4 Dr. Douglas Niehaus © 2005 Real-Time and Linux Two basic methods for real-time computation using Linux Two OS method Single OS method Two OS method establishes a RT OS on top of the hardware and then runs Linux under the RTOS control Linux is generally a non-real-time best effort computation RTLinux and RTAI use this approach Relatively simple way to provide accurate scheduling support for simple RT computations Drawback is that access to Linux services is more complex and RTOS services must all be implemented

ITTC IAB 4/7/05 5 Dr. Douglas Niehaus © 2005 Real-Time and Linux Single OS approach modifies Linux internals to make it appropriate for real-time Harder to do but it permits use of all Linux system services much easier KURT-Linux was the first system to use this approach Monta Vista and Timesys are commercial companies using this architectural approach Several components of Linux need to consider RT Interrupt handling, thread scheduling, concurrency control, preemption, kernel computations Single OS solutions now approach two OS performance

ITTC IAB 4/7/05 6 Dr. Douglas Niehaus © 2005 KURT-Linux Project started roughly eight years ago Initial areas of concentration were: Time keeping Event scheduling Performance evaluation Simple programming model with explicit plan scheduling Execution intervals for computations were explicitly specified on a cyclically repeating time line Adequate for demonstrating microsecond resolution soft-real-time scheduling of fairly simple computations

ITTC IAB 4/7/05 7 Dr. Douglas Niehaus © 2005 KURT-Linux:Time Keeping Vanilla Linux uses a periodic interrupt to keep time and to control the occurrence of explicitly scheduled events KURT-Linux separated time keeping and event interrupts Uses x86 Time Stamp Counter (TSC) as time standard TSC ticks with CPU clock frequency Can be calibrated against an Network Time Protocol standard within parts per billion On a 1 GHz Pentium machine Provides fine grain local clock to applications and performance evaluation framework

ITTC IAB 4/7/05 8 Dr. Douglas Niehaus © 2005 KURT-Linux: Event Scheduling Explicitly schedules each event interrupt using microsecond (x86) resolution PC timer chip Simple and portable internal API: ARM, PPC Resolution varies with the platform Explicit plan model is a simple way to ensure that execution of periodic real-time computations is controlled accurately Within microseconds to tens of microseconds depending on configuration options and CPU clock speed However 10 2 outlier events still occurred per 10 7 events Interrupt blocking and lack of kernel preemption

ITTC IAB 4/7/05 9 Dr. Douglas Niehaus © 2005 KURT-Linux:Performance Evaluation Basic principle of real-time and embedded system development is that the accuracy of the performance evaluation methods must keep pace with the accuracy of the behavior control desired KURT-Linux thus required performance evaluation methods much more accurate than vanilla Linux provided Data Streams sub-system addresses this Pseudo-device driver for kernel event recording Utility library for application instrumentation Unified application/OS performance evaluation and analysis methods permits end-to-end computation view

ITTC IAB 4/7/05 10 Dr. Douglas Niehaus © 2005 Data Stream Overview

ITTC IAB 4/7/05 11 Dr. Douglas Niehaus © 2005 Data Streams Basic data type is the event Time stamp (TSC), unique ID, context information Instrumentation points in OS or application code produce an event when a thread of control cross them Aggregate data types have lower overhead Histograms and counters Data added to aggregate by instrumentation points Administrative events can include snapshots of aggregate data in the output event stream during an experiment as well as at the end Instrumentation points can be individually enabled

ITTC IAB 4/7/05 12 Dr. Douglas Niehaus © 2005 Data Streams Powerful post processing is key to analysis of often huge data sets gathered from and experiment Post-processing filters are easy to use and to write Python based implementation GUI tool support for filter configuration Constraint checking and scenario specific filters useful Visualization is also quite useful for understanding higher level patterns of computation behavior Detection of unusual situations Data streaming across sockets especially useful for embedded systems

ITTC IAB 4/7/05 13 Dr. Douglas Niehaus © 2005 Pipeline Processing Visualization

ITTC IAB 4/7/05 14 Dr. Douglas Niehaus © 2005 LibeRTOS Collaborative project with: Linutronix, several industrial automation customers in Germany, and Washington U. Industrial-use version of KURT-Linux KURT-Linux should remain research version and venue LibeRTOS will receive updates from research efforts LibeRTOS industrial driving problems enrich KURT- Linux research and development LibeRTOS motivated feature development Unified Computation Control: Group Scheduling Model Configurability of many aspects of system semantics Tool support

ITTC IAB 4/7/05 15 Dr. Douglas Niehaus © 2005 LibeRTOS: Unified Computation Control Motivation: many application computations are implemented by groups of computational components Emphasizes flexible representation and control of entire computations, including internal Linux components Internal Linux computation types are “off-the-books” Linux internal types: hard-IRQ, soft-IRQ, tasklets Execution manifests as noise in thread scheduling models Key Group Scheduling issues Customizable scheduling of applications End-to-end scheduling of all computation components Precise fine-grain resource control and partitioning

ITTC IAB 4/7/05 16 Dr. Douglas Niehaus © 2005 Group Scheduling (GS) Model Simple form of hierarchical scheduling permits composition of a wide variety of scheduling semantics Middleware version for portability, OS for complete control GS controls arbitrary groups of computational components Threads: OS and Middleware versions Hard-IRQ, soft-IRQ, tasklet, bottom-halves only under OS Groups nest to form scheduling hierarchy Scheduling Decision Function (SDF) associated with each group selects among members when asked Application specific semantics can be created by implementing an SDF embodying those semantics

ITTC IAB 4/7/05 17 Dr. Douglas Niehaus © 2005 Group Scheduling Model Generally we use decision trees but could be DAGs Scheduling Decision Tree (SDT) is any graph subset System Scheduling Decision function (SSDF) is SDT controlling system as a whole ● This SSDF illustrates the “first refusal” semantics for GS under KURT-Linux/LibeRTOS ● Only desired computations are under GS others default to Linux vanilla scheduler control for convenience

ITTC IAB 4/7/05 18 Dr. Douglas Niehaus © 2005 Group Scheduling Model Emphasizes modularity and configurability Controls selected computations Does not require specification for all computations SSDF can clearly represent desired semantics Users can define SDFs Functions in MW, Kernel modules in OS Application centric and state-aware SDF use was the driving problem for this paper SSDF integrates several scheduling constraints

ITTC IAB 4/7/05 19 Dr. Douglas Niehaus © 2005 Group Scheduling Target Examples Many computations have multiple components and it is the computations developers really wish to schedule Ethernet Time Division Multiplexing on a LAN Data Streams instrumentation effect reduction Rodent behavioral experiment data collection & control Real-time graphics calculation and display Coordinate execution of threads doing calculation with execution of the X-server thread doing the display Application-aware scheduling Balanced progress of multiple multi-threaded pipelines for video processing, sensor fusion, or multi-track audio

ITTC IAB 4/7/05 20 Dr. Douglas Niehaus © 2005 Application Aware Driving Problem Based on DARPA PCES driving problem Critical and non-critical streams of data frames Each stream is processed by a pipeline of threads Processing time of each frame varies By frame and pipeline stage Balanced progress by many streams is desired Multiple stream video monitoring Sensor fusion and control loops in more general terms Difficult to implement without an algorithm using application state information System where progress of a set of computations matters

ITTC IAB 4/7/05 21 Dr. Douglas Niehaus © 2005 Application Aware Driving Problem

ITTC IAB 4/7/05 22 Dr. Douglas Niehaus © 2005 Scheduling Scenario 1 1. Preference to Critical Streams 2. Frames move through pipelines as fast as possible 3. Receiving frames more important than processing 4. Non-critical streams under default Linux scheduling

ITTC IAB 4/7/05 23 Dr. Douglas Niehaus © 2005 Scheduling Scenario 2 1. Preference to Critical Streams 2. Frames move through pipelines as fast as possible 3. Receiving frames more important than processing 4. Equal execution opportunity for non-critical streams

ITTC IAB 4/7/05 24 Dr. Douglas Niehaus © 2005 Is Round Robin Enough? As implemented in the RR SDF, each non-critical stream received opportunity to use equal CPU share We could have made it equal consumption Under either policy, the variable processing time of each pipeline stage for each frame means that frame progress is not balanced Such variable execution behavior is common to many groups of computations Need for behavior to meet specific and unchanging constraints despite variable behavior also common Application aware scheduling addresses this

ITTC IAB 4/7/05 25 Dr. Douglas Niehaus © 2005 Scheduling Scenario 3 1. Preference to Critical Streams 2. Frames move through pipelines as fast as possible 3. Receiving frames more important than processing 4. Balanced Progress of non-critical Streams

ITTC IAB 4/7/05 26 Dr. Douglas Niehaus © 2005 Scenario 1 – Linux Scheduler

ITTC IAB 4/7/05 27 Dr. Douglas Niehaus © 2005 Linux Scheduler Conclusions Linux scheduler is oblivious to non-critical stream Existence Pipeline component relationships Frame progress over various streams thus varies widely Complete failure to produce desired application behavior

ITTC IAB 4/7/05 28 Dr. Douglas Niehaus © 2005 Scenario 2 – Round Robin SDF

ITTC IAB 4/7/05 29 Dr. Douglas Niehaus © 2005 Round Robin Scheduling Conclusions Non-critical streams under GS control Control computations as groups of threads Gives much more rational execution behavior RR gives streams opportunities to use roughly equal portions of the left over CPU RR is, however, unaware of progress by the application so progress is not balanced across streams Complete failure to produce desired application behavior

ITTC IAB 4/7/05 30 Dr. Douglas Niehaus © 2005 Scenario 3 – Frame Progress SDF

ITTC IAB 4/7/05 31 Dr. Douglas Niehaus © 2005 Frame Progress Scheduling Conclusions Substitution of the FP SDF makes the SSDF aware of the non-critical application state Extremely tight balance between progress of all streams since the SDF has access to published application state Application aware SDF is a simple and effective way to produce the desired application semantics

ITTC IAB 4/7/05 32 Dr. Douglas Niehaus © 2005 LibeRTOS: Configurability Group scheduling provides flexible scheduling semantics configurability for precise computation component control Interrupt enable/disable software semaphore control Reduces interrupt response latency Permits configurability of IRQ handling semantics and integration under the group scheduling model Similar treatment of soft-IRQ and tasklet control permits configuration and integration under group scheduling Loadable module can completely replace default vanilla Linux semantics with the desired programming model Concurrency control in OS is a remaining issue

ITTC IAB 4/7/05 33 Dr. Douglas Niehaus © 2005 LibeRTOS: Tool Support Data Streams Event name space GUI editor Post processing filter configuration GUI editor Used as EECS 448 Software Engineering projects (Sp04) Experiment data collection configuration GUI editor Group Scheduling Hierarchy configuration language, GUI editor, and system software interface to OS and MW group schedulers GUI is EECS 448 (SPR05) driving problem Eclipse Data Streams and Group Scheduling plugins Under development

ITTC IAB 4/7/05 34 Dr. Douglas Niehaus © 2005 Project Support Portions of the work described here have been supported by a variety of sources and projects including: Sprint NSF E&HS program DARPA PCES program DARPA ANTS program Linutronix (Eclipse plugin prototypes) NIH (Behavior Experiment support)

ITTC IAB 4/7/05 35 Dr. Douglas Niehaus © 2005 Conclusions Linux is an increasingly popular platform for real-time and embedded systems for a variety of strong reasons KURT-Linux/LibeRTOS addresses many of the existing constraints keeping Linux from addressing an even wider range of distributed real-time and embedded applications Configurability of the system makes it applicable to a variety of application areas beyond real-time Real-time scheduling and behavior semantics is now simply one configuration choice among many Real-time embedded systems will remain a focus but will not be the only one

ITTC IAB 4/7/05 36 Dr. Douglas Niehaus © 2005 Future Work Unification of concurrency control interface in Linux Will simplify code, making analysis and study easier Will provide clean structure within which to integrate OS concurrency control within LibeRTOS flexibly configurable framework Will permit much cleaner structure within which to integrate OS concurrency control and group scheduling Proxy execution Simple accounting will track semaphore holders Integration with group scheduling will prevent deadlock when preemption of threads holding locks is permitted

ITTC IAB 4/7/05 37 Dr. Douglas Niehaus © 2005 Future Work Consolidation of end-to-end scheduling for computation components on real-time and embedded endsystems Extension of end-to-end scheduling to sets of distributed computation components crossing endsystem boundaries Integration of formal modeling of distributed real-time and embedded computation components to OS components Extend current Washington University PhD thesis Explore use of LibeRTOS component oriented configurability for HW/SW co-design of OS and application support and integration with group scheduling

ITTC IAB 4/7/05 38 Dr. Douglas Niehaus © 2005 Future Work Apply KURT-Linux/LibeRTOS to various target applications areas Behavioral science experiment control and data collection Dr. Fowler at KU and Dr. Snyder at Washington University Video processing: Dr Gauch (KU) and Dr. Fowler Performance evaluation of GRID computations CHARMM distributed molecular dynamics simulation Drs. Clark, Kuczera, Houdonougbo Apply group scheduling to improve performance