1. Group Scheduling in System Software
Michael Frisbie, Douglas Niehaus
University of Kansas
Venkita Subramonian, Christopher Gill
Computer Science and Engineering
Washington University

2. Motivation
Real-time and embedded systems have widely varying computation semantics
Varying policies for controlling specific computations
Varying levels of focus for systems
Single purpose embedded systems
General purpose systems with a RT application
Multi-media, machine control and user-interface
No single policy is likely to be appropriate for all

3. Motivation
Computation in computer systems is not exclusively done under the publicly exposed scheduling model
OS computational components
Interrupts, SoftIRQs, Tasklets, and Bottom halves
Execute outside exposed scheduling policies
Manifest as noise in the scheduling model
Active middleware
Linux PThreads library, TAO
Manifest as competing load

4. Goals
Highly configurable framework within which a wide range of policies can be specified
Selection of predefined scheduling semantics
Implementation of customized schedulers
Application computation oriented representation
Representation of all computation components on system under scheduling framework
Current semantics available as default policies
Requires some new types of information

5. Platform Linux Growing popularity for real-time and embedded
Middleware version for portability and range of mechanism control
KU Real-Time Linux (KURT-Linux)
OS computation components integration
Interrupt handling modifications
Number of related projects
Data Streams Performance Evaluation Framework
Ability to gather detailed scenario-oriented data

6. Group Scheduling Application computation centric scheduling view
Computations are implemented by a group of one or more computation components
Threads, IRQ handlers, SoftIRQ, Tasklets, BH's
Flexible framework for composing and configuring the system scheduling decision function (SSDF)
SSDF chooses the computation component using the CPU at any given time
Framework explicitly supports description of both computations and relations among computations

7. Group Structure
Group: a set of computation components with an associated Scheduling Decision Function (SDF)
Elements within a group can be threads, other groups, or other computation components
Elements can belong to more than one group
Scheduling decision tree (SDT) composed of one or more groups
Control semantics for computation components
SDT for computations are composed to form the System Scheduling Decision Tree (SSDT)

8. System Scheduling Decision Tree (SSDT)
Controls the system's computation components
Explicitly or implicitly
Ultimate goal is to make all of it explicit and easily configurable across a wide semantic range
Can co-exist with the default system scheduler
Semantic hooks
SSDT invocation before default scheduler (DS)
Method of making DS skip components under SSDT control

9. First-Refusal (FR) SSDT
FR-SSDT uses a sequential SDF at the top level
SDT controlling components under group scheduling model has first refusal
Linux SDF (default scheduler) makes the decision if no component under the group model should run
Exclusion of components from DS ensures precise control as needed

10. MLFQ SDT Example
Top level priority SDF maintains the priority equivalence class view
Each priority class is a group using a round-robin policy to share the CPU among members
Dynamic priority adjustment of processes can move them among priority classes

11. Related Work Hierarchical Scheduling Regher and Stankovic (RTSS 2001)
Likely computationally equivalent (capability)
Distinguished by which abstractions are emphasized
CPU Inheritance Scheduling
Ford and Susarla (Flux Project)
Group scheduling emphasizes
Application structure reflected in groups
Integration of all computation components
Interrupts, tasklets, etc

12. Kernel Implementation
Modifies the default Linux scheduler to permit the GS framework to have a chance to choose
Hook to make default scheduler exclude a component is the most subtle change
Changes to existing code to consult the exclusion notation, rather than trying to remove the component from the base data structure
Control for components other than threads is the most significant feature for real-time systems

13. Middleware Implementation
Currently controls only threads at the user level
Part of DARPA PCES2 project
Layering on top of supplied Linux scheduler requires indirect control through available mechanisms
Separates managing and managed threads into equivalence classes to determine CPU use
Uses Fixed Priority POSIX scheduling model as implemented by Linux SCHED_FIFO

14. Middleware Implementation
Scheduler has two threads
SSDT thread selects current thread
API thread processes group operation requests
Block Catcher detects when current thread blocks
Signals SSDT thread
Uses SIGSTOP and SIGCONT to control availability of thread for execution
Model is incomplete because it cannot know when a previously blocked thread becomes unblocked

15. Thread Priority Classes
Reaper spawns scheduler and then blocks
Scheduler SSDT thread chooses current thread
API thread processes group operation requests
Block Catcher detects current thread block, signals SSDT
Non-current threads are both at lower priority and SIGSTOP
Linux threads at level 0

16. Context Switch Event Sequence
Thread A is current thread
Timer or other event blocks or pre-empts Thread A
Scheduling Thread runs and selects Thread B, blocks in nanosleep
Context switch to Thread B begins its execution

17. Middleware Implementation Tradeoffs
Portable standards based implementation
POSIX fixed priority scheduling
Socket based group API access
Significantly greater context switch delay compared to existing kernel based implementation
SSDT thread context switch and Block Catcher as well if current thread blocks
Most significant need is SSDT thread notification that a threads unblocks
Scheduler Activations

18. Performance Evaluation
Metric
Scheduling overhead
Context switch latency (A to B)
Parameters
Number of Processes
CPU bound or I/O bound
User/Kernel Implementation
Others
Signal delivery details and semantics

19. Context Switch Overhead - Kernel

20. Kernel Performance Constant with respect to CPU or I/O bound
Considerably lower than MW version
Simple SSDT
Does not require signal delivery

21. Context Switch Overhead – Compute Bound – MW

22. Context Switch Overhead – Blocking – MW

23. Middleware Performance
Different with respect to CPU or I/O bound
Requires signal delivery
Block Catcher mechanism adds latency
Considerably higher than kernel version
Simple SSDT
Some extension to existing system semantics required for completeness
Unblocking notification upcall

24. Group Scheduling – Summary
Provides a flexible control framework
Within which resource control and
Distributed end-to-end scheduling constraints can be expressed and enforced
Portable middleware version
Limited by lack of unblocking notification upcall
Implementation under KURT-Linux is simple
ACE system call wrappers
VxWorks threads state change notification

25. Current Status
Integration of all KURT-Linux OS computational components under group scheduling framework
Recently completed
Michael Frisbie’s Master’s Thesis topic
We are currently working on Group Scheduling control of service classes in
Event Channel
TAO based computations
Includes control of middleware threads, queues, etc.

26. Future Work
Middleware use of group scheduling to provide support for service classes in Event Channel and TAO
Concurrency constraint representation in KURT- Linux to permit fine grain computation component control under group scheduling
Experimentation with application aware scheduling decision functions
Integrated DSKI/DSUI instrumentation to diagnose/deduce scheduling-related optimizations and fine-grain points of inefficiency (cruft sleuthing)