1. Multidimensional QoS Management in Distributed Real-time & Embedded Systems
Aniruddha Gokhale
Assistant Professor
ISIS, Dept. of EECS
Vanderbilt University
Nashville, Tennessee
April 30 & May 1, 2007
Colorado State U & TechX Corporation

2. Distributed Real-time & Embedded (DRE) Systems
Network-centric and large-scale “systems of systems”
e.g., industrial automation, emergency response
Satisfying tradeoffs between multiple (often conflicting) QoS demands
e.g., secure, real-time, reliable, etc.
Regulating & adapting to (dis)continuous changes in runtime environments
e.g., online prognostics, dependable upgrades, keep mission critical tasks operational, dynamic resource mgmt
Jai, here are my comments on this slide:
Please redraw the example application to focus on something from the context of ARMS than a general-purpose enterprise system. The reasons for this are that (1) Ken Birman works in the space of general-purpose enterprise systems & he’s got a LOT more experience/clout in the area of survivability than we do at this point so it’s pointless to try & go up against him head-to-head since he’s got ~5-10 big successes in this area over the past two decades & (2) we should focus on things that we really understand better than focus like Ken, i.e., namely enterprise DRE systems, such as ARMS. Please see
Please try to emphasize multi-dimensional QoS properties explicitly in the middle set of bullets in these slides. It’s implicit in what you say here, but please make it explicit since it ought to be a key focus to differentiate from the kinds of work that Ken does.
DRE systems developed via robust and reliable system composition and integration of services and applications

3. Challenges in Realizing DRE Systems
Variability in the problem space (domain expert role)
Functional diversity
Composition, deployment and configuration diversity
QoS requirements diversity
Mapping problem artifacts to solution artifacts is hard
Variability in the solution space (systems integrator role)
Diversity in platforms, languages, protocols & tool environments
Enormous accidental & inherent complexities
Continuous evolution & change

4. Sources of Variability Impacting QoS (1/3)
Understand the sources of variability in the problem/solution space that impacts QoS
Find solutions to automate the mapping of the problem space to solution space
Per-component concerns
Problem space: Functionality often supplied as third party COTS
Solution space: Implementations for different platforms and languages
P->S mapping challenges:
Implementations must match platform and language choice
Choice of implementation impacts end-to-end QoS
Communication concerns
Problem space: Choice of communication paradigms (pub-sub, RPC, MOM) and requirements on QoS
Solution space: Diversity in communication mechanisms e.g, CORBA IIOP, Java RMI, DDS, Event channels, and QoS mechanisms e.g., DiffServ, IntServ, MPLS
Decoupling application logic from provisioning communication QoS
Right optimizations in communication paths needed to enhance QoS
Redundancy and mixed mode communications needed to support QoS

5. Sources of Variability Impacting QoS (2/3)
Assembly concerns
Problem space: Discovering services and composing them together
Solution space: Interfaces and their semantics must match
P->S mapping challenges:
Must ensure functional and systemic compatibility in composition
Heterogeneity in platforms and languages in composition impacts QoS
Deployment concerns
Problem space: Need resources e.g., CPU, memory, bandwidth, storage
Solution space: Diversity in resource types and their configurations
How to select and provision resources such that end-to-end QoS is realized?
How to (re)allocate resources to maintain end-to-end QoS?
How to place components so that near optimal QoS tradeoffs are made?
How to allow sharing of components or assemblies?

6. Sources of Variability Impacting QoS (3/3)
Configuration concerns
Problem space: Multiple QoS requires right set of configurations
What is the unit of failover? What is the replication degree? Are entire assemblies replicated?
How to define access control rights?
Determining system task partitioning, schedulability
Defining network resource needs
Solution space: Platforms provide high degree of configuration flexibility
P->S mapping challenges:
What configuration options control a specific dimension of QoS?
How do different configuration options interact with each other?
How to configure heterogeneous platforms?
What impact deployment decisions have on configurations?

7. A Single Perspective: Tangling of QoS Issues
Demonstrates numerous tangled para-functional concerns
Significant sources of variability that affect end-to-end QoS
QoS concerns tangled across system lifecycle
Lifecycle-wide separation of concerns (SoC) & variability management is the key
Design-time
Deployment-time
Run-time

8. Requirements of a Solution Approach
Need expressive power to define QoS intent in the problem space, and perform design-time analysis
e.g., network QoS needs of application flows or end-to-end latencies
Need to decouple DRE system functionality from systemic capabilities in the solution space
e.g., decouple robot manager logic from fault tolerance configurations
Need to automate the mapping from problem to solution space
e.g., automate the (re)deployment and (re)configuration of system functionality to maintain QoS
Need to provide a hosting platform with dynamic adaptation capabilities
e.g., survivability management of robot manager

9. QoS Management via Multistage Architecture
Multiple levels of abstraction required for resolving tangling of QoS issues
Model driven engineering (MDE) provides intuitive, variability management in the problem space
Resource allocation and control framework decouples problem space from solution space
Deployment and configuration tools transparently map problem space QoS needs to solution space artifacts
QoS-enabling component middleware provides hosting environment and runtime adaptation features

10. Our MDE Solution: CoSMIC Toolsuite
Work supported by DARPA PCES & ARMS Programs
CoSMIC tools e.g., PICML used to model application components, CQML for QoS
Captures the data model of the OMG D&C specification
Synthesis of static deployment plans for DRE systems
Capabilities being added for QoS provisioning (real-time, fault tolerance, security)
CoSMIC can be downloaded at

11. Expressing Multidimensional QoS Intent
A QoS Modeling Language

12. Component QoS Modeling Language (CQML)
Objectives:
Express QoS design intent
Model crosscutting QoS concerns
Perform design-time QoS tradeoff analysis
Challenges:
Intuitive QoS abstractions
Separation as well as unification of QoS concerns
4 types of QoS modeling capabilities
Real-time
Fault tolerance
Security
Network QoS
Developed as overlay on a composition modeling language
Facility to integrate behavioral modeling
Unified internal mechanism for QoS integration
Pluggable back end analysis tools for QoS tradeoff analysis
e.g., Times tool

13. Example of Failover Unit Intent
Primary Component A B C
primary IOR
“Client”
container/component server
container/component server
container/component server
Replica FOU A’ B’ C’
secondary IOR
container/component server
Primary FOU
Replica Component

14. Example of Shared Risk Group Intent
Ship_SRG
DataCenter1_SRG
DataCenter2_SRG
Rack1_SRG
Rack2_SRG
Node1
(blade31)
Node2
(blade32)
Shelf1_SRG
Shelf2_SRG
Shelf1_SRG
Blade30
Blade34
Blade29
Blade33
Blade36

15. Example Replica Placement Intent
Define N orthogonal vectors, one for each of the distance values computed for the N components (with respect to a primary) and vector-sum these to obtain a resultant.  Compute the magnitude of the resultant as a representation of the composite distance captured by the placement . 
Compute the distance from each of the replicas to the primary for a placement. 
Record each distance as a vector, where all vectors are orthogonal.
Add the vectors to obtain a resultant.
Compute the magnitude of the resultant.
Use the resultant in all comparisons (either among placements or against a threshold)
Apply a penalty function to the composite distance (e.g. pair wise replica distance or uniformity) R1 P R2 R3

16. CQML FT Modeling Abstractions
QoS Enhancements to PICML (Platform Independent Component Modeling Language) Metamodel
Fail-over Unit (FOU): Abstracts away details of granularity of protection (e.g. Component, Assembly, App-string)
Replica Group (RPG): Abstracts away fault-tolerance policy details (e.g. Active/passive replication, state-synchronization, topology of replica)
Shared Risk Group (SRG): Captures associations related to risk. (e.g. shared power supply among processors, shared LAN)
Protection granularity concerns
State-synchronization concerns
Component Placement constraints
Model Interpreter (component placement constraint solver): Encapsulates an algorithm for component-node assignment based on replica distance metric
Replica Distance Metric

17. CQML RT & NetQoS Modeling Abstractions
Real-time modeling provided by the RT-Model element
Leverages known patterns of real-time programming usage:
e.g., real-time CORBA
Enables modeling of:
Priority models
Banded connections
Thread pools
Thread pools with Lanes
Resources e.g., CPU, memory
Network QoS modeling allows modeling QoS per application flow
Classification into high priority, high reliability, multimedia and best effort classes
Enables bandwidth reservation in both directions
Client propagated or server declared models

18. Modeling & Generative Steps in CQML
Model components and application strings in PICML
e.g., Model Fail Over Units (FOUs) and Shared Risk Groups (SRGs) using CQML
Generate deployment plan
GME/PICML
Model Information
Domain, Deployment, SRG, and FOU
injection
Replica Placement Algorithm
model
FT Interpreter
Augmented Deployment Plan
Interpreter automatically injects
replicas and associated CCM IOGRs
5. Distance-based constraint algorithm determines replica placement in deployment descriptors.

19. Example: FT Requirements Modeling in CQML
Component FOU
Replication Style = Active
Deployment plan to be augmented
LEGEND
FOU: Fail Over Unit
SRG: Shared Risk Group
Replica = 3
Min Distance = 4
Shared Risk Group (SRG) hierarchy

20. Automated Sample FT QoS Provisioning
Automatic Injection of replicas
Augmentation of deployment plan based on number of replicas
Automatic Injection of FT infrastructure components
E.g. Collocated “heartbeat” (HB) component with every protected component.
Automatic Injection of connection meta-data
Specialized connection setup for protected components (e.g. Interoperable Group References IOGR) HB
Container

21. Automated Heartbeat Component Injection
Collocated heartbeat component
Primary Component
FPC
intra-FOU
heartbeat HB HB HB A B C
“client”
primary IOR
container/component server
container/component server
container/component server
IOGR
Primary FOU
periodic FPC heartbeat
FPC
secondary IOR HB HB HB
Connection
Injection A’ B’ C’
Replica Component
container/component server
container/component server
container/component server
Replica FOU

22. Example: NetQoS Modeling in CQML
Expressing QoS intent for application flows
Multiple connections sharing QoS can use same element

23. Example: Automated Component Placement
Ship_SRG
DataCenter1_SRG
DataCenter2_SRG
Rack1_SRG
Rack2_SRG
Node1
(blade31)
Node2
(blade32)
Composite Distance
Primary
Shelf1_SRG
Shelf2_SRG
Shelf1_SRG
Blade30
Blade34
Blade29
Blade33
Blade36
Replica 3
Replica 1
Replica 2

24. QoS Analysis Requires Behavior
Enhancing CQML with Behavior Modeling

25. Adding Behavioral Modeling to CQML
Realized via metamodel composition i.e., CQML with behavioral DSMLs
Component Behavior Modeling Language (CBML)
a high-level domain-specific modeling language for capturing the behavior of components
e.g., its actions & states
Workload Modeling Language (WML)
a domain-specific modeling language for capturing workload, & used to parameterize the actions on CBML
CBML & WML were both developed using GME (

26. Integrating Behavioral and Structural DSMLs
Context
Structural QoS DSML (e.g.,CQML) capture the makeup of component-based systems and their QoS
There is no correlation between a component’s ports & its behavior
Integration Enablers
Defined a set of connector elements that allow structural DSMLs to integrate with (or contain) CBML models
Input ports directly connect to Input Action elements
Output actions have the same name as their output port to reduce model clutter
i.e., prevent many to one connections
input connections
output name

27. Unifying QoS Dimensions & Tradeoff Analysis
The CQML EventBus Architecture

28. CQML QoS Unification & Tradeoff Analysis
Leverages QoS and behavioral models
Model interpretation using an EventBus framework
Each QoS dimension publishes its modeled QoS requirements
A particular QoS dimension is chosen as the driver e.g., real-time
Acts as subscriber of events generated by other dimensions
Driver interpreter integrates other QoS dimensions making tradeoffs on the way
Pluggable back end analysis and generative tools

29. Example: QoS Tradeoff Analysis using TIMES tool
CQML EventBus architecture weaves in FT and Security concerns into RT concerns
Feeds to backend schedulability analysis tools e.g., Times

30. Model Transformations and Model Checking
Automated Problem Space to Solution Space Mapping for Configuration Options
Model Transformations and Model Checking

31. Mapping QoS Intent to Platform Configurations
9/12/2018
Mapping QoS Intent to Platform Configurations
Prioritized service invocations (QoS Policy) needs to be mapped to Banded Connection (QoS configuration)
Large gap between application QoS policies & middleware QoS configuration options
Necessary to realize the desired QoS policies
Not a straightforward mapping
Requires understanding of middleware configuration space
e.g., multiple levels of QoS requires configuring appropriate number of thread pools, threadpool lanes (server) and banded connections(client)
How do you validate the options?
How do you maintain compatibility across components? 32 32

32. Solution Approach: Model-Driven QoS Mapping
9/12/2018
Solution Approach: Model-Driven QoS Mapping
QUality of service pICKER (QUICKER)
Allows modeling of application QoS policies
Provides automatic mapping of QoS policies to configuration options
Allows validating the generated QoS options
Automated QoS mapping and validation process – can be used iteratively in the design process 33 33

33. Research Summary www.dre.vanderbilt.edu/~gokhale
Hardware
Middleware
OS & Protocols
Applications
R&D in new, holistic approaches to end-to-end QoS management in distributed real-time & embedded systems
Research Challenge
Research Approach
Benefits
Managing problem space variability
Model-driven generative approach using separation of concerns
Enhance the state-of-art in MDE and AOSD technologies
Design-time “What-if” analysis using generative prog
Variety of analysis techniques including non traditional mechanisms
Generative technologies for automated analysis
Application of Engineering Mechanics
Deployment-time intelligent decisions
New applications of constraints optimization theory
Middleware specializations
Near optimal deployment
Specialized middleware stacks
Run-time Mechanisms
Multilevel, proactive QoS mgmt schemes
Dynamic resource management
Largely autonomic
Survivable systems

34. Acknowledgments DOC Students & Staff Krishnakumar Balasubramanian
My research achievements are made possible due to the sponsors (DARPA, NSF, industry); efforts of my students (both primary and co-advisees); working with collaborators; & advise from mentors
Collaborations
Christopher Andrews
Theckla Louchios
Dr. Cemal Yilmaz
Dr. Sylvester Fernandez
Dr. Adam Porter
Thomas Damiano
Dr. Sherif Abdelwahed
Dr. Jeff Gray
Dr. Swapna Gokhale
DOC Students & Staff
Krishnakumar Balasubramanian
Arvind Krishna
Jaiganesh Balasubramanian
Emre Turkay
Jeff Parsons
Sumant Tambe
James Hill
Akshay Dabholkar
Amogh Kavimandan
Gan Deng
Joe Hoffert
George Edwards
Dimple Kaul
Arundhati Kogekar
Gabriele Trombetti
Mentors
Dr. Doug Schmidt
Dr. Janos Sztipanovits
Dr. Gabor Karsai
Dr. Joe Loyall
Dr. Joe Cross

35. Concluding Remarks www.dre.vanderbilt.edu
Significant sources of variability in problem and solution space for managing QoS in DRE systems
Multidimensional QoS management requires a multistage architecture
MDE for expressing QoS intent and backend analysis
Resource allocation to decouple application logic from systemic issues
Deployment & configuration tools to automate the mapping
Runtime frameworks for dynamic adaptation & resource mgmt
Unresolved Issues:
Automated QoS tradeoff analysis
Optimizations for performance/footprint
Dealing with heterogeneity
Dealing with system scale
Unresolved Issues:
Near optimal deployment
Impact of new technologies (multicore)
Applicability to emerging/existing areas
Cyberphysical systems
High performance computing

36. EXTRA SLIDES

37. The Component Behavior Modeling Language
Context
Component’s behavior can be classified as: communication & internal actions
Need to define these actions as close as possible to their real counterpart
Research Contributions of CBML
Based on the semantics of Input/Output (I/O) Automata
i.e., contains representative elements
Behavior is specified using a series of action to state transitions
Transitions have preconditions that represent guards & effects have postconditions that determine the new state after an action occurs
Variables are used to store state & can be used within pre & postconditions

38. Domain-Specific Extensions to CBML
Context
Some aspects of component-based systems are not first-class entities in I/O automata
e.g., lifecycle events & monitoring notification events
Extended CBML (without affecting formal semantics) to support domain-specific extensions
Domain-Specific Extensions
Environment events – input actions to a component that are triggered by the hosting system rather than another component
Periodic events - input actions from the hosting environment that occur periodically
environment
periodic

39. Ensuring Scalability of CBML
Context
One of the main goals of higher-level of abstraction is simplicity & ease of use
It is known that one of the main drawbacks of automata languages is scalability
Usability Extensions
Composite Action – contains other actions & helps reduce model clutter
Repetitions - determines how many times to repeat an action to prevent duplicate sequential actions
Log Action - an attribute of an Action element that determines if the action should be logged
Tool Specific – GME add-on that auto-generates required elements (e.g., states)

40. The Workload Modeling Language
generic action
Context
CBML as a standalone language is sufficient enough to capture behavior of a component
For emulation purposes, it does not capture the reusable objects of a component & its workload
Research Contributions of WML
Middleware, hardware, platform, & programming language independent DSML
Used to define workload generators (workers) that contains actions to represent realistic operations
Defined using a hierarchical structure that resembles common object-oriented programming packaging techniques that are consistent with conventional component technologies
no payload
executable actions
workload generator

41. Integrating WML Models with CBML Models
Context
CBML & WML are standalone DSMLs with a distinct purpose
i.e., model behavior & workload, respectively
WML is designed to complement CBML by providing CBML with reusable operations that can map to realistic operations
Integration Enablers
WML worker elements have same model semantics as variables
WML actions have same modeling semantics as CBML actions
Allows WML elements to be used in CBML models
CBML
WML action
WML

42. Challenge 2: Choosing appropriate values for QoS options
9/12/2018
Challenge 2: Choosing appropriate values for QoS options
Individually configuring components is tedious & error-prone
e.g., ~140 QoS options for MMS mission
Choosing valid values for configuration options inherently doesn’t scale well as size of application increases 43 43

43. Challenge 3: Validating QoS options
9/12/2018
Challenge 3: Validating QoS options
Each QoS option value chosen has to be validated
Every system reconfiguration (design time) should be validated
e.g., reconfiguration of bands of Analysis should be validated such that the modified value corresponds to (some) lane priority of Comm component 44 44

44. Challenge 4: Resolving QoS options dependencies(2/2)
9/12/2018
Challenge 4: Resolving QoS options dependencies(2/2)
ThreadPool priorities of Comm should match priority bands defined at Gizmo
Manually tracking dependencies difficult or in some cases infeasible
Dependent components may belong to more than one assembly
Dependency may span beyond immediate neighbors
e.g., dependency between Gizmo and Comm components
Empirically validating a change in configuration slows down design process considerably
Several iterations before desired QoS is achieved (if at all) 45 45

45. Enabling Technologies
9/12/2018
Enabling Technologies
Enhanced Platform Independent Component Modeling Language (PICML), a DSML for modeling CCM applications
QoS Mapping using Graph Rewriting and Transformation (GReAT) model transformation tool
Customized Bogor model-checker to define new types and primitives to validate QoS options
Uses model interpreters to automate generation of system specification in Bogor 46 46

46. QUICKER concepts: Transformation of QoS policies(1/2)
9/12/2018
QUICKER concepts: Transformation of QoS policies(1/2)
Representation of application QoS policies
Platform-independent semantics closely follow application QoS requirements
Representation of policies at component- or assembly-level
Representation of QoS options
Component QoS Modeling Language (CQML) captures CCM-specific QoS configuration options
RequirementProxy can be per component or assembly instance 47 47

47. QUICKER concepts: Transformations of QoS policies(2/2)
9/12/2018
QUICKER concepts: Transformations of QoS policies(2/2)
Translation of policies into QoS options
Semantic translation rules specified in terms of input (PICML) and output (CQML) type graph
QUICKER Transformation engine performs mapping of QoS policies (in PICML) to QoS configuration options (in CQML) 48 48

48. QUICKER concepts: Validation of QoS options(1/2)
9/12/2018
QUICKER concepts: Validation of QoS options(1/2)
Representation of middleware QoS options in Bogor
Bogor extensions allow representing domain-level concepts of system model
QUICKER defines new Bogor extension for QoS options
Uses Bogor Input Representation (BIR) at a higher level of abstraction
e.g., Component, lane, band etc. data types defined in QoS extensions
Reduces size of the system model by avoiding (redundant) auxiliary variables in specification 49 49

49. QUICKER concepts: Validation of QoS options(2/2)
9/12/2018
QUICKER concepts: Validation of QoS options(2/2)
Representation of properties (that a system should satisfy) in Bogor
BIR primitives define language constructs to access & manipulate domain-level data types
Used to define rules that validate options and check if property is satisfied
Used to construct dependency structure of components which is needed for validation of connected components
Automatic generation of BIR specification from CQML models
Rule determines if ThreadPool priorities at Comm match with priority bands at Analysis 50 50