Model Driven Engineering for QoS Management in Distributed Real-time & Embedded Systems Dr. Aniruddha S. Gokhale Assistant Professor, EECS Vanderbilt University Nashville, Tennessee Institute for Software Integrated Systems Presented at Avaya Labs April 14 th, 2006

2 DOC Group Research CoSMIC - Modeling Deployment & Configuration (D&C) crosscutting concerns CIAO – QoS-enabled component middleware DAnCE – Deployment And Configuration Engine RACE – Resource and Control Engine

3 Notion of Enterprise Distributed Real-time & Embedded (DRE) Systems Real-time & embedded systems Deeply embedded Resource constrained Fixed and often static operational modes and environments Real-time & embedded systems Deeply embedded Resource constrained Fixed and often static operational modes and environments The Past Distributed real-time & embedded (DRE) systems Network-centric & larger-scale “systems of systems” Composable from COTS components QoS requirements a mix of enterprise & real-time Dynamic operational modes and environment Distributed real-time & embedded (DRE) systems Network-centric & larger-scale “systems of systems” Composable from COTS components QoS requirements a mix of enterprise & real-time Dynamic operational modes and environment The Future

4 Components encapsulate application “business” logic Components interact via ports Provided interfaces, e.g.,facets Required connection points, e.g., receptacles Event sinks & sources Attributes Containers provide execution environment for components with common operating requirements Components/containers can also Communicate via a middleware bus and Reuse common middleware services SecurityReplicationNotificationPersistence SchedulingA/V StreamingLoad Balancing … Container … … Middleware Bus Container … Technology Enablers: Component Middleware “Write Code That Reuses Code”

5 Key challenges in the solution space Enormous accidental & inherent complexities Continuous evolution & change Highly heterogeneous platform, language, & tool environments New Demands on DRE Systems Key challenges in the problem space Network-centric, dynamic, very large- scale “systems of systems” Stringent simultaneous quality of service (QoS) demands Highly diverse & complex problem domains Mapping problem artifacts to solution artifacts is very problematic

6 Promising Solution: Model Driven Engineering (MDE) Develop, validate, & standardize generative software technologies that: 1. Model 2. Analyze 3. Synthesize & 4. Provision multiple layers of middleware & application components that require simultaneous control of multiple quality of service properties end-to-end Specialization is essential for inter-/intra-layer optimization & advanced product-line architectures Middleware Services DRE Applications Operating Sys & Protocols Hardware & Networks <…events this component supplies…> <…events this component supplies…> Goal is not to replace programmers per se – it is to provide higher-level domain-specific languages for middleware/application developers & users

7 MDE Tool Developer (Metamodeler) Application Developers (Modelers) Technology Enabler: Generic Modeling Environment (GME) “Write Code That Writes Code That Writes Code!” Decorator GModel GMeta CORE Metamodel XML Paradigm Definition Storage Options … DB #n DB #1 XML … UML / OCL COM XML ODBC Constraint Manager Browser Translator(s) Add-On(s) GME Editor GME Architecture Goal: Correct-by-construction DRE systems

8 Specification & Implementation Defining, partitioning, & implementing appln functionality as standalone components Assembly & Packaging Bundling a suite of software binary modules & metadata representing app components Installation Populating a repository with packages required by app Configuration Configuring packages with appropriate parameters to satisfy functional & systemic requirements of an application without constraining to physical resources Planning Making deployment decisions to identify nodes in target environment where packages will be deployed Preparation Moving binaries to identified entities of target environment Launching Triggering installed binaries & bringing appln to ready state QoS Assurance & Adaptation Runtime (re)configuration & resource management to maintain end-to-end QoS OMG Deployment & Configuration (D&C) specification (ptc/ ) DRE Lifecycle Stages Affecting QoS

9 Our MDE Solution: CoSMIC CoSMIC tools e.g., PICML used to model application components Captures the data model of the OMG D&C specification Synthesis of static deployment plans for DRE applications New capabilities being added for QoS provisioning (real-time, fault tolerance) CoSMIC can be downloaded at

10 Nav Sensors Expendable Management Data Links Mission Computer Vehicle Mgmt Expendables Avionics mission computing product-line architecture for Boeing aircraft, e.g., F-18 E/F, 15E, Harrier, UCAV DRE system with 100+ developers, 3,000+ software components, 3-5 million lines of C++ code Based on COTS hardware, networks, operating systems, & middleware Used as Open Experimentation Platform (OEP) for DARPA IXO PCES, MoBIES, SEC, MICA programs Bold Stroke Architecture Hardware (CPU, Memory, I/O) Networking Interfaces Operating System Middleware Infrastructure Mission Computing Services Radar DRE System Example: Boeing Bold Stroke

11 (I) The Packaging Aspect Application components are bundled together into assemblies Several different assemblies tailored towards delivering different end-to- end QoS and/or using different algorithms can be part of the package e.g., large-scale DRE systems require 100s-1,000s of components Packages describing the components & assemblies can be scripted via XML descriptors

12 Packaging Aspect Problems (1/2) RT Event Channel Ad hoc techniques for ensuring component syntactic & semantic compatibility Distribution & deployment done in ad hoc manner Ad hoc means to determine event notification support

13 Packaging Aspect Problems (2/2) XML file in excess of 3,000 lines, even for medium sized scenarios Existing practices involve handcrafting XML descriptors Modifications to the assemblies requires modifying XML file

14 Platform-Independent Component Modeling Language (PICML) Developed in Generic Modeling Environment (GME) Core of Component Synthesis using Model-Integrated Computing (CoSMIC) toolchain Capture elements & dependencies visually Define “static semantics” using Object Constraint Language (OCL) Define “dynamic semantics” via model interpreters Also used for generating domain specific meta-data “Correct-by-construction” CoSMIC MDE Solution for Packaging Aspect

15 Example Metadata Generated by PICML Component Interface Descriptor (.ccd) Describes the interface, ports, properties of a single component Implementation Artifact Descriptor (.iad) Describes the implementation artifacts (e.g., DLLs, OS, etc.) of one component Component Package Descriptor (.cpd) Describes multiple alternative implementations of a single component Package Configuration Descriptor (.pcd) Describes a configuration of a component package Top-level Package Descriptor (package.tpd) Describes the top-level component package in a package (.cpk) Component Implementation Descriptor (.cid) Describes a specific implementation of a component interface Implementation can be either monolithic- or assembly-based Contains sub-component instantiations in case of assembly based implementations Contains inter-connection information between components Component Packages (.cpk) A component package can contain a single component A component package can also contain an assembly Based on OMG (D&C) specification (ptc/ )

16 Example Output from PICML GPS Implementation 154cf3cd e92-b19b-8c2c921fea38 GPS Implementation artifacts... GPS-RateGen Refresh a_GPS Pulse a_RateGen NavDisplay-GPS Refresh a_NavDisplay Ready a_GPS... Describes a specific implementation of a component interface Contains inter-connection information between components A Component Implementation Descriptor (*.cid) file

17 (II) The Multilayer Configuration Aspect Configuration of components & assemblies Configuration of component containers Configuration of the middleware bus Configuration of component server Configuration of event notifiers Component-based software must be configurable at many levels e.g., application components & containers, component servers, & middleware services & infrastructure

18 Challenge: The M/W Bus Configuration Component middleware is characterized by a large configuration space that maps known variations in the application requirements space to known variations in the middleware solution space Hook for the concurrency strategy Hook for the request demuxing strategy Hook for marshaling strategy Hook for the connection management strategy Hook for the underlying transport strategy Hook for the event demuxing strategy

19 server object management middleware Challenge: Configuring Container Policies Existing techniques for metadata configurations rely on ad hoc manual configurations e.g., CORBA server-side programming Determine the server object management policies Determine right buffer sizes Determine thread pool sizes; how are they shared; number of lanes and their priorities; if borrowing is enabled Determine various middleware policies for server objects e.g., security, lifetime, replication This “glue code” is traditionally handcrafted Ensure semantic compatibility among chosen configurations Determine end-to-end priority propagation model to use

20 CoSMIC MDE Solutions for Configuration Aspect Approach: Develop an Options Configuration Modeling Language (OCML) using GME OCML is used by Middleware developer to design the configuration model Application developer to configure the middleware for a specific application OCML metamodel is platform- independent OCML models are platform-specific

21 Applying OCML to CIAO+TAO Configuration space Constraints Middleware developers specify

22 Applying OCML to CIAO+TAO Middleware developers specify Configuration space Constraints Application developers provide a model of desired options & their values, e.g., Middleware network resources Concurrency & connection management strategies Constraint checker flags incompatible options

23 Applying OCML to CIAO+TAO Middleware developers specify Configuration space Constraints Application developers provide a model of desired options & their values, e.g., Middleware network resources Concurrency & connection management strategies Constraint checker flags incompatible options Synthesizes XML descriptors for middleware configuration Generates documentation for the middleware configuration Validates the configurations

24 (III) Deployment Planning Aspect Determine current resource allocations on target platforms Select the appropriate package to deploy on selected target Select appropriate target platform to deploy packages Component integrators must make appropriate deployment decisions, including identifying the entities (e.g., CPUs) of the target environment where the packages will be deployed

25 Planning Aspect Problems How do you determine current resource allocations? How do you ensure that the selected targets will deliver required QoS How do you correlate QoS requirements of packages to resource needs How to ensure a particular deployment configuration maximizes QoS

26 Example: Fault Tolerance Planning CoSMIC (PICML) enhancements to represent failover units and replication groups capture requirements, such as replication degrees, heartbeat intervals, importance, replication style, state synchronization mechanisms FOU and requirements

27 Modeling Shared Risk for FT Planning Model shared risk group elements in a system e.g., ship comprising regions, data centers, shelves, racks and blades Generative programming to synthesize deployment plans for FOUs that minimize shared risk (e.g., co-failure probability) system wide risk hierarchy

28 (IV) Quality Assurance & QoS Validation Aspect Existing Quality Assurance Processes: Auto-build scoreboard systems e.g. Mozilla’s Tinderbox Error reporting based on prepackaged installation tests e.g. GNU GCC and ACE & TAO Online crash reporting e.g. Netscape’s QFA and Microsoft’s Watson Shortcomings: inadequate, opaque, inefficient and inflexible QA processes Scope: Generally restricted to functional testing and often incomplete Documentation: No knowledge of what has or hasn’t undergone QA Control: Developers have no control over the QA processes Adaptation: Can’t learning from the earlier test results Persistent Challenges Scale Massive configuration & optimization space Time to market pressure Rapid updates, frequent releases Heterogeneity Scattered resources & distributed developers Incomplete information Unobservable usage context Emerging Opportunities Leverage remote computing resources and network ubiquity for distributed, continuous QA

29 CoSMIC MDE Solution for QA Aspect BGML Workflow 1.End-user composes the scenario in the BGML modeling paradigm 2.Associate QoS properties with this scenario, such as latency, jitter, or thruput 3.Synthesize the appropriate test code to run the experiment & measure the QoS 4.Feedback metrics into models to verify if system meets appropriate QoS at design time Approach: Develop an Benchmark Generation Modeling Language (BGML) w/GME BGML provides a model-based toolsuite to empirically evaluate & refine deployment configurations to maximize application QoS Use in conjunction with OCML

30 Resolving Planning Challenges with BGML (1/2) Example Scenario Navigation Display – displays GPS position updates GPS Component – generates periodic position updates Airframe Component – processes input from the GPS component and feeds to Navigation display Trigger Component – generates periodic refresh rates Goal Determine the configuration settings for the individual components to achieve the QoS required for this scenario Step1: Model component Interaction using BGML Step2: Configure each component, associating the appropriate IDL interfaces

31 Step3: Associate operations with the component ports Step4: Associate QoS metrics to measure in the scenario BGML tool allows actual composition of the target interaction scenario, auto-generates benchmarking code Each configuration option can then be tested to identify the configuration that maximizes the QoS for the scenario These empirically refined configurations can be reused across applications that have similar/same application domains These configurations can be viewed as Configuration & Customization (C&C) patterns void start_time_probe () { if (!timer_count) test_start = ACE_OS::gethrtime (); start_timer_probe = ACE_OS::gethrtime (); } void stop_time_probe () { finish_timer_probe = ACE_OS::gethrtime (); history.sample (finish_timer_probe - start_timer_probe); if (++ timer_count == niterations) { test_end = ACE_OS::gethrtime (); ACE_DEBUG ((LM_DEBUG, "test finished\n")); ACE_Throughput_Stats::dump_throughput ("Total", gsf, test_end - test_start, stats.samples_count ()); } Resolving Planning Challenges with BGML (2/2)

32 Configuration Model Adaptation strategies e.g., nearest neighbor Sub-task Code Automatic characterization e.g., classification trees Intelligent Steering Agent (ISA) Visualization e.g., scoreboard The Skoll DCQA System register client kit subtask req. subtask subtask res. Skoll Clients Skoll Server(s) Configuration Model Intelligent Steering Agent (ISA) Intelligent Steering Agent Adaptation Strategies … Subtask Results Configuration Model Client Characteristic Output: Subtask Subtask CodeAdaptation strategies e.g., nearest neighbor Automatic characterization e.g., classification trees Visualization e.g., scoreboard

33 Example: ACE+TAO+CIAO Build Scoreboard Nerve center of quality control scoreboard Updated by automatic builds 40+ builds on ~8 OS platforms Many configurations tested Participating machines located at Washington University UC Irvine BBN Technologies Remedy, Netherlands Object Sciences Corps Hunleth.com Riverace Corp ISIS/Vanderbilt PrismTechnologies Object Computing Inc Monitored by the “build-czar” & by all core developers

34 (V) Adaptation & Resource Management Aspect CONTEXT Appln Server Static configuration Need to provide runtime QoS adaptation along functional path Made feasible using adaptive & reflective middleware Global, dynamic resource management

35 RACE Control Architecture Static Inputs Resource (CPU) utilization set-point 0.8 End-to-end deadlines 800 Dynamic Inputs Current CPU utilization End-to-end execution time

36 RACE Control Architecture Processing RACE Controller Reallocates resources to meet control objectives RACE Control Agent Maps resource reallocation to OS/string specific parameters Uses OS & string control agents to modify OS/string parameters

37 CUTS Environment for Emulating DRE Systems Component Workload Emulator (CoWorkEr) Utilization Test Suite (CUTS) consists of a test network of CoWorkEr nodes Outside the test network is a Benchmark Data Collector (BDC) for collecting test metrics Makeup & functionality of a CoWorkEr CCM assembly constructed from CCM monolithic components Can perform CPU operations, database operations & memory allocations & deallocations Can perform background workloads Can send/receive events to/from other CoWorkErs to form application strings

38 CoWorkEr Behavioral Modeling in CUTS I/O Automata used to model behavior using state transitions and actions WML integrates into I/O automata at the action level Extends semantics of binding between worker and its actions

39 Measuring the Performance of Components Time to transmit an event is measured Time to process an event is measured Critical paths can be selected & evaluated to determine if end-to-end QoS deadlines are meet All data is collected by the Benchmark Data Collector (BDC) & stored in a database for offline evaluation

40 CUTS BMW Utility Dynamic updates of test runs Enables viewing of collected data and metrics

41 CUTS BMW: Observable Metrics Test selection based on critical path Metrics include best, average, worst case timings

42 Observing RACE Control Behavior Demonstrates increased load due to “hog string” Web enabled application of RACE to control QoS

43 F-15 product variant A/V 8-B product variant F/A 18 product variant UCAV product variant Product-line architecture (VI) Optimizations for Product-line Architectures PLAs define a framework of components that adhere to a common architectural style with a clean separation of commonalities and appropriate provisions for incorporating variations Middleware factors out many reusable general-purpose & domain-specific services from traditional DRE application responsibility Air Frame AP Nav HUDGPS IFF FLIR Hardware (CPU, Memory, I/O) OS & Network Protocols Host Infrastructure Middleware Distribution Middleware Common Middleware Services Domain-specific Services Standards middleware is a key technology candidate for supporting and sustaining vision of software product-lines

44 Technology Gaps in Middleware for PLAs PLAs have very “focused but crosscutting” requirements of underlying middleware infrastructure Optimized for the platform Lean footprint Efficient configuration & deployment Support run-time adaptations & reconfigurations Standards middleware development & optimizations philosophy catered to maintaining “generality, wide applicability, portability & reusability” OS, compiler and hardware independent e.g., CORBA, J2EE..NET These technology gaps are hindering PLA progress => adverse economic and societal consequences e.g. shortcomings of pre-postulated middleware [Jacobsen 04] Need to tailor and optimize standards middleware for PLAs while continuing to provide standards compliance, portability and flexibility

45 Middleware Specialization Catalog Domain-specific language (DSL) tools & process for automating the specializations Specification-imposed specializations Layer-folding Constant propagation Memoization Framework specializations Aspect Weaving techniques Bridge  Reactor Template method  Protocol Strategy  Messaging, Wait, Demultiplexing Deployment platform specializations Unroll copy loops Use emulated exceptions Leverage zero-copy data transfer buffers Development of reusable specialization patterns Identifying specialization points in middleware where patterns are applicable

46 Challenge: Lack of Application Context Missed middleware optimization opportunities Without application context Every invocation performs check for locality Middleware glue code for both code paths present in all use cases Impossible for middleware (alone) to predict application usage Cannot be solved by operating solely at middleware level!

47 Approach: Supply Application Context w/Models Build a global assembly optimizer framework Use models to capture & derive application context Explicit, e.g., radar & sensor are collocated (user-specified) Implicit, e.g., radar & sensor are deployed onto the same node Example Detect components internal to an assembly Eliminate overhead of Multiple Homes

48 Workflow MetaModel Domain Specific Workflow Model Workflow Model Interpreter Graph Transformation Tool Inputs to Graph Meta ModelInputs to Graph Execution Engine Output Modeling Framework Meta Model DSMs Hierarchical Enterprise communications applications analysis Supports generative programming. Partial views may be useful and can be directly used for for interactive dialog generation tools Static Analysis/Constraint checking. Validation of a communications application for a specific domain/end user devices Communications Applications Modeling Environment

49 Structure Spectrum of DSLs Routine configurationCreative construction matrix [element type]shapeformatbounds checking rect lower Triang vectarray upper Triang symm C-likeFrtn-like scr1 scr2scr3 scr4 scr7scr8 scr5 scr6 scr11scr12 scr9scr10 Path through a decision treeSubtree of a feature tree Subgraph of an (infinite) graph WizardsFeature-based configuration Graph-like language (with user-defined elements)

50 Research Impact & Future Work Current progress stems from years of iteration, refinement, & successful use Year Internet RPC Micro-kernels CORBA & DCOM Real-time (RT) CORBA Component Models (EJB) CORBA Component Model (CCM) RT/CCM DCE CoSMIC Long-term Research Directions High confidence, geographically distributed DRE systems Grid applications Large enterprise systems Greater focus on platform- independent models Heterogeneous systems Long-term Research Directions High confidence, geographically distributed DRE systems Grid applications Large enterprise systems Greater focus on platform- independent models Heterogeneous systems Model driven middleware Shape the standards e.g., OMG’s Model Driven Architecture (MDA) for DRE systems Advanced MDE ACE+TAO

51 Concluding Remarks Model-Driven Engineering (MDE) is an important emerging generative technology paradigm that addresses key lifecycle challenges of DRE middleware & applications OMG PSIG on Model Integrated Computing CoSMIC MDE tools are based on the Generic Modeling Environment (GME) CoSMIC is available from GME is available from agendas/mic.htm Many hard R&D problems with model-driven engineering remain unresolved!!

EXTRA SLIDES