International Service Availability Symposium (ISAS) 2007 MDDPro: Model-Driven Dependability Provisioning in Enterprise Distributed Real-time and Embedded Systems Sumant Tambe* Jaiganesh Balasubramanian Aniruddha Gokhale Thomas Damiano Vanderbilt University, Nashville, TN, USA Contact : *sutambe@dre.vanderbilt.edu International Service Availability Symposium (ISAS) 2007 May 21-22, 2007, University of New Hampshire, Durham, New Hampshire, USA This work is supported by subcontracts from LMCO & BBN

Component-based Enterprise DRE Systems Characteristics of component-based enterprise DRE systems Applications composed of one or more “operational string” of services or systems of systems Simultaneous QoS (Availability, Time Critical) requirements Dynamic (re)-deployment of components into operational strings Examples of Enterprise DRE systems Advanced air-traffic control systems Continuous patient monitoring systems Goal: Simplify and automate Fault-Tolerance provisioning in the DRE systems

Fault-Tolerance Design Considerations in DRE Systems Per-component concern – choice of implementation Depends of resources, compatibility with other components in assembly Failure recovery concern – what is the unit of failover? Availability concern – what is the degree of redundancy? What replication styles to use? Does it apply to whole assembly? State synchronization concerns – What is data-sync rate? Deployment concern – how to place components? Minimize failure risk to the system

Tangled Fault-Tolerance Concerns Implementation determines replication style and vice-versa Replication degree affects resources and deployment Replication style determines state synchronization style Availability of domain artifacts determines deployment Significant sources of variability that affect end-to-end QoS (performance + availability) Separation of Concerns using higher level abstractions is the key Design-time Deployment-time Run-time

Model-Driven Engineering – A Promising Approach Higher level of abstraction than third generation programming languages Modeling each concern separately alleviates system complexity Deployment model Component assembly model System structural model Different QoS models e.g., Fault-tolerance Generative and model transformation techniques to weave in appropriate glue code Complex System

Fault-tolerance Modeling Abstractions in MDDPro CQML (Component QoS Modeling Language) A DSML in the CoSMIC tool suite Fail-over Unit (FOU): Abstracts away details of granularity of protection (e.g., Component, Assembly, App-string) Replica Group (RPG): Abstracts away fault-tolerance policy details (e.g., Active/passive replication, rate and topology of state-synchronization) Shared Risk Group (SRG): Captures associations related to failure risk. (e.g., shared power supply among processors, shared LAN) Protection granularity concerns State-synchronization concerns Component Placement constraints Interpreter (component placement constraint solver): Encapsulates an algorithm for component-node assignment based on replica distance metric Replica Distance Metric

Fault-Tolerance Model in CQML CQML (Component QoS Modeling Language) A graphical QoS modeling language on top of a system composition language (e.g., PICML) Enhances system structure with QoS annotations (e.g., FOUs for granularity of protection) A FOU itself is a model and captures heartbeat frequency and replication groups A Replication group captures per component replication style, data synchronization rate

Fail-over Unit Example Primary Component A B C primary IOR “Client” container/component server container/component server container/component server Replica FOU A’ B’ C’ secondary IOR container/component server Primary FOU Replica Component

Shared Risk Group Example Ship_SRG DataCenter1_SRG DataCenter2_SRG Rack1_SRG Rack2_SRG Node1 (blade31) Node2 (blade32) Shelf1_SRG Shelf2_SRG Shelf1_SRG Blade30 Blade34 Blade29 Blade33 Blade36

Formulation of Replica Placement Problem Define N orthogonal vectors, one for each of the distance values computed for the N components (with respect to a primary) and vector-sum these to obtain a resultant.  Compute the magnitude of the resultant as a representation of the composite distance captured by the placement .  Compute the distance from each of the replicas to the primary for a placement.  Record each distance as a vector, where all vectors are orthogonal. Add the vectors to obtain a resultant. Compute the magnitude of the resultant. Use the resultant in all comparisons (either among placements or against a threshold) Apply a penalty function to the composite distance (e.g. pair wise replica distance or uniformity) R1 P R2 R3

Component Placement Example using SRGs Replica 1 2 3 Ship_SRG DataCenter1_SRG DataCenter2_SRG Rack1_SRG Rack2_SRG Node1 (blade31) Node2 (blade32) Composite Distance Primary Shelf1_SRG Shelf2_SRG Shelf1_SRG Blade30 Blade34 Blade29 Blade33 Blade36

FT Modeling & Generative Steps Model components and application strings in CQML Model Fail Over Units (FOUs) and Shared Risk Groups (SRGs) Determine deployment of primary components GME/PICML Model Information Domain, Deployment, SRG, and FOU injection Replica Placement Algorithm model FT Interpreter Augmented Deployment Plan Interpreter automatically injects replicas and associated CCM IOGRs 5. Distance-based constraint algorithm determines replica placement in deployment descriptors.

Fault-Tolerance Model in CQML Replica = 3 Min Distance = 4

Shared Risk Group Model in CQML

Generative Capabilities for Provisioning FT Automatic Injection of replicas Augmentation of deployment plan based on number of replicas Automatic Injection of FT infrastructure components E.g. Collocated “heartbeat” (HB) component with every protected component. Automatic Injection of connection meta-data Specialized connection setup for protected components (e.g. Interoperable Group References IOGR) HB Container M x N

Example of Automated Heartbeat Component Injection Collocated heartbeat component Primary Component FPC intra-FOU heartbeat HB HB HB A B C “client” primary IOR container/component server container/component server container/component server IOGR Primary FOU periodic FPC heartbeat FPC secondary IOR HB HB HB Connection Injection A’ B’ C’ Replica Component container/component server container/component server container/component server Replica FOU

Configurable FT Infrastructure Future Work Developing advanced constraint solver algorithms to incorporate multiple dimensions of constraints in component placement decision (e.g. resources, communication latency) Optimizing the number of generated heartbeat components for collocated, protected application components. Enhancing the DSL and the tools to capture the configurability required by the new availability specifications e.g., Lightweight RT/FT CORBA Spec. e.g., SAF AIS Spec. Enhancing working prototypes and evaluating them in representative DRE systems HB Container Try merging this in conclusions. Configurable FT Infrastructure

Tools available for download from Concluding Remarks Model-Driven Engineering separates dependability concerns from other system development concerns Separation of concerns helps alleviate system complexity Productivity increase at model interpretation time using automatic generation of FT infrastructure (e.g. heartbeat components and connections) Tools available for download from www.dre.vanderbilt.edu/cosmic www.dre.vanderbilt.edu/CIAO

Questions?