1. Fault-tolerance for Component-based Systems – An Automated Middleware Specialization Approach Sumant Tambe* Akshay Dabholkar Aniruddha Gokhale Abhishek Dubey (Presenter) Institute of Software Integrated Systems (ISIS), Vanderbilt University, Nashville, TN, USA *Contact : sutambe@dre.vanderbilt.edusutambe@dre.vanderbilt.edu 12 th IEEE International Symposium on Object/component/service- oriented Real-time distributed Computing (ISORC 2009)

2. 2 Sumant Tambe et al. GRAFT – FT for Component-based SystemsISORC 2009 Motivation  Contemporary General-purpose middleware (CORBA, J2EE)  Generic: Well designed for broad applicability  Feature-rich: Supports non-functional properties, such as security, real-time, FT etc.  However,  Does not support domain-specific variations of non-functional semantics out-of-the-box  E.g., Domain-specific fault-tolerance (coming next)  Cost of developing proprietary middleware is high

3. 3 Sumant Tambe et al. GRAFT – FT for Component-based SystemsISORC 2009 Motivational Case Study (Material Handling System)  Representative Examples  FedEx, UPS, DHL  Airport Baggage Handling  Food Processing/Bottling  High availability and safety is critical  Communicating software components  Material Flow Control (MFC)  Hardware Interface Layer (HIL)  E.g. Flipper and Motor Controllers

4. 4 Sumant Tambe et al. GRAFT – FT for Component-based SystemsISORC 2009 Material Handling System (Fault model and domain-specific recovery)  Hardware and software faults  E.g., Jamming of the flipper  Crash faults of software components  Detected by software components  Communicated using software exceptions  E.g., FlipperJamException  Communication failure CORBA::COMM_FAILURE  Domain-specific group recovery semantics Shut down entire primary assembly (F, MC1, MC2) Start the replica assembly (F’, MC1’, MC2’)

5. 5 Sumant Tambe et al. GRAFT – FT for Component-based SystemsISORC 2009 Backup Distributed Processing Unit (DPU) Primary Distributed Processing Unit (DPU) System Design and Implementation Challenges  Lack of middleware abstractions  Failure of one component means failure of all in a DPU  Recover a collection of components simultaneously (even those who don’t have direct exposure to the fault)  Application-level solution is undesirable  Technical concerns is (ideally) no business of applications  Failure handling behavior crosscuts every component in a DPU  Lot of manual programming ABCA’B’C’ Problem Statements 1.How to add new semantics in COTS middleware retroactively? 2.How to automate it to improve productivity and reduce cost? Problem Statements 1.How to add new semantics in COTS middleware retroactively? 2.How to automate it to improve productivity and reduce cost?

6. 6 Sumant Tambe et al. GRAFT – FT for Component-based SystemsISORC 2009 Solution Approach  Promising approach for retroactive behavior augmentation  Aspect-oriented Programming (AOP)  Modularizes domain-specific functionality, potentially cross-cutting  Aspect compiler can weave in the domain-specific behavior into the original fabric of the COTS middleware  Promising approach for automation  Domain-specific modeling (DSM)  Inherently supports tool-driven generation of programming artifacts  Simplifies specification of domain-specific requirements Our Solution GRAFT (GeneRative Aspects for Fault-Tolerance) Our Solution GRAFT (GeneRative Aspects for Fault-Tolerance)

7. 7 Sumant Tambe et al. GRAFT – FT for Component-based SystemsISORC 2009 GRAFT - Overview  GeneRative Aspects for Fault-Tolerance  A two step process for specializing middleware 1.Design-time support to specify domain-specific FT requirements  Component Availability Modeling Language (CAML)  Automate FT modeling using a model transformation 2.Run-time support for middleware specialization  Generate application-specific aspect code for group failover  Weave in generated code in application-specific stubs.

8. 8 Sumant Tambe et al. GRAFT – FT for Component-based SystemsISORC 2009 Component Availability Modeling Language (CAML)  Annotate component structural models with fault-tolerance attributes  FT requirements captured using FailOverUnit (FOU)  FOU abstracts away the details of granularity of protection  E.g., Component, Assembly  Treats a group of components as a single unit of failover  Semantics: If one fails, all fail; Clients failover to replica  Configurable degree of replication  Captures application-specific failure exceptions  E.g., FlipperJamException Replica = 2

9. 9 Sumant Tambe et al. GRAFT – FT for Component-based SystemsISORC 2009 Step 1: Automated Structural FT Modeling

10. 10 Sumant Tambe et al. GRAFT – FT for Component-based SystemsISORC 2009 Step 2: Automated Aspect Code Generation  Two behaviors based on component role  FOU participant’s behavior  Detects the failure, if any  Shuts down all other participants, including itself  FOU client’s behavior  Detects the failure, if any  Shuts down the FOU  Does an automatic failover to a replica FOU  Generated code:AspectC++  AspectC++ compiler weaves in the generated code in the respective component stubs

11. 11 Sumant Tambe et al. GRAFT – FT for Component-based SystemsISORC 2009 Sample Generated Aspect Code (MFC component) aspect FailOverUnit_Client { // Auto-generated array of names of FailOverUnit participants. char * FOU_Participants[] = { "FlipperController“, "MotorController1“, "MotorController2“, 0 }; size_t failure_count_; // Initialized to zero. // Contains remote object reference of the replica. HIL::IFlipperController_var replica_ref_; // Weave advice around local stub of the flip() method of MFC. advice execution ("void HIL::IFlipperController::flip()") : around () // The advice is applied around the flip method. { do { // Use the remote reference of the backup FlipperController component only if the primary component has failed. if (failure_count_ > 0) // "_that" is used to change "this" pointer before proceeding. // Use live object reference of the replica. tjp->action()._that = replica_ref_.in(); try { // Continue the flip() function call as usual. tjp->proceed (); break; } catch(HIL::FlipperJamException & e) { handle_exception(e); // deactivates FailOverUnit participants } catch(CORBA::COMM_FAILURE & e) { handle_exception(e); // deactivates FailOverUnit participants } catch(CORBA::TRANSIENT & e) { handle_exception(e); // deactivates FailOverUnit participants } // Application-specific non-catastrophic exceptions are passed. } while (replica_ref_.in() != NULL_POINTER); } };

12. 12 Sumant Tambe et al. GRAFT – FT for Component-based SystemsISORC 2009 Run-time Coordination 1.FOU participant detects the failure of another participant 2.Shuts down the primary FOU 3.Deployment infrastructure (DAM) removes the components 4.Clients detect the failure of FOU 5.Clients obtain the replica references from the naming service 6.Successful failover of all the clients

13. 13 Sumant Tambe et al. GRAFT – FT for Component-based SystemsISORC 2009 Evaluation of Efforts Reduction (Replica = 2) Component name# of original connections # of replica components # of replica connections Material Flow Control1 / 10 / 02 / 0 Flipper Controller2 / 22 / 04 / 0 Motor Controller 11 / 12 / 0 Motor Controller 22 / 12 / 0 Fault-tolerance Modeling Efforts Without/With GRAFT Component name# of try blocks# of catch blocksTotal # of lines Material Flow Control1 / 03 / 045 / 0 Flipper Controller2 / 06 / 090 / 0 Motor Controller 10 / 0 Motor Controller 20 / 0 Fault-tolerance Programming Efforts Without/With GRAFT

14. 14 Sumant Tambe et al. GRAFT – FT for Component-based SystemsISORC 2009 Concluding Remarks  Specializing middleware for non-functional properties is desirable using aspects  Aspect-oriented code can be auto-generated from higher-level domain/application-specific models  Higher to lower-level model transformations, and code generation improves productivity  GRAFT realizes this approach  for fault-tolerant component-based systems built using Component Integrated ACE ORB (CIAO)  using Component Availability Modeling Language (CAML)  using C-SAW and ECL for model transformation  using AspectC++ for aspect-oriented programming www.dre.vanderbilt.edu/cosmic www.cis.uab.edu/gray/Research/C-SAW www.dre.vanderbilt.edu/cosmic www.cis.uab.edu/gray/Research/C-SAW

15. 15 Sumant Tambe et al. GRAFT – FT for Component-based SystemsISORC 2009 Thank you!