Enabling autonomic Grid Applications: Requirements, Models and Infrastructure Authors: M. Parashar, Z. Li, H. Liu, V. Matossian and C. Schmidt Presenter: Khalid Saleem

Introduction Investigate challenges and requirements of programming Grid Applications Investigate challenges and requirements of programming Grid Applications Self-Management Applications Self-Management Applications Introduce Project Automate Introduce Project Automate Investigates Autonomic solutions to deal with Investigates Autonomic solutions to deal with Complexity Complexity Dynamism Dynamism Heterogeneity Heterogeneity Uncertainty Uncertainty

Grid Computing – Challenges & Requirements (1) Goal: Goal: Applications utilizing intellectual and physical resources across different disciplines and organizations Applications utilizing intellectual and physical resources across different disciplines and organizations Providing more effective and efficient solutions to important Scientific, Engineering, Business and Government problems Providing more effective and efficient solutions to important Scientific, Engineering, Business and Government problems Built on Seamless and secure mechanisms for Built on Seamless and secure mechanisms for Discovery Discovery Access Access Interactions Interactions

Grid Computing – Challenges & Requirements (2) Key characteristics of Grid Execution Environment and Applications are Key characteristics of Grid Execution Environment and Applications are Heterogeneity Heterogeneity Dynamism Dynamism Uncertainty Uncertainty Security Security

Grid Computing – Challenges & Requirements (3) Goals attainment requires Goals attainment requires Implementation Models Implementation Models Conceptual Models Conceptual Models Implementation Models address Implementation Models address Virtualization of organizations (GRIDs) Virtualization of organizations (GRIDs) Creation and Management of virtual organizations Creation and Management of virtual organizations Instantiation of a virtual machine as the execution environment of an application Instantiation of a virtual machine as the execution environment of an application Conceptual Models define Conceptual Models define Abstract machines supporting programming models and systems enabling application development Abstract machines supporting programming models and systems enabling application development

Grid Computing – Challenges & Requirements (4) Requirements for Grid Programming Systems Requirements for Grid Programming Systems Specify applications that can detect and dynamically respond during execution to the changes in Specify applications that can detect and dynamically respond during execution to the changes in state of execution environment state of execution environment state and requirements of the application state and requirements of the application Grid Applications should Grid Applications should Constitute of discrete, self managing components Constitute of discrete, self managing components Provide separate specifications for functional, non- functional, interaction and coordination behaviors Provide separate specifications for functional, non- functional, interaction and coordination behaviors Provide implementation independent interface definitions for above specifications Provide implementation independent interface definitions for above specifications

Requirements for Self-managing Applications on the Grid Self-managing applications on the Grid require Self-managing applications on the Grid require Static application requirements Static application requirements System and Application behaviors should be relaxed System and Application behaviors should be relaxed Sensitivity and Ability of elements and applications to adapt to the Sensitivity and Ability of elements and applications to adapt to the Dynamic state of the system; and Dynamic state of the system; and Changing requirements of the application Changing requirements of the application Essential common knowledge be expressed semantically (i.e. Ontology) rather than names addresses and identifiers Essential common knowledge be expressed semantically (i.e. Ontology) rather than names addresses and identifiers Core Middleware services (e.g. Discovery) driven by semantic knowledge Core Middleware services (e.g. Discovery) driven by semantic knowledge AutoMate AutoMate An attempt to address the above challenges and requirements An attempt to address the above challenges and requirements

AutoMate Investigates autonomic solutions to address challenges posed to Grid Computing Investigates autonomic solutions to address challenges posed to Grid Computing Based on Biological Systems Based on Biological Systems Development of Conceptual and Implementation models/architectures Development of Conceptual and Implementation models/architectures Programming Models, Framework and Middleware Services Programming Models, Framework and Middleware Services Definition of autonomic elements Definition of autonomic elements Development of autonomic applications Development of autonomic applications Policy, Content and Context driven application definition, management and execution Policy, Content and Context driven application definition, management and execution

Components of AutoMate

Accord: A Programming Framework for Autonomic Applications Extends existing distributed programming systems Extends existing distributed programming systems Realizes Realizes Separation of computations from coordination and interactions Separation of computations from coordination and interactions Separation of non-functional behaviors (e.g. Resource requirements, performance) from functional behaviors Separation of non-functional behaviors (e.g. Resource requirements, performance) from functional behaviors Separation of policy and mechanism Separation of policy and mechanism Accord Programming Model key components Accord Programming Model key components Autonomic Elements Autonomic Elements Accord Rules Accord Rules Autonomic Composition Autonomic Composition

Autonomic Elements (1) Autonomic Element Autonomic Element Extends programming elements Extends programming elements Objects, Components and Services Objects, Components and Services Defines self-contained Modular software unit with Defines self-contained Modular software unit with Specified interfaces Specified interfaces Explicit context dependencies Explicit context dependencies Encapsulates Encapsulates Rules Rules Constraints Constraints Mechanisms Mechanisms Can dynamically interact with other elements and the system Can dynamically interact with other elements and the system

Autonomic Elements (2) Autonomic Element

Autonomic Elements (3) Autonomic Element defined by 3 ports Autonomic Element defined by 3 ports Functional Port Functional Port Defines a set of functional behaviors provided and used by the element Defines a set of functional behaviors provided and used by the element Control Port Control Port Defines a set of sensors/actuators and the corresponding access constraints Defines a set of sensors/actuators and the corresponding access constraints Sensors: Interfaces providing info about the elements Sensors: Interfaces providing info about the elements Actuators: Interfaces for modifying state of the elements Actuators: Interfaces for modifying state of the elements Operational Port Operational Port Defines the interfaces to formulate and manage rules that govern the runtime behavior of the elements and their interactions. Defines the interfaces to formulate and manage rules that govern the runtime behavior of the elements and their interactions.

Autonomic Elements (4) Element Manager Element Manager Manages element execution Manages element execution Monitors state of the element and its context Monitors state of the element and its context Controls execution rules Controls execution rules Cooperates with other element managers Cooperates with other element managers

Accord Rules (1) Rules in the form Rules in the form IF condition THEN action IF condition THEN action Condition: Logical combination of sensors, elements, function interfaces and events Condition: Logical combination of sensors, elements, function interfaces and events Action: Sequence of invocations of element and/or system actuators and other interfaces Action: Sequence of invocations of element and/or system actuators and other interfaces Priority based mechanism to resolve conflicts Priority based mechanism to resolve conflicts

Accord Rules (2) Rule Classes Rule Classes Behavioral Rules Behavioral Rules Controls runtime functional behaviors of an autonomic element Controls runtime functional behaviors of an autonomic element Executed by the element manager within a single element Executed by the element manager within a single element Interaction Rules Interaction Rules Controls interactions and coordination of autonomic elements Controls interactions and coordination of autonomic elements Element managers collaborate to provide coordinated execution Element managers collaborate to provide coordinated execution

Autonomic Composition (1) Enables relationship between elements to be established and modified at runtime Enables relationship between elements to be established and modified at runtime Dynamic composition consists of composition plan and execution Dynamic composition consists of composition plan and execution Plans created at runtime Plans created at runtime Based on dynamically defined objectives, policies and the context and content of applications and systems Based on dynamically defined objectives, policies and the context and content of applications and systems Plan execution involves Plan execution involves Discovering elements, configuring them and defining interaction relationships and mechanisms Discovering elements, configuring them and defining interaction relationships and mechanisms

Autonomic Composition (2) Composition Plans Composition Plans Generated using Accord Composition Engine (ACE) Generated using Accord Composition Engine (ACE) Expressed in XML Expressed in XML Element Discovery uses Element Discovery uses Meteor content-based Middleware Meteor content-based Middleware Squid Discovery Service Squid Discovery Service Plan execution Plan execution P2P control network of element managers and agents within Rudder P2P control network of element managers and agents within Rudder Library of rule sets Library of rule sets Common control and communication relationships between elements Common control and communication relationships between elements Runtime Negotiation Protocols Runtime Negotiation Protocols Address runtime conflicts and conflicting decisions Address runtime conflicts and conflicting decisions

Accord Implementation Issues Accord allows interaction & coordination behaviors to be managed at runtime using rules Accord allows interaction & coordination behaviors to be managed at runtime using rules Deploying and executing rules Deploying and executing rules Impacts performance Impacts performance Increases robustness of applications; and Increases robustness of applications; and Ability to manage dynamism Ability to manage dynamism Runtime changes to interaction and coordination behaviors relatively infrequent and have small overheads Runtime changes to interaction and coordination behaviors relatively infrequent and have small overheads Time spent in establishing and modifying interactions is small as compared to computation times Time spent in establishing and modifying interactions is small as compared to computation times References [35, 36] References [35, 36]

Rudder Coordination Framework Scalable coordination middleware Scalable coordination middleware Supports self managing applications in decentralized distributed environment Supports self managing applications in decentralized distributed environment Consists of two key components Consists of two key components COMET COMET Enables flexible and scalable coordination among agents and autonomic elements Enables flexible and scalable coordination among agents and autonomic elements Agent Framework Agent Framework Composed of software agents, agent interaction and negotiation protocols Composed of software agents, agent interaction and negotiation protocols

COMET (1) Provides a global virtual shared coordination space Provides a global virtual shared coordination space Accessible by all peer agents, independent of physical location of tuples or identifiers of the host Accessible by all peer agents, independent of physical location of tuples or identifiers of the host Build on an associative messaging substrate Build on an associative messaging substrate Implements a distributed hash table Implements a distributed hash table Index space generated from semantic information space (ontology) Index space generated from semantic information space (ontology) Can be transient and dynamically constructed Can be transient and dynamically constructed Provides an associative communication abstraction Provides an associative communication abstraction Maps virtual information space to a dynamic set of available peer nodes Maps virtual information space to a dynamic set of available peer nodes Maintains content locality Maintains content locality

COMET (2) Provides a coordination abstraction layer Provides a coordination abstraction layer Extends traditional data-driven model with event based reactivity to changes in Extends traditional data-driven model with event based reactivity to changes in System state System state Data access operations Data access operations Defines a Reactive tuple abstraction consisting of Defines a Reactive tuple abstraction consisting of Condition that associates Reaction to events Condition that associates Reaction to events A Guard specifying how and when the Reaction will be executed A Guard specifying how and when the Reaction will be executed Provides Linda-like primitives Provides Linda-like primitives

Agent Framework (1) Composed of Dynamic Network of Software Agents existing at different levels Composed of Dynamic Network of Software Agents existing at different levels Agents are Agents are Processing units Processing units Perform actions based on dynamically defined rules Perform actions based on dynamically defined rules Agents Agents Monitor element states Monitor element states Manage element behaviors and dependencies Manage element behaviors and dependencies Coordinate element interactions Coordinate element interactions Cooperate to manage overall system/application behavior Cooperate to manage overall system/application behavior Agents use profiles to Agents use profiles to Identify and describe elements Identify and describe elements Interact with elements Interact with elements Control elements Control elements

Agent Framework (2) Defines a set of protocols for Defines a set of protocols for Agent coordination Agent coordination Application/system management Application/system management Examples of protocols include Examples of protocols include Discovery protocols Discovery protocols Control Protocols Control Protocols

Meteor: A content-based Middleware Scalable content-based Middleware infrastructure Scalable content-based Middleware infrastructure Consists of Consists of Self organizing content overlay Self organizing content overlay Content based routing engine and discovery service (Squid) Content based routing engine and discovery service (Squid) Associative Rendezvous Messaging Substrate (ARMS) Associative Rendezvous Messaging Substrate (ARMS)

Meteor Content Overlay Composed of Rendezvous Peer nodes (RP) Composed of Rendezvous Peer nodes (RP) RP nodes can join or leave the network at anytime RP nodes can join or leave the network at anytime Provides a single operation Provides a single operation Lookup (identifier) Lookup (identifier) Locates the peer node where the content should be stored or fetched Locates the peer node where the content should be stored or fetched

Squid Content based routing engine and decentralized discovery service Content based routing engine and decentralized discovery service Supports Supports Flexible content-based routing Flexible content-based routing Complex queries containing partial keywords, wildcards and ranges Complex queries containing partial keywords, wildcards and ranges Keywords can be common words or globally defined attributes based on ontologies or taxonomies Keywords can be common words or globally defined attributes based on ontologies or taxonomies Uses Hilbert Space Filling Curve (SFC) Uses Hilbert Space Filling Curve (SFC) Maps multidimensional information space to peer identifier space Maps multidimensional information space to peer identifier space

Associative Rendezvous Messaging Substrate (ARMS) Implements Associative Rendezvous (AR) interaction paradigm Implements Associative Rendezvous (AR) interaction paradigm Content-based decoupled interactions with programmable reactive behaviors Content-based decoupled interactions with programmable reactive behaviors Allows for asynchronous interactions among senders and receivers Allows for asynchronous interactions among senders and receivers Extends conventional name/identifier based rendezvous by Extends conventional name/identifier based rendezvous by Using flexible combination of keywords, wildcards and ranges from the semantic information space instead of identifiers Using flexible combination of keywords, wildcards and ranges from the semantic information space instead of identifiers Enabling reactive behaviors at rendezvous points encapsulated within messages, thus increasing flexibility and enabling multiple interaction semantics (broadcast, multicast) Enabling reactive behaviors at rendezvous points encapsulated within messages, thus increasing flexibility and enabling multiple interaction semantics (broadcast, multicast)

Autonomic Grid Applications

Autonomic Oil Reservoir Optimization (1) Problems Problems Selection of appropriate optimization algorithms Selection of appropriate optimization algorithms Runtime configuration and invocation of these algorithms Runtime configuration and invocation of these algorithms Dynamic optimization of reservoir Dynamic optimization of reservoir Autonomic application based on AutoMate consists of Autonomic application based on AutoMate consists of 1. Sophisticated reservoir simulation components Encapsulates complex mathematical models Encapsulates complex mathematical models Execution on Grid Execution on Grid 2. Grid Services providing secure and coordinated access to resources 3. Distributed data archives Stores historical, experimental and observed data Stores historical, experimental and observed data 4. Sensors embedded in the instrumented oilfield Provides real time data about the current state of oil field Provides real time data about the current state of oil field 5. External services providing data relevant to optimization of oil production or the economic profits Optimization based on VFSA and SPSA algorithms Optimization based on VFSA and SPSA algorithms 6. Actions of scientists, engineers in field, labs and management offices

Autonomic Oil Reservoir Optimization (2) Peers automatically Peers automatically Detect sub-optimal production behaviors at runtime Detect sub-optimal production behaviors at runtime Adjust interactions to correct sub-optimal production Adjust interactions to correct sub-optimal production Uses policies to Uses policies to Discover, select, configure and invoke appropriate optimization services to determine optimal well location Discover, select, configure and invoke appropriate optimization services to determine optimal well location

Autonomic Forest Fire Management Simulation (1) Predicts the speed, direction and intensity of fire front Predicts the speed, direction and intensity of fire front Uses dynamic environment and vegetation conditions Uses dynamic environment and vegetation conditions Consists of Consists of Data Space Manager (DSM) Data Space Manager (DSM) Partitions the forest represented by 2D data space into sub spaces based on information from CRM Partitions the forest represented by 2D data space into sub spaces based on information from CRM Computational Resource Manager (CRM) Computational Resource Manager (CRM) Provides system resources information to DSM Provides system resources information to DSM Rothermel Rothermel Simulates in parallel the fire spread in each sub space Simulates in parallel the fire spread in each sub space Uses current wind direction and intensity info simulated by Wind Model Uses current wind direction and intensity info simulated by Wind Model Wind Model Wind Model Provides wind direction and intensity simulations Provides wind direction and intensity simulations GUI GUI Allows experts to interact with the above elements Allows experts to interact with the above elements

Autonomic Forest Fire Management Simulation (2) Implementation of Accord Port types

Autonomic Forest Fire Management Simulation (3) Component Addition and Change in Interaction Relationship Example

Conclusion AutoMate AutoMate Implementation architecture and conceptual models that enable the development and execution of self-managing Grid applications Implementation architecture and conceptual models that enable the development and execution of self-managing Grid applications

Agnostic Question 1 How does AutoMate deal with the heterogeneity among all the elements of a grid? How does AutoMate integrate so many heterogeneous autonomic components to work together? How does AutoMate deal with the heterogeneity among all the elements of a grid? How does AutoMate integrate so many heterogeneous autonomic components to work together? By separating the policy from mechanism. Policies in the form of rules are used to orchestrate a repertoire of mechanisms to achieve context-aware adaptive runtime computational behaviors and coordination and interaction relationships based on functional, performance, and QoS requirements thus responding to heterogeneity and dynamics. Uses Dynamic composition to enable relationships between elements (via element managers) to be established and modified at runtime By separating the policy from mechanism. Policies in the form of rules are used to orchestrate a repertoire of mechanisms to achieve context-aware adaptive runtime computational behaviors and coordination and interaction relationships based on functional, performance, and QoS requirements thus responding to heterogeneity and dynamics. Uses Dynamic composition to enable relationships between elements (via element managers) to be established and modified at runtime

Agnostic Question 2 Security and trust is one of the key issues in grid environments, and the maintenance of users and permissions may become a bottleneck. Does AutoMate provide a specific autonomic solution for a self- maintenance of users and permissions across administrative boundaries? Security and trust is one of the key issues in grid environments, and the maintenance of users and permissions may become a bottleneck. Does AutoMate provide a specific autonomic solution for a self- maintenance of users and permissions across administrative boundaries? AutoMate does not explicitly provide a methodology for self maintenance of users and permissions across administrative boundaries. However, SESAME that is embedded into AutoMate allows for setting up security policies and permissions for avoiding fraud and intrusion. AutoMate does not explicitly provide a methodology for self maintenance of users and permissions across administrative boundaries. However, SESAME that is embedded into AutoMate allows for setting up security policies and permissions for avoiding fraud and intrusion.

Agnostic Question 3 The Accord framework incorporates practical human knowledge in the form of behavioral rules, but with the vastness of communication paradigms (e.g., RPC, RMI, publish/subscribe), how does Accord ensure the correctness of these rules? The Accord framework incorporates practical human knowledge in the form of behavioral rules, but with the vastness of communication paradigms (e.g., RPC, RMI, publish/subscribe), how does Accord ensure the correctness of these rules? “To ensure rule correctness, we design a rule template for each communication paradigm (e.g., RPC/RMI, publish/subscribe) and coordination pattern (e.g., conditional branch, loop, sequence, parallel execution). A rule template is instantiated via filling in parameters, such as the names of interaction parties and the exchanging data.” “To ensure rule correctness, we design a rule template for each communication paradigm (e.g., RPC/RMI, publish/subscribe) and coordination pattern (e.g., conditional branch, loop, sequence, parallel execution). A rule template is instantiated via filling in parameters, such as the names of interaction parties and the exchanging data.”

Agnostic Question 4 AutoMate utilizes a peer-to-peer middleware to support and enable the autonomic interactions among components. Can you think of a better architecture to achieve this goal? AutoMate utilizes a peer-to-peer middleware to support and enable the autonomic interactions among components. Can you think of a better architecture to achieve this goal? In this scenario using peer-to-peer would be a better option as AutoMate is based upon autonomic agent based control networks. Element managers execute rules to establish control and communication relationships among these elements in a decentralized and parallel manner. In this scenario using peer-to-peer would be a better option as AutoMate is based upon autonomic agent based control networks. Element managers execute rules to establish control and communication relationships among these elements in a decentralized and parallel manner.

Agnostic Question 5 Does Accord target a specific programming language? Does Accord target a specific programming language? Nothing particular. However, the prototype implementation was done using C++ and MPI Nothing particular. However, the prototype implementation was done using C++ and MPI

Agnostic Question 6 Can you extend on how AutoMate lets Globus- enabled grids to achieve an autonomic management? What elements of the AutoMate architecture communicate with Globus elements? Can you extend on how AutoMate lets Globus- enabled grids to achieve an autonomic management? What elements of the AutoMate architecture communicate with Globus elements? AutoMate makes use of its Accord autonomic elements and rule set, the Rudder Agent Framework and COMET to provide dynamic flexible and scaleable coordination among the peer nodes along with the use of conflict resolution and negotiation protocols. No specific information mapping the components of AutoMate to that of Globus has been provided. AutoMate makes use of its Accord autonomic elements and rule set, the Rudder Agent Framework and COMET to provide dynamic flexible and scaleable coordination among the peer nodes along with the use of conflict resolution and negotiation protocols. No specific information mapping the components of AutoMate to that of Globus has been provided.

Agnostic Question 7 Are you aware of other projects with the same goals than AutoMate? Are you aware of other projects with the same goals than AutoMate? Grid MP by United Devices Grid MP by United Devices Optimal Grid Optimal Grid opgrid/ opgrid/ opgrid/ opgrid/ GridARM for Structured Adaptive Mesh Refinement (SAMR) application GridARM for Structured Adaptive Mesh Refinement (SAMR) application OGSA to some extent OGSA to some extent

Agnostic Question 8 On a previous presentation of the same authors, we discussed that their so-called peer-to-peer architecture was in fact Client-Server. Do you think it is correct that they somewhat reuse this architecture for the Meteor and COMET components? On a previous presentation of the same authors, we discussed that their so-called peer-to-peer architecture was in fact Client-Server. Do you think it is correct that they somewhat reuse this architecture for the Meteor and COMET components? The previous presentation focused on PAWN where as the authors made use of Meteor in the current framework. Meteor is based on JXTA which is a general purpose P2P framework. Hence, it seems correct to use Meteor. The previous presentation focused on PAWN where as the authors made use of Meteor in the current framework. Meteor is based on JXTA which is a general purpose P2P framework. Hence, it seems correct to use Meteor.