Polish Infrastructure for Supporting Computational Science in the European Research Space Component Approach to Distributed Multiscale Simulations Katarzyna Rycerz(1,2), Marian Bubak(1,2) (1) Institute of Computer Science AGH, Mickiewicza 30, Kraków, Poland (2) ACC Cyfronet AGH, ul. Nawojki 11, Kraków, Poland KU KDM, Zakopane,

2 Outline  Requirements of multiscale simulations  Motivation for a component model for such simulations  HLA-based component model (idea, design challenges, possible solutions)  Experiment with Multiscale Multiphysics Scientific Environment (MUSE)  Possible integration with GridSpace VL  Summary

3 Multiscale Simulations  Consists of modules of different scale  Examples – e.g. modelling:  reacting gas flows  capillary growth  colloidal dynamics  stellar systems (e.g. Multiscale Multiphysics Scientific Environment – MUSE used in this work)  and many more...

4 Multiscale Simulations - Requirements  Actual connection of two or more models together  obeying the law of physics (e.g. conservation law)  advanced time management: ability to connect modules with different time scales and internal time management strategies  support for connecting models of different space scale  Composability and reusability of existing models of different scale  finding existing models needed and connecting them either together or to the new models  ease of plugging in and unplugging them from the running system  standarized models’ connections + many users sharing their models = more chances for general solutions

5 Motivation  To wrap simulations into recombinant components that can be selected and assembled in various combinations to satisfy requirements of multiscale simulations  Need for a special component model that: provides machanisms specyfic for distributed multiscale simulations  adaptation of one of the existing solutions for distributed simulations – our choice – High Level Architecture (HLA) supports long running simulations - setup and steering of components should be possible also during runtime gives a possibility to wrap legacy simulation kernels into components  Need for an infrastructure that facilitates cross-domain exchange of components among scientists  need for support for the component model  using Grid solutions (e-infrastructures) for crossing administrative domains

6 Related work  Model Couling Toolkit  applies a message passing (MPI) style of communication between simulation models.  oriented towards domain data decomposition of the simulated problem  provides a support for advanced data transformations between different models  J. Larson, R. Jacob, E. Ong ”The Model Coupling Toolkit: A New Fortran90 Toolkit for Building Multiphysics Parallel Coupled Models.” 2005: Int. J. High Perf. Comp. App.,19(3),  Multiscale Multiphysics Scientific Environment (MUSE)  a software environment for astrophysical applications  scripting approach (Python) is used to couple models together.  models include: stellar evolution, hydrodynamics, stellar dynamics and radiative transfer  sequential execution  S. Portegies Zwart, S. McMillan, at al. A Multiphysics and Multiscale Software Environment for Modeling Astrophysical Systems, New Astronomy, volume 14, issue 4, year 2009, pp  The Multiscale Coupling Library and Environment (MUSCLE)  provides a software framework to build simulations according to the complex automata theory  introduces concept of kernels that communicate by unidirectional pipelines dedicated to pass a specific kind of data from/to a kernel (asynchronous communication)  J. Hegewald, M. Krafczyk, J. Tlke, A. G. Hoekstra, and B. Chopard. An agent-based coupling platform for complex automata. ICCS, volume 5102 of Lecture Notes in Computer Science, pages Springer, 2008.

7 Why High Level Architecture (HLA) ?  Introduces the concept of simulation systems (federations) built from distributed elements (federates)  Supports joining models of different time scale - ability to connect simulations with different internal time management in one system  Supports data management (publish/subscribe mechanism)  Separates actual simulation from communication between fedarates  Partial support for interoperability and reusability (Simulation Object Model (SOM), Federation Object Model (FOM), Base Object Model (BOM))  Well-known IEEE and OMT standard  Reference implementation – HLA Runtime Infrastructure (HLA RTI)  Open source implementations available – e.g. CERTI, ohla

8 HLA Component Model  Model differs from common models (e.g. CCA) – no direct connections, no remote procedure call (RPC)  Components run concurrently and communicate using HLA mechanisms  Components use HLA facilities (e.g. time and data management)  Differs from original HLA mechanism:  interactions can be dynamically changed at runtime by a user  change of state is triggered from outside of any federate CCA model HLA model

9 HLA components design challenges  Transfer of control between many layers  requests from the Grid layer outside the component  simulation code layer  HLA RTI layer.  The component should be able to efficiently process concurrently:  actual simulation that communicates with other simulation components via RTI layer  external requests of changing state of simulation in HLA RTI layer. Simulation Code CompoHLA library HLA RTI Component HLA Component HLA Grid platform (H2O) External requests: start/stop join/resign set time policy publish/subscribe Grid platform (H2O)

10 Preliminary solution - Mechanism of HLA RTI Concurrent Access Control  Use concurrent access exception handling available in HLA  Transparent to developer  Synchronous mode - requests processed as they come  simulation is running in a separate thread  Dependent on implementation of concurrency control in used HLA RTI  Concurrency difficult to handle effectively  e.g starvation of requests that causes overhead in simulation execution Simulation Code CompoHLA library HLA RTI (concurrent access control) Component HLA Component HLA Grid platform (H2O) External requests Grid platform (H2O)

11 Advanced Solution - Use Active Object Pattern  Requires to call a single routine in a simulation loop  Asynchronous mode - separates invocation from execution  Requests processed when scheduler is called from simulation loop  Independent on behavior of HLA implementation  Concurrency easy to handle  JNI used for communication between Simulation Code, Scheduler and CompoHLA library Simulation Code CompoHLA library HLA RTI Component HLA Component HLA Grid platform (H2O) External requests Grid platform (H2O) Scheduler Queue

12 Interactions between components in example experiment  Modules taken from Multiscale Multiphysics Scientific Environment (MUSE)  Multiscale simulation of dense stellar systems  Two modules of different time scale:  stellar evolution (macro scale)  stellar dynamics - N-body simulation (meso scale)  Data management  mass of changed stars are sent from evolution (macro scale) to dynamics (meso scale)  no data is needed from dynamics to evolution  data flow affects whole dynamics simulation  Dynamics takes more steps than evolution to reach the same point of simulation time  Time management - Regulating federate (evolution) regulate the progress in time of constrained federate (dynamics)  The maximal point in time which the constrained federate can reach (LBTS) at certain moment is calculated dynamically according to the position of regulating federate on the time axis

13 Usage example – MUSE application H2O kernel Grid side A H2O kernel Grid side B Component user Component Client  Asks chosen components to join into a simulation system (called federation in HLA terminology)  Asks chosen components to publish or subscribe to certain data objects (e.g. Stars)  Asks components to set their time policy Dynamics HLAComponent Evolution HLAComponent HLA federation join federation subscribe publish be constrained be regulating Dynamics HLAComponent Evolution HLAComponent

14 Usage example – MUSE application H2O kernel Grid side A H2O kernel Grid side B Component user Component Client  Asks components to start  Alter the publications/subscriptions/ti me policy during runtime Dynamics HLAComponent Evolution HLAComponent HLA federation start unpublish Star data object

15 Experiment Results  Comparision of:  Concurrent execution, conservative approach of dynamics and evolution as HLA components  Sequential execution (MUSE)  Timing of:  Request processing (through grid and component layer)  Request realisation (scheduler)  H2O v2.1 as a Grid platform and HLA CERTI v – open source  Experiment run on DAS3 grid nodes in:  Delft (MUSE sequential version and dynamics component)  Amsterdam UvA (evolution component)  Leiden (component client)  Amsterdam VU (RTIexec control process)  Each grid node is a cluster of two 1- GHz Pentium-IIIs nodes connected with internal Myrinet-2000 network  10Gb ethernet used as the external network between Grid nodes

16 Possible Integration with GridSpace VL  Modules that can be reused:  IDE for Experiment Script  Execution Engine  Registry  Scenario Repository  Extensions needed:  Support for HLA component descriptions that include events/objects produced/consumed by a component  Component Description Assember will guide the user in joining component descriptions into simulation description (Federation Object Model–like files).

17 Future work  A description language for connecting HLA components:  Currently used: HLA FOM - definition of structures of data objects and events that need to be passed between HLA components  Needs to contain more information especially related to modules’ scale.  Needs to support different data types e.g. arrays often used in legacy implementations of simulation models etc.  Interactivity:  the support for components that are sources of data streams - often a long running simulations - produce partial results that should be streamed to the user before the simulation actually stops.  the ability to interpret commands given to HLA components in the interactive mode

18 Summary  Presented HLA component model enables the user to dynamically compose/decompose distributed simulations from multiscale elements residing on the Grid  Architecture of the HLA component supports steering of interactions with other components during simulation runtime  The presented approach differs from that in original HLA, where all decisions about actual interactions are made by federates themselves.  The functionality of the prototype is shown on the example of multiscale simulation of a dense stellar system – MUSE environment.  Experiment results show that that grid and component layers do not introduce much overhead.  In the future we plan to fully integrate the HLA components with GridSpace Virtual Laboratory

19 References  K. Rycerz, M. Bubak, and P. M. A. Sloot, Using HLA and Grid for Distributed Multiscale Simulations, in: R. Wyrzykowski, J. Dongarra, K. Karczewski, and J. Wasniewski (Eds.), Proceedings of 7-th International Conference, PPAM 2007, Gdansk, Poland, September 2007, LNCS 4967, Springer 2008, pp  K. Rycerz, M. Bubak and P.M.A. Sloot, Dynamic Interactions in HLA Component Model for Multiscale Simulations, ICCS, volume 5102 of Lecture Notes in Computer Science, pages Springer,  K. Rycerz, M. Bubak, P. M. A. Sloot: HLA Component Based Environment For Distributed Multiscale Simulations In: T. Priol and M. Vanneschi (Eds.), From Grids to Service and Pervasive Computing, Springer, 2008, pp  K. Rycerz, M. Bubak, P. M. A. Sloot : Collaborative Environment for HLA Component- Based Distributed Multiscale Simulations (in preparation)  Grid Space webpage  PL-Grid Project,

20 Slajdy do ew. wykorzystania, zrodla rysunkow

21 User Code CompoHLA library HLA RTI Component HLA Component HLA Grid platform (H2O) External requests Grid platform (H2O)

22 Active object in HLA Component

23  CompoHLA library does not free the developer from understanding HLA, but simplifies use of it  Developer has to inherit from CompoHLASimulator class and point to the main function starting simulation  CompoHLADataObject class has to be inherited for each data object visible for external user  A developer has to override FederateAmbassador class callbacks (used by RTI when receiving data from other federates)  To use HLA services the developer calls CompoHLAFederate class (that uses the HLA RTIambassador) CompoHLA Library

24 Usage example – MUSE application H2O kernel Grid side A H2O kernel Grid side B Component user Component Client  Asks chosen components to join into a simulation system (called federation in HLA terminology)  Asks chosen components to publish or subscribe to certain data objects (e.g. Stars)  Asks components to set their time policy Dynamics HLAComponent Evolution HLAComponent HLA federation join federation subscribe publish be constrained be regulating Dynamics HLAComponent Evolution HLAComponent

25 Usage example – MUSE application H2O kernel Grid side A H2O kernel Grid side B Component user Component Client  Asks components to start  Alter the publications/subscriptions/ti me policy during runtime Dynamics HLAComponent Evolution HLAComponent HLA federation start unpublish Star data object

26 Future work - Possible Integration with GridSpace VL  Modules that can be reused:  Experiment Script Editor  Execution Engine  Registry  Scenario Repository  Extensions needed:  Support for HLA component descriptions that include events/objects produced/consumed by a component  Component Description Assember will guide the user in joining component descriptions into simulation description (Federation Object Model–like files).

27 Future work - Possible Integration with GridSpace VL IDE for Experiment Script Component Description Assembler Repository viewer Simulations’ descriptions (based on HLA FOM) -objects and events exchanged between Components by HLA RTI Execution engine Registry -models descriptions - data sources Simulation Scenario Repository -experiments scripts -FOM-like simulation’s descriptions E-infrastructure HLA Component Simulation Model A HLA Component Simulation Model B HLA Component Simulation Model C Experiment script -operations on components -definiton of I/O data sources  Modules that can be reused:  IDE for Experiment Script  Execution Engine  Registry  Scenario Repository  Extensions needed:  Support for HLA component descriptions that include events/objects produced/consumed by a component  Component Description Assember will guide the user in joining component descriptions into simulation description (Federation Object Model–like files).

28 Multiscale Multiphysics Scientific Environment (MUSE)  Multiscale simulation of dense stellar systems  Three modules of different scale:  stellar evolution (macro scale)  stellar dynamics - N-body simulation (meso scale)  hydro dynamics - simulation of collisions (micro scale)  Number of steps in the figure is chosen for simplicity – in general there are more steps of dynamics within time of one evolution step  Three different types of time interactions between multiscale elements  Collision is triggered by dynamics seen by evolution and dynamics as a point in time  Dynamics should not go more forward in time than evolution Conservative approach  Output of collision can cause evolution to roll back Optimistic approach data trigger evolution dynamics collision MUSE

29 Interactions between components in example experiment  Modules taken from Multiscale Multiphysics Scientific Environment (MUSE)  Multiscale simulation of dense stellar systems  Two modules of different time scale:  stellar evolution (macro scale)  stellar dynamics - N-body simulation (meso scale)  Data management  mass of changed stars are sent from evolution (macro scale) to dynamics (meso scale)  no data is needed from dynamics to evolution  data flow affects whole dynamics simulation  Dynamics takes more steps than evolution to reach the same point of simulation time  Time management - Regulating federate (evolution) regulate the progress in time of constrained federate (dynamics)  The maximal point in time which the constrained federate can reach (LBTS) at certain moment is calculated dynamically according to the position of regulating federate on the time axis LBTS-Other federates will not send messages before this time. Federate may only advance time within this interval Federate’scurrent logical time. Federate’s effective logical time. Federate may not publish messages within this interval Federate’scurrent logical time. t=0 Lookahead Constrained federate(dynamics) Regulating federate (evolution)