1. DISCOGRID Sophia, 13-14 mars 2006 The Padico Environment Christian Pérez PARIS Research Team, IRISA/INRIA, Rennes, France

2. Outline  Motivation  A Distributed Parallel Component Model  GridCCM  Handing Network Issues  PadicoTM  Automatic Deployment of Applications  ADAGE  Conclusion

3. Programming Grid? Visualization Homogeneous cluster SAN Homogeneous cluster LAN WAN Supercomputer Flow code Transport code Controller velocity scalars concentration Programming Environment ( Virtualization)

4. Requested Features  Support of legacy codes  Multi-languages, multi-OS, multi-architectures  Support of parallel codes  MPI, PVM, OpenMP,...  Support for network transparency  Support for deployment  “Single-provider” application  “Multi-provider” application  Software component! Transport Flow

5. Distributed components: OMG CCM  A distributed component-oriented model  An architecture for defining components and their interaction  Interaction implemented through input/output interfaces  Synchronous and asynchronous communication  A runtime environment based on the notion of containers (lifecycle, security, transaction, persistent, events)  Multi-languages/OS/ORBs/vendors, …  With a packaging/deployment model  A packaging technology for deploying binary multi-lingual executables  Could be deployed and run on several distributed nodes simultaneously My Component Component interface Facets Event sources Event sinks Attributes Receptacles OFFERED REQUIRED Component-based application Courtesy of P. Merle, LIFL, France

6. High Performance CORBA Request  Use a zero-copy ORB  OmniORB  Use a High Perf network  Myrinet-2000  Bandwidth  Myrinet-2000 : 240MB/s  96% of the hardware bandwidth!  Latency  CORBA/Ethernet : 150 µs  CORBA/Myrinet-2000 : 18 µs  MPI/Myrinet-2000 : 11 µs Bandwidth of several runtimes over Myrinet-2000 and SCI networks

7. Component and Parallelism  How to embed a parallel code into a component? Parallel Code AParallel Code B MPI PVM

8. Parallel Component HPC Component A HPC Component B What the application designer should see… … and how it must be implemented ! // Comp. A// Comp. B

9. Distributed Component Model CallerReceiver // Emitter Code o->factorize(m); Distributed component (a process) // Receiver Code void serv::factorize(const Matrix mat) {... ;}

10. SPMD Parallel Component Model (1) Data Redistribution CallerParallel Receiver Distributed component (a process) // SPMD Receiver code void factorize(const DMatrix mat) {.... MPI_Bcast(...)... ;} Parallel Distributed Component // Emitter Code o->factorize(m);

11. SPMD Parallel Component Model (2) Data Redistribution Parallel Caller Receiver // Receiver Code void factoriser(const Matrice mat) {... ;} Distributed component (a process)Parallel Distributed Component // SPMD Emitter Code o->factorize(m);

12. Parallel Receiver Parallel Caller // SPMD Emitter Code o->factorize(m); // SPMD Receiver Code SPMD void factorizer(const Matrix mat) {.... MPI_Bcast(...)... ;} Data Redistribution Distributed component (a process)Parallel Distributed Component SPMD Parallel Component Model (3)

13. GridCCM: Parallel CORBA Component  Extends the CORBA Component Model  No modification of CCM specification  Portable  Extension of CCM descriptor (XML)  Parallel component  Parallelism is a non-functional property of a component  It is an implementation issue  Collection of sequential component  SPMD execution model  Support of distributed arguments  API for data redistribution  API for communication scheduling w.r.t. network properties  Support of parallel exceptions Parallel Comp. SPMD Proc. SPMD Proc. SPMD Proc. SPMD Proc. SPMD Proc.

14. GridCCM Architecture Overview Object Request Broker CORBA stub/skeleton Communication Library ( MPI) Application Client view management - Data distribution description Communication management - Communication matrix computation - Communication scheduling computation - Communication execution Redistribution Library 1 Communication Library GridCCM runtime Scheduling Library

15. Latency  SAN  Myrinet-200/GM  Follows MPI barrier times  Pt-2-Pt overhead  CORBA : 25 µ s  PaCO++ : 35 µ s  WAN  French VTHD Network  2 Gbit/s  ~ 1000 Km long  Negligible overhead  10 µ s << 8 ms Nodesmilliseconds 1x18.01 2x28.04 4x48.08 8x88.22 16x168.40

16. Bandwidth  SAN  Myrinet-2000/GM  Aggregation  4 nodes :600 MB/s 4x150 MB/s  WAN  French VTHD Network  1 Gbit/s switch  Saturate for n > 12  Sustaining bandwidth  Using a Scheduling Algorille algorithm INRIA/LORIA  But need to know network properties

17. Relationship between Models  GridCCM: a parallel distributed component model  Extension of CCM  Based on PaCO++  PaCO++: a parallel distributed object model  Extension of CCM CORBA Object Model CORBA Component Model (CCM) Parallel Object Model (PaCO++) Parallel Component Model (GridCCM) Parallel Extension Component Extension

18. Status  PaCO  Obsolete, not maintain any longer  PaCO++  Available on the Web (version 0.2)  LGPL/GPL licence, cf http://www.irisa.fr/paris/Padico  MPI support  Data Redistribution  Identity, 1D Bloc-cyclic, “EPSN”  Still going on  GridCCM  Not yet available, only prototype version  Being developed

19. Outline  Motivation  A Distributed Parallel Component Model  GridCCM  Handing Network Issues  PadicoTM  Automatic Deployment of Applications  ADAGE  Status  Conclusion

20. Deploy the same code using different networking resources SAN LAN WAN process processus process processus process Configuration 1 process processus process processus process Configuration 2 WAN SAN High performance middleware Middleware over a WAN The middleware adapt itself to the available network Mapping Objects onto Resources

21. Handling network issues: PadicoTM  Network heterogeneity  Several networks, Non IP network, parallel stream, online compression, firewalls  Decouple communication API from actual network  Enables different access to the networks  Transparent network access  High Performance provided to middleware systems  Based on PM2  Consequences: Control the mapping of a middleware to a network Add special adapters TCP/IPMyrinetSCI... AdOC Parallel Stream PadicoTM MPICORBASOAPJXTA SSL

22.  Bandwidth  Myrinet-2000 : 240MB/s  SCI : 86MB/s  96% of the hardware bandwidth for CORBA, MPI and Java  Latency  CORBA/Myrinet-2000: 18 µs  MPI/Myrinet-2000: 11 µs  Java/Myrinet-2000: 40 µs  CORBA/Ethernet: 150 µs PadicoTM-enabled Middleware bandwidth over Myrinet-2000 and SCI PadicoTM-enabled Middleware Perfs

23. WAN PadicoTM Performance  Parallel streams  VTHD Network (Rennes-Nice)  Without : 8 MB/s (64 Mb/s)  With (4 streams): 12 MB/s (96 Mb/s)  For all runtimes  Overhead negligible  Adaptive Compression (AdOC)  ADSL link 608 kbit/s  Without compression : 580 kbit/s  With compression : 800 to 1200 kbit/s  For all runtimes  Overhead negligible The VTHD network

24.  Middleware and methods supported by PadicoTM :  MPI mpich-1.1.2 & mpich-1.2.4 (ch_mad)  CORBA  omniORB omniORB-3.0.2 through omniORB-4.0.6  MicoCCM mico-2.3.4 through mico-2.3.12  others Iona ORBacus-4 and Orbix/E  SOAP gSOAP-2.1 through 2.6  JVM kaffe-1.0.6  HLA Certi-3.0 (Onera)  DSM Mome (Y. Jégou)  ICE ICE-1.0 & 1.1 (ZeroC Inc.)  JXTA JXTA-C (CVS cersion)  PadicoTM 0.1.5 available  0.3 release coming very soon  Now, co-developed with RUNTIME/LaBRI  http://runtime.futurs.inria.fr/PadicoTM PadicoTM Status

25. Outline  Motivation  A Distributed Parallel Component Model  GridCCM  Handing Network Issues  PadicoTM  Automatic Deployment of Applications  ADAGE  Status  Conclusion

26. How to Deploy my Application on Grid Resources? Visualization Homogeneous cluster SAN Homogeneous cluster LAN WAN Supercomputer Flow code Transport code Controller velocity scalars concentration

27. Manual Deployment  Discover available grid resources  Select grid resources for execution  OS, architecture compatibility  Map the application onto the selected resources  MPI processes  Components  Select compatible compiled executables  Upload and install executables, stage input files in  Launch processes on remote computers  Set configuration parameters of the application  Components' attributes  Network topology information Too complex!

28. Automatic Deployment  Automatic  Resource discovery  Execution node selection  File installation  Process launch  Application configuration  Hide application complexity  Hide grid complexity stop reading your e- mails! Grid

29. Identification of the Steps of Automatic Deployment MPI Application DescriptionCCM Application Description Resource DescriptionGeneric Application DescriptionControl Parameters Deployment Planning Deployment Plan Execution Application Configuration Static Applications Deployment Tool

30. Generic Application Description  Translator  From specific to generic application description  Straightforward to write CCM Appl. Description MPI Appl. Description GridCCM Appl. Description CCM Appl. Description MPI Appl. Description GridCCM Appl. Description CCM Planner MPI Planner GridCCM Planner Plan Execution Generic Application Description Deployment Planner Deployment Plan Execution

31. ADAGE  Automatic Deployment of Applications in a Grid Environment  http://www.irisa.fr/paris/ADAGE/  Already implemented  MPI, CCM, and JXTA application description  Network topology description  Generic application description  Simple control parameters, simple planner  Deployment plan execution (SSH, Globus)  Not fully implemented yet  GridCCM application description and configuration  Integrate more planners (GRAAL, LIP, Lyon)

32. How Easy is it, in Practice?  As simple as A-B-C  adage_grid_deploy–resource http://host/file.xml –application my_appl.zip –ctrl_param my_control_parameters  Grid resource description  Written once for all by grid admins  Optional control_parameters constraints  Keep control on the deployment process

33. Conclusion  Software component is a promising technology for handling code & grid complexity  Padico = GridCCM + PadicoTM + ADAGE  http://www.irisa.fr/paris/Padico  GridCCM : extends CCM  Parallel component, scalable communication achieved  Data redistribution / communication scheduling API  Approach applicable to other component/service models  PadicoTM: an open integration framework for communication middleware and runtimes  Decouple middleware from actual networking technology  Supports several middleware at the same time  Adage: Automatic Deployment of Applications in a Grid Environment  Application/resource constraints, user goal, and mapping algorithm needed