2. Denis Caromel1 Institut universitaire de France (IUF) OASIS Team INRIA -- CNRS - I3S -- Univ. of Nice Sophia-Antipolis JAOO, Cannes, May 2004 Programming, Composing, Deploying for the GRID 1. Grid principles 2. Distributed Objects and Components: ObjectWeb ProActive LGPL environment 3. Application: 3D Electromagnetism 4. Towards Peer-To-Peer (P2P)

3. Denis Caromel2 The GRID PCs : 1 Milliard in 2002 (25 years) Forecast: 2 Milliards in 2008

4. Denis Caromel3 The Grid idea GRID = Electric Network Computer Power (CPU cycles) Electricity Can hardly be stored, if not used --> Lost But CPU cycles much harder to share than electricity: Production cannot be adjusted! Cannot really be delivered where needed! Not yet interoperability: Multiple Administrative Domains 2 important aspects : Computational + Data Grid Global management, Mutual sharing of the resource

5. Denis Caromel4 Example 50 Machines, 1.5 year of computation 5000 Machines, with only 50 % Efficiency ==> 10 days Applications: Simulate the stock market evolution Research for an urgent vaccine Forecast a bush-fire path Forecast in real time a flooding consequences... etc. Challenge: Scale up

6. Denis Caromel5 Ubiquitous: some numbers PCs in my lab (INRIA Sophia) : ~ 1500 French Riviera : 1.3 Millions France : 25 Millions Europe : 108 Millions USA : 400 Millions World : 1 Milliard in 2002 (25 ans) Forecast: 2 Milliards in 2008 France : 36 Millions of cellular phones 2.2 Millions of laptops 630 Thousand PDA (sources: ITU, Gartner Dataquest, IDC, 02-03, )

7. Denis Caromel6 The multiple GRIDs Scientific Grids : Parallel machines, Clusters Large equipments: Telescopes, Particle accelerators, etc. Enterprise Grids : Data, Integration: Web Services Remote connection, Security Intranet and Internet Grids, (miscalled P2P grid): Desktop office PCs: Desktop Intranet Grid Home PC: Internet Grid (e.g. SETI@HOME)

8. Denis Caromel7 Enterprise Grids Internet EJBServlets Apache Databases

9. Denis Caromel8 Scientific Grids Internet Clusters Parallel Machine Large Equipment

10. Denis Caromel9 Internet Grids Internet Job management for embarrassingly parallel application (e.g. SETI)

11. Denis Caromel10 The multiple GRIDs Scientific Grids Enterprise Grids Intranet and Internet Grids Strong convergence in process! At least at the infrastructure level, i.e. WS

12. Denis Caromel11 Grid: from enterprise... to regional Very hard deployment problems … right from the beginning

13. Denis Caromel12 Grid: from regional... to worldwide Communication Cannes-Los Angeles: 70 ms Light Speed Challenge: Hide the latency ! Define adequate programming model

14. Denis Caromel13 Distributed Objects and Components ProActive Programming

15. Denis Caromel14 Model: Remote Objects (Active Objects vs. Java RMI) Asynchronous Communications, with automatic Futures Group Communications, Migration (computation mobility) Environment: XML Deployment, dynamic class-loading Various protocols: rsh, ssh, LSF, Globus, BPS,... Graphical Visualization and monitoring: IC2D ProActive: A Java API + Tools for the GRID Parallel, Distributed, Mobile, Activities, across the world ! SMPClustersLAN Desktop

16. Denis Caromel15 A Creating AO and Groups Typed Group Java or Active Object A ag = newActiveGroup (“A”, […], VirtualNode) V v = ag.foo(param);... v.bar(); //Wait-by-necessity V Group, Type, and Asynchrony are crucial for Cpt. and GRID

17. Denis Caromel16 Mobility Same semantics guaranteed (RDV, FIFO order point to point, asynchronous) Safe migration (no agent in the air!) Local references if possible when arriving within a VM Tensionning (removal of forwarder)

18. Denis Caromel17 Mobility Same semantics guaranteed (RDV, FIFO order point to point, asynchronous) Safe migration (no agent in the air!) Local references if possible when arriving within a VM Tensionning (removal of forwarder)

19. Denis Caromel18 Mobility Same semantics guaranteed (RDV, FIFO order point to point, asynchronous) Safe migration (no agent in the air!) Local references if possible when arriving within a VM Tensionning (removal of forwarder) direct

20. Denis Caromel19 Mobility Same semantics guaranteed (RDV, FIFO order point to point, asynchronous) Safe migration (no agent in the air!) Local references if possible when arriving within a VM Tensionning (removal of forwarder) direct

21. Denis Caromel20 Mobility Same semantics guaranteed (RDV, FIFO order point to point, asynchronous) Safe migration (no agent in the air!) Local references if possible when arriving within a VM Tensionning (removal of forwarder) direct forwarder

22. Denis Caromel21 Mobility Same semantics guaranteed (RDV, FIFO order point to point, asynchronous) Safe migration (no agent in the air!) Local references if possible when arriving within a VM Tensionning (removal of forwarder) direct forwarder

23. Denis Caromel22 Mobility Same semantics guaranteed (RDV, FIFO order point to point, asynchronous) Safe migration (no agent in the air!) Local references if possible when arriving within a VM Tensionning (removal of forwarder) direct forwarder

24. Denis Caromel23 Mobility Same semantics guaranteed (RDV, FIFO order point to point, asynchronous) Safe migration (no agent in the air!) Local references if possible when arriving within a VM Tensionning (removal of forwarder) direct forwarder

25. Denis Caromel24 Parallel, Distributed, Hierarchical Components for the Grid Composing

26. Denis Caromel25 A CORBA Component My Business Component Component interface Facets Event sources Event sinks Attributes Receptacles OFFERED REQUIRED Courtesy of Philippe Merle, Lille, OpenCCM platform

27. Denis Caromel26 Building CCM Applications = Assembling CORBA Component Instances Provide + Use, but flat assembly

28. Denis Caromel27 Content Controller The Fractal model: Hierarchical Component Defined by E. Bruneton, T. Coupaye, J.B. Stefani, INRIA & FT

29. Denis Caromel28 Content Controller Interface = access point

30. Denis Caromel29 Content Controller Hierarchical model : composites encapsulate primitives, which encapsulates Java code

31. Denis Caromel30 Content Controller Binding = interaction

32. Denis Caromel31 Binding = interaction Content Controller

33. Denis Caromel32 Controllers : non-functional properties Component Identity Binding Controller LifeCycle Controller Content Controller Content Controller Component = runtime entity

34. Denis Caromel33 3. Parallel and composite component 1. Primitive component Java + Legacy 2. Composite component ProActive Components for the GRID An activity, a process, … potentially in its own JVM C D Composite: Hierarchical, and Distributed over machines Parallel: Composite + Broadcast (group)

35. Denis Caromel34 Composing: XML ADL Primitive-component "cd-player” implementation = "CdPlayer” // Java class with functional code Provides interface "input” … Requires … VirtualNode = VNa // Virtual Node name Composite-component ”stereo” VirtualNode = VNc, vn...// Virtual Node Provides … Requires … Primitive-component ”cd-player” Primitive-component ”speaker” Bindings bind "cd-player.output" to "speaker.input" Merging VNa, VNb, ---> VNc // Co-allocation in that case Example of an XML component file:

36. Denis Caromel35 A A B C P Group proxy A B C D Groups in Components Broadcast at binding, on client interface At composition, on composite inner server interface A parallel component!

37. Denis Caromel36 Migration Capability of composites Migrate sets of components, including composites

38. Denis Caromel37 Migration Capability of composites Migrate sets of components, including composites

39. Denis Caromel38 Co-allocation, Re-distribution e.g. upon communication intensive phase

40. Denis Caromel39 Co-allocation, Re-distribution e.g. upon communication intensive phase

41. Denis Caromel40 Co-allocation, Re-distribution e.g. upon communication intensive phase

42. Denis Caromel41 Environment Deploying

43. Denis Caromel42 How to deploy on the Various Kind of Grids ? Internet EJBServlets Apache Databases Internet Clusters Parallel Machine Large Equipment Internet Job management for embarrassingly parallel application (e.g. SETI)

44. Denis Caromel43 Abstract Deployment Model Problem: Difficulties and lack of flexibility in deployment Avoid scripting for: configuration, getting nodes, connecting, etc. A key principle: Virtual Node (VN) + XML deployment file Abstract Away from source code: Machines Creation Protocols Lookup and Registry Protocols Protocols and infrastructures: Globus, ssh, rsh, LSF, PBS, … Web Services, WSRF,...

45. Denis Caromel44 Mapping Virtual Nodes: example (1) <sshProcess class="org.objectweb.proactive.core.process.ssh.SSHJVMProcess" hostname="sea.inria.fr"> Infrastructure informations JVM on the current HostJVM started using SSH

46. Denis Caromel45 <bsubProcess class="org.objectweb.proactive.core.process.lsf.LSFBSubProcess" hostname=”cluster.inria.fr"> 12 Mapping Virtual Nodes: example (2) Infrastructure informations Definition of LSF deployment, … Globus

47. Denis Caromel46 C AB VNaVNb C AB VNc = VN(a,b) XML Deployment (Not in source) Separate or Co-allocation

48. Denis Caromel47 IC2D: Interactive Control and Debugging of Distribution

49. Denis Caromel48 Monitoring of RMI, Globus, Jini, LSF cluster Nice -- Baltimore ProActive IC2D: Width of links proportional to the number of com- munications

50. Denis Caromel49 Application

51. Denis Caromel50 A Parallel Object-Oriented Application for 3D Electromagnetism Visualize the radar reflection of a plane, medicine on head, etc. Pre-existing Fortran MPI version: EM3D Jem3D: Sequential object-oriented design, modular and extensible (in Java) Sequential version can be smoothly distributed: --> keeping structuring and object abstractions Efficient distributed version, large domains, Grid environment

52. Denis Caromel51 The interface

53. Denis Caromel52 Some Results Seq. Java/Fortran: 2 Comparison: Jem3D over - ProActive/RMI Sun - ProActive/RMI Ibis Em3D in - Fortran/MPI On 16 machines: Fortran: 13.8 ProActive/Ibis: 12 ProActive/RMI: 8.8 Grid experiment on 5 clusters (DAS 2): Speed up of 100 on 150 machines

54. Denis Caromel53 Perspective: P2P

55. Denis Caromel54 Architectures: Server to Peer-To-Peer (P2P) Internet EJBServlets Apache Databases SOA: Service Oriented Architectures

56. Denis Caromel55 Pure P2P: Definition Only PEERs, no above everything, top level, server(s) Every peer is, somehow, also a server No master … No slave ! System get organized dynamically, without static configuration Coherent, desired behavior, dynamically emerges

57. Denis Caromel56 P2P Examples (1)

58. Denis Caromel57 P2P can be difficult: need to be fault-tolerant, self healing

59. Denis Caromel58 Not a P2P system

60. Denis Caromel59 Neither a P2P system

61. Denis Caromel60 P2P Examples (2)

62. Denis Caromel61 P2P Examples (2bis)

63. Denis Caromel62 A P2P system at work Credit: from the movie Atlantis, Luc Besson

64. Denis Caromel63 Conclusion GRIDs: - Scientific - Enterprise - Internet Strong convergence in process (infrastructure): WS, WSRF Challenge: adequate and precise programming+composing model Asynchronous distributed objects, Mobility, Components High-level of abstraction, still Open and flexible Modern languages: Java, or others not Fortran, MPI, C++, … not the answer Perspectives: Real P2P Interactive, Graphical, Deployment and Control: Content http://ProActive.ObjectWeb.org

65. Denis Caromel64 Interactive Composition in IC2D Content Composition View Instead of XML ADL <sshProcess class="org.objectweb.proactive.core.process.ssh.SSHJVMProcess" hostname="sea.inria.fr">

66. Denis Caromel65 Merging Component + Activity / JVM Views: Component Monitoring Content Dynamic View

67. Denis Caromel66 Content Dynamic View Merging Component + Activity / JVM Views: Component Monitoring

68. Denis Caromel67 Conclusion (2) Infrastructure and Deployment Virtual Nodes and XML Protocols: ssh, Globus, LSF, PBS, … Transport: RMI, Ibis (Amsterdam) Web Services: XML, WSDL, SOAP (ongoing) OSGi (future work) Available in LGPL with ProActive http://ProActive.ObjectWeb.org in The freedom your applications deserve! Currently: a single application on 300 machines Goal: towards a few 1000s !

69. Denis Caromel68

70. Denis Caromel69

71. Denis Caromel70 ProActive Component Definition A component is: Formed from one (or several) Active Object Executing on one (or several) JVM Provides a set of server ports: Java Interfaces Uses a set of client ports: Java Attributes Point-to-point or Group communication between components Hierarchical: Primitive component: define with Java code and a descriptor Composite component: composition of primitive + composite Parallel component: multicast of calls in composites Descriptor: XML definition of primitive and composite (ADL) Virtual nodes capture the deployment capacities and needs Virtual Node is a very important abstraction for GRID components

72. Denis Caromel71 Virtual Nodes in Programs (1) Load the descriptor file Descriptor pad = ProActive.getDescriptor ("file://ProActiveDescriptor.xml"); Activate the mapping VirtualNode vn = pad.activateMapping ("Dispatcher"); // Triggers the JVMs Use nodes Node node = vn.getNode();... C3D c3d = ProActive.newActive("C3D", param, node);

73. Denis Caromel72 Descriptors: Virtual Nodes in Programs (2) Descriptor pad = ProActive.getDescriptor ("file:.ProActiveDescriptor.xml"); VirtualNode vn = pad.activateMapping ("Dispatcher"); // Triggers the JVMs Node node = vn.getNode();... C3D c3d = ProActive.newActive("C3D", param, node); log (... "created at: " + node.name() + node.JVM() + node.host() );

74. Denis Caromel73 Descriptors: Virtual Nodes in Programs (3) Descriptor pad = ProActive.getDescriptor ("file:.ProActiveDescriptor.xml"); VirtualNode vn = pad.activateMapping ("Dispatcher"); // Triggers the JVMs Node node = vn.getNode();... C3D c3d = ProActive.newActive("C3D", param, node); log (... "created at: " + node.name() + node.JVM() + node.host() ); // Cyclic mapping: set of nodes VirtualNode vn = pad.activateMapping ("RendererSet"); while ( … vn.getNbNodes … ) { Node node = vn.getNode(); Renderer re = ProActive.newActive(”Renderer", param, node);

75. Denis Caromel74 Component Orientedness Level 1: Instantiate - Deploy - Configure Simple Pattern Meta-information (file, XML, etc.)JavaBeans, EJB Level 2: Assembly (flat) Server and client interfacesCCM Level 3: Hierarchic CompositeFractal, ProActive,... Level 4: Distributed + Reconfiguration Binding, Inclusion, Location ProActive + On going work Interactions / Communications: Functional Calls: service, event, stream Non-Functional: instantiate, deploy, start/stop, inner/outer, re-bind This talk

76. Denis Caromel75 C3D: distributed-//-collaborative