1. F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal Grid eXplorer (GdX) An Instrument for eXploring the GRID F. Cappello, O. Richard, P. Sens (fci@lri.fr, olivier.richard@imag.fr, pierre.sens@lip6.fr) fci@lri.frolivier.richard@imag.frpierre.sens@lip6.frfci@lri.frolivier.richard@imag.frpierre.sens@lip6.fr LRI, ID-Imag, Lip6, etc.

2. F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal What we know about Grid/P2P? Infrastructure features: Large scale distributed systems Large scale distributed systems Dynamic infrastructures (network, nodes) Dynamic infrastructures (network, nodes) Dynamic workloads Dynamic workloads Heterogeneous (parallel & uniproc. nodes) Heterogeneous (parallel & uniproc. nodes) High Computation/Communication perf. Ratio High Computation/Communication perf. Ratio

3. F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal What we know about Grid/P2P? Application features: Large data sets (data bases, physics Large data sets (data bases, physics instruments, generation of very large results) instruments, generation of very large results) Huge computations (in terms of #operations) Huge computations (in terms of #operations) Long run times Long run times Large number of data operations (movements, transactions, etc.) Large number of data operations (movements, transactions, etc.)

4. F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal What we know about Grid/P2P? Utilization: A large number of users (researcher, industry, individual, etc.) A large number of users (researcher, industry, individual, etc.) A large set of applications (research, business) A large set of applications (research, business) High throughput (file download, multi-parameters, computation, etc.) High throughput (file download, multi-parameters, computation, etc.) Real time (Sensors, Entertainment, HPC, etc.) Real time (Sensors, Entertainment, HPC, etc.)

5. F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal What are open issues? Security (GSI, CAS) Security (GSI, CAS) Data Storage/consultation/movement Data Storage/consultation/movement Multi users/ Multi applications scheduling Multi users/ Multi applications scheduling Coordination (virtual, ephemeral infrastructure) Coordination (virtual, ephemeral infrastructure) Programming Programming Fault Tolerance! Fault Tolerance! Scalability Scalability Performance Performance Easy/efficient deployment techniques Easy/efficient deployment techniques Application characterization techniques Application characterization techniques Etc. Etc.

6. F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal What is Grid today? Middleware: Globus, Legion, Netsolve, Unicore, DIET, Condor, XtremWeb, Boinc, NWS Middleware: Globus, Legion, Netsolve, Unicore, DIET, Condor, XtremWeb, Boinc, NWS  they are actually working! Testbed: DataGRID, TeraGRID, Etoile, Grads, XW, Boinc Testbed: DataGRID, TeraGRID, Etoile, Grads, XW, Boinc  Difficult to build (debug, human factor, etc.) Dedicated applications (SETI, Kazaa, Jabber, etc.) Dedicated applications (SETI, Kazaa, Jabber, etc.)  they are working already (at large scale!),  BUT they address much less issues!

7. F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal Why Grid is so difficult?  Grids are very complex systems (we still have problems with large scale parallel computers!, we have less control on Grid resources! )  Many issues are addressed simultaneously  We need a methodology enabling the study of Grid issues, independently but realistically.

8. F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal What are the current approaches? Simulators: SimGRID, MicroGRID, etc. Simulators: SimGRID, MicroGRID, etc.  they have strong limitations (scalability, != than execution of real codes, validation) Experimental testbed (is there any?) Experimental testbed (is there any?)  Most testbed are for production, each testeb is specific, representativeness?  Most testbed are for production, each testeb is specific, representativeness?  We have no way to test: ideas independently, at a significant scale, ideas independently, at a significant scale, with realistic parameters and behaviors! with realistic parameters and behaviors!

9. F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal What is missing? A full fledge scientific environment A full fledge scientific environment (reproducible realistic experimental conditions) Probes measuring the performance of real resources and networks (Ganglia, NWS, la grenouille) Probes measuring the performance of real resources and networks (Ganglia, NWS, la grenouille) Fully experimental testbed, (GRID 5000 would remove this lack) Fully experimental testbed, (GRID 5000 would remove this lack)  Not enough, we need instruments with parametrisable reproducible experimental conditions

10. F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal Methodology When physicists can’t measure a phenomenon When physicists can’t measure a phenomenon because a) the item to measure is not reachable or b) the phenomenon is hidden by others, because a) the item to measure is not reachable or b) the phenomenon is hidden by others,  They build instruments!  And they observe…

11. F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal Grid eXplorer An experimental conditions database or generation An experimental hardware platform A tool set for conducting experiments & measurements

12. F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal Grid eXplorer: Analogy with physic instruments A set of sensors Inside real platform A database of Real platform A Hardware Platform for Emulation Simulation A tool set for Observation Results Analysis Validation on real platforms

13. F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal Grid eXplorer (GdX): Instrument for Grid exploration: An instrument for exploring Grid middleware, algorithms, performance and applications under reproducible experimental conditions An instrument for exploring Grid middleware, algorithms, performance and applications under reproducible experimental conditions A tool set for emulation/simulation of large scale distributed systems A tool set for emulation/simulation of large scale distributed systems 1K CPU clusters + configurable network & OS 1K CPU clusters + configurable network & OS For Grid, P2P, etc. For Grid, P2P, etc. Potentially connected to Grid testbeds Potentially connected to Grid testbeds F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal

14. Grid eXplorer (GdX): eXperimental conditions database A set of sensors (Nodes, Networks): A set of sensors (Nodes, Networks):  Academic Networks (x K nodes, GRID 5000)  ADSL (la grenouille  60 K nodes) A common format for traces A common format for traces A tool set for accessing, managing traces A tool set for accessing, managing traces Tools for trace analysis Tools for trace analysis F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal

15. Grid eXplorer (GdX): Hardware platform for eXperiments: 1K CPU clusters 1K CPU clusters configurable network configurable network configurable OS configurable OS Multi-users Multi-users Potentially located/managed by IDRIS or CEA Potentially located/managed by IDRIS or CEA F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal

16. Grid eXplorer (GdX): Tool set: Emulators (folding 10K virtual nodes on 1K nodes) Emulators (folding 10K virtual nodes on 1K nodes) Parallel simulators (difficult!) Parallel simulators (difficult!) Virtual GRID environment (1k virtual nodes on 1k nodes) Virtual GRID environment (1k virtual nodes on 1k nodes) Measurement tools, Measurement tools, Visualization tools. Visualization tools. F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal

17. Grid eXplorer (GdX): eXperiments: Real scale pseudo-emulation (1:1 scale) Real scale pseudo-emulation (1:1 scale) Large scale emulation (10:1 scale) Large scale emulation (10:1 scale) Large scale simulation (100:1 scale) Large scale simulation (100:1 scale) Application test (bottleneck discovery, performance evaluation, optimization) Application test (bottleneck discovery, performance evaluation, optimization) Infrastructure test (bottleneck discovery, performance evaluation, optimization) Infrastructure test (bottleneck discovery, performance evaluation, optimization) Connection to Grid 5000, International Grids, etc. Connection to Grid 5000, International Grids, etc. F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal

18. Grid eXplorer (GdX): Instrument related projects: Instrument projects Instrument projects – Cluster management (ID Imag), – Sensor event data base (LRI-fci, la grenouille), – Event analysis (ID Imag), – Emulators (LRI-fci, etc.), – Experimental conditions injection, – Result visualization (???), – Result analysis, – Validation methodology, – User validation (physicists, biologists, chemists, etc.) F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal

19. Grid eXplorer (GdX): Grid/P2P research projects: Grid Grid – DataGRID technology (???) – GRIDRPC (LIP, Desprez?) – Code coupling (IRISA?) P2P P2P – Large scale desktop Grids (LRI-fci) – Large scale storage (Laria, Irisa?) – Large scale data consistency (Lip6-P. sens) – Large scale fault tolerance (LRI-Beauquier) Network Network – Emulation of latency, packet loss, bandwidth – Emulation of topology F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal

20. Grid eXplorer (GdX): User applications: Bio Grid (Genomining, IBBMC, etc.) Bio Grid (Genomining, IBBMC, etc.) Medi GRID (???) Medi GRID (???) Geo GRID (IFP, CGG) Geo GRID (IFP, CGG) Power GRID (EDF) Power GRID (EDF) Industry GRID (EADS, Alcatel Space) Industry GRID (EADS, Alcatel Space) Academic GRID (Orsay, Berkeley, SDSC, Suitzerland, etc.) Academic GRID (Orsay, Berkeley, SDSC, Suitzerland, etc.) Industry Partners (GridXpert???, IBM???) Industry Partners (GridXpert???, IBM???) F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal

21. Grid eXplorer (GdX) Project organisation A Virtual laboratory A Virtual laboratory A director (or a group), A director (or a group), A scientific director, an administrative director A scientific director, an administrative director A scientific council (with physicists), A scientific council (with physicists), A Researcher group A Researcher group A group of engineers, A group of engineers, A user group A user group A set of scientific events A set of scientific events F. Cappello, O. Richard, P. Sens ---oo Draft oo--- Contact us for experiment proposal

22. Grid eXplorer GdX A long term effort A long term effort – A medium term milestone: 4 years  A fully functional prototype Many scientific issues (large scale emulation, experimental conditions injection, distance to reality, etc.) Many scientific issues (large scale emulation, experimental conditions injection, distance to reality, etc.) A tool for Grid users or potential users A tool for Grid users or potential users A tool for Grid/P2P developers A tool for Grid/P2P developers