1. Production Grids Fabrizio Gagliardi Director, Technical Computing, Microsoft EMEA Computer and Information Sciences Life Sciences Multidisciplinary Research Math and Physical Science Earth Sciences General Overview Social Sciences New Materials, Technologies and Processes

2. Talk Outline  A few personal remarks  Need for e-Infrastructures and Grids  Background history  Outlook to the future  MS contribution

3. A few personal remarks  GGF1 and EDG  EGEE and GGF experience  Acknowledgements  Move to MS

4. Need for e-Infrastructures  Science, industry and commerce are more and more digital, process vast amounts of data and need massive computing power  We live in a “flat” world:  Science is more and more an international collaboration and often requires a multidisciplinary approach  Need to use technology for the good cause  Fight Digital/Divide  Industrial uptake has become essential

5. Some examples (from EU-EGEE): Earth sciences applications  Satellite Observations:  ozone profiles  Solid Earth Physics  Fast Determination of mechanisms of important earthquakes  Hydrology  Management of water resources in Mediterranean area (SWIMED)  Geology  Geocluster: R&D initiative of the Compagnie Générale de Géophysique

6. GGF16, Athens, 13 February 2006 6 SEEREN > http://www.see-grid.org The SEE-GRID initiative

7. GGF16, Athens, 13 February 2006 7 SEE-GRID e-Infrastructure Infrastructure use for regional applications Visualization of Medical diagnostic images, Earth sciences, Thermodynamics, Fluid dynamics, etc. Already deployed and used in University Children’s Hospital in Belgrade. Search Engine for South-East Europe (SE4SEE) Grid-based Web-crawling and domain- specific Web page filtering service From the original 21 organizations in from 11 countries to 40 organizations in two years

8. Drug Discovery: a new hope for developing countries  Grid-enabled drug discovery process  Reduce time required to develop drugs  Develop the next steps of the process (molecular dynamics)  Data challenge proposal for docking on malaria  Never done on a large scale production infrastructure  Never done for a neglected disease  Data challenge during the summer  5 different structures of the most promising target  Output Data: 16,5 million results, ~10 TB  Added value  Facilitates inclusion of developing countries  Tool to enhance collaboration building  Facilitate distributed software development for complex integrated workflows

9. The underlining infrastructure

10. Enabling Grids for E-sciencE HP HPC Forum Divonne BioMed Grid Infrastructure –~2.000 CPUs –~21 TB disks –in 12 countries >50 users in 7 countries working with 12 applications 18 research labs ~80.000 jobs launched since 04/2004 ~10 CPU years Month Number of jobs

11. The EGEE Grid www.eu-egee.org

12. The EU research network

13. 9/2/2006HP HPC Forum Divonne Backbone Network and Athens Metropolitan Area Network (MAN) : 2,5 Gbps lambda (Packet over SDH-PoS) Crete Regional Network and Major Institute Access: Gigabit Ethernet Dark fibre acquired for 2 network spans Major dark fibre tender for the whole backbone underway HellasGrid Grid infrastructure hosted by major Research & Academic Institutes in Athens (Demokritos, IASA, NDC), Crete (ICS-FORTH), Patra (CTI-CEID), Thessaloniki (AUTH-UoM): ~800 CPUs (x86_64, 2 GB RAM, 80GB HDD, 2x Gbit) ~30 TB total raw SAN storage capacity in all other HG nodes ~60 TB Tape Library 4 Access Grid nodes GRNET e-Infrastructure

14. 9/2/2006HP HPC Forum Divonne Infrastructure and targeted projects funded by HellasGrid 2M Euro project Operations, training, policies supported mainly by EC co-funded FP6 project EGEE EGEE major catalyst to secure HellasGrid funding Major institutes involved as Third Parties in the EGEE project EuGridPMA accredited Certification Authority Applications areas supported by local Ministry: Physics, Biomedical, Chemistry, other Major call for applications integration under way ~ 0,5M Euro (might be upgraded to > 1M because of great interest shown) HellasGrid Production Infrastructure

15. Current situation: accomplishments and challenges  Many Grids around the world, very few maintained as a persistent infrastructure  Need for public and open Grids (OSG, EGEE and related projects, NAREGI, and TERAGRID, DEISA good prototypes)  Persistence, support, sustainability, long term funding, easy access are the major challenges

16. Key Challenges  Security (see Blair Dillaway’s talk)  Stable accepted industrial standards (GGF and EGA converging)  Learning curve for applications  Complexity of running a Grid infrastructure across different administrative domains

17. How did we get here?  Meta-computing and distributed computing early examples in the 80’ and 90’ (CASA, I- Way, Unicore, Condor etc.)  EU FP5 and US Trillium and national Grids  EU FP6, US OSG, NAREGI/Japan…  UK e-Science programme and similar other national programmes

18. EU Information Day 25 03 2003 EDG status 18 An ecosystem of prototype Grid projects in US, EU and A/P in 2003 Related Grid Projects

19. GRID projects in EU IST-FP5 (1999/2003) (~36m Euro) Science Industry / businessApplications Middleware & Tools Underlying Infrastructures CROSSGRID DATAGRID DATATAG GRIDLAB EGSO GRIA GRIP EUROGRID DAMIEN GRIDSTART

20. Enabling Grids for E-sciencE HP HPC Forum Divonne Projects in Europe (I) Access to IT-resources (connectivity, computing, data, instrumentation…) for scientists: –Providing e-Infrastructure  Géant2  EGEE  DEISA  SEE-GRID –Benefiting from e-Infrastructure  DILIGENT  SIMDAT  GRIDCC  CoreGRID  GridLab –Concertation: GRIDSTART, GridCoord –Grid mobility: Akogrimo

21. Enabling Grids for E-sciencE HP HPC Forum Divonne Projects in Europe (II) Sample of National Grid projects: –Austrian Grid Initiative –DutchGrid –France:  e-Toile  ACI Grid –Germany  D-Grid  Unicore –Grid Ireland –Italy  INFNGrid  GRID.IT –NorduGrid –UK e-Science  National Grid Service  OMII  GridPP project D-GRID

22. Where are we going?

23. Top 500 Supercomputer Trends Industry usage rising Clusters over 50% IA is winning GigE is gaining

24. Supercomputing Goes Personal 199119982005 SystemCray Y-MP C916Sun HPC10000Shuttle @ NewEgg.com Architecture 16 x Vector 4GB, Bus 24 x 333MHz Ultra- SPARCII, 24GB, SBus 4 x 2.2GHz x64 4GB, GigE OS UNICOSSolaris 2.5.1Windows Server 2003 SP1 GFlops ~10 Top500 # 1500N/A Price $40,000,000$1,000,000 (40x drop)< $4,000 (250x drop) Customers Government LabsLarge EnterprisesEvery Engineer & Scientist Applications Classified, Climate, Physics Research Manufacturing, Energy, Finance, Telecom Bioinformatics, Materials Sciences, Digital Media

25. The Future: Supercomputing on a Chip IBM Cell processor 256 Gflops today 4 node personal cluster => 1 Tflops 32 node personal cluster => Top100 MS Xbox 3 custom PowerPCs + ATI graphics processor 1 Tflops today $300 8 node personal cluster => “Top100” for $2500 (ignoring all that you don’t get for $300) Intel many-core chips “100’s of cores on a chip in 2015” (Justin Rattner, Intel) “4 cores”/Tflop => 25 Tflops/chip

26. The Continuing Trend Towards Decentralized, Networked Resources Grids of personal & departmental clusters Personal workstations & departmental servers Minicomputers Mainframes

27. MS HPC road map Computer Cluster Solution (CCS) V1: Beowulf-style compute cluster seamlessly integrated into a Windows-based infrastructure, including security infrastructure (Kerberos and Active Directory); user jobs run under the Windows user credentials ISVs support for most common software applications Job scheduler accessible via command-line, COM and via a published Web services protocol. Can be customized via sys admin-defined admission and release filters that run in the scheduler when a job is submitted and when it becomes scheduled as ready-to-run. In V2 the scheduler will be made even more extensible. Performance will be comparable to Linux MPICH (incl. MPICH-2) supported (open-source version from ANL)

28. MS HPC road map CCS areas of focus for V2: Extend to forests of clusters and meta-schedulers Storage and parallel I/O issues, as well as possibly simple workflow support Development of new the tools

29. Conclusions Production Grids are a reality and here to stay HPC is becoming commodity seamlessly stretching from desktop to back-end resources MS HPC products are coming soon Microsoft is participating in major standardisation bodies MS is present at this meeting and prime sponsor

30. Wish everybody a fruitful and positive week in Athens you can contact me at: fabrig@microsoft.com