1. Congreso Cuidad, Spain May 15, 2007 GridWay 1/26 EGEE’07 Conference Budapest, Hungary October 1st – 5th, 2007 Uniform Grid Access with GridWay GridWay José Luis Vázquez-Poletti Distributed Systems Architecture Group Universidad Complutense de Madrid

2. 2/29 GridWay DSA Group GridWay DSA Group Contents 1.What is GridWay? 2.A Global Vision 3.The GridWay Project 4.Uniform Access Methods

3. 3/29 GridWay DSA Group GridWay DSA Group What is GridWay? GridWay is a Globus Toolkit component for meta-scheduling, creating a scheduler virtualization layer on top of Globus services (GRAM, MDS & GridFTP) For project and infrastructure directors GridWay is an open-source community project, adhering to Globus philosophy and guidelines for collaborative development. For system integrators GridWay is highly modular, allowing adaptation to different grid infrastructures, and supports several OGF standards. For system managers GridWay gives a scheduling framework similar to that found on local LRM systems, supporting resource accounting and the definition of state-of-the-art scheduling policies. For application developers GridWay implements the OGF standard DRMAA API (C and JAVA bindings), assuring compatibility of applications with LRM systems that implement the standard, such as SGE, Condor, Torque,... For end users GridWay provides a LRM-like CLI for submitting, monitoring, synchronizing and controlling jobs, that could be described using the OGF standard JSDL.

4. 4/29 GridWay DSA Group GridWay DSA Group Contents 1.What is GridWay? 2.A Global Vision 3.The GridWay Project 4.Uniform Access Methods

5. 5/29 GridWay DSA Group GridWay DSA Group Global Architecture of a Computational Grid Application-Infrastructure decoupling PBS GridWay SGE $> CLI Results.C,.java DRMAA.C,.java Infrastructure Grid Middleware Applications Globus Grid Meta- Scheduler Standard API (OGF DRMAA) Command Line Interface open source job execution management resource brokering Globus services Standard interfaces end-to-end (e.g. TCP/IP) highly dynamic & heterogeneous high fault rate A Global Vision

6. 6/29 GridWay DSA Group GridWay DSA Group Integration of non-interoperable computational platforms (Organization) Establishment of a uniform and flexible infrastructure Achievement of greater utilization of resources and higher application throughput Support for the existing platforms and LRM Systems (Sys. Admin.) Allocation of grid resources according to management specified policies Analysis of trends in resource usage Monitoring of user behavior Familiar CLI and standard APIs (End Users & Developers) High Throughput Computing Applications Workflows Benefits A Global Vision

7. 7/29 GridWay DSA Group GridWay DSA Group Workload Management Advanced (Grid-specific) scheduling policies Fault detection & recovery Accounting Array jobs and DAG workflows User Interface OGF standards: JSDL & DRMAA (C and JAVA) Analysis of trends in resource usage Command line interface, similar to that found on local LRM Systems Integration Straightforward deployment as new services are not required Interoperability between different infrastructures Features A Global Vision

8. 8/29 GridWay DSA Group GridWay DSA Group Execution Manager Transfer Manager Information Manager Dispatch Manager Request Manager Scheduler Job PoolHost Pool DRMAA libraryCLI GridWay Core Grid File Transfer Services Grid Execution Services GridFTPRFT pre-WS GRAM WS GRAM Grid Information Services MDS2 GLUE MDS4 Resource Discovery Resource Monitoring Resource Discovery Resource Monitoring Job Preparation Job Termination Job Migration Job Preparation Job Termination Job Migration Job Submission Job Monitoring Job Control Job Migration Job Submission Job Monitoring Job Control Job Migration A Global Vision GridWay Internals

9. 9/29 GridWay DSA Group GridWay DSA Group A Tool for Interoperability SGE Cluster Users Globus/WS GridWay Globus/WS gLite SGE ClusterPBS Cluster Different Middlewares (e.g. WS and pre-WS) Different Data/Execution architectures Different Information models Integration through adapters Global DN’s Demo in June 2007, TeraGrid07 Globus/WS SGE ClusterPBS Cluster A Global Vision

10. 10/29 GridWay DSA Group GridWay DSA Group A Global Vision Matching Resources for each job (user)‏ Pending Jobs Grid Scheduling = Job + Resource Policies Resource Policies Rank Expressions Fixed Priority User Usage History Failure Rate Job Policies Fixed Priority Urgent Jobs User Share Deadline Waiting Time

11. 11/29 GridWay DSA Group GridWay DSA Group Contents 1.What is GridWay? 2.A Global Vision 3.The GridWay Project 4.Uniform Access Methods

12. 12/29 GridWay DSA Group GridWay DSA Group Started in 2002, first releases were only distributed on request in binary format First open source release (v4.0) in January 2005 (Apache license v2.0) Adhering to Globus philosophy and guidelines for collaborative development In June 2007 GridWay became part of the Globus Toolkit Since January 2005, more than 1000 downloads from 80 different countries, 25% are private companies and 75% are universities and research centers. Best-effort support provided (contract support is also available) Based on a strong open source community History of the Project The GridWay Project

13. 13/29 GridWay DSA Group GridWay DSA Group Community – Open Source Project. Globus Development Philosophy Development Infrastructure (thanks to Globus Project!) Mailing Lists Bugzilla CVS You are very welcome to contribute: Reporting Bugs (gridway-user@globus.org) Making feature requests for the next GridWay release (gridway-user@globus.org) Contributing your own developments (bug fixes, new features, documentation) Detailed Roadmap: GridWay Campaigns at bugzilla.mcs.anl.gov/globus/query.cgi www-unix.mcs.anl.gov/~bacon/cgi-bin/big-roadmap.cgi#Gridway Development Process The GridWay Project

14. 14/29 GridWay DSA Group GridWay DSA Group Life-Sciences Aerospace Fusion Physics Computational Chemistry  IRISGrid  Politecnico di Torino  CABGrid (Centro de Astrobiología)‏  C2VO (Universidad de Castilla La Mancha)‏  Grid en ESAC (Agencia Espacial Europea)‏  CRO-GRID (Croacia)‏  Sun Microsystems Solution Center World Grid  Infraestructura EGEE  Proyecto BeinGRID  GridX1 (Canadian Grid for HEP applications)‏  Universidade do Porto  Madras Institute of Technology  National Center for High-Performance Computing More at: http://www.gridway.org/ (Success Stories)‏ The GridWay Project Some Projects and Infrastructures Some Application Porting Areas

15. 15/29 GridWay DSA Group GridWay DSA Group European Space Astronomy Center Data Analysis from space missions (DRMAA) Site-level meta-scheduler Several clusters In Detail… The GridWay Project

16. 16/29 GridWay DSA Group GridWay DSA Group UABGrid, University of Alabama at Birmingham Bioinformatics applications Campus-level meta-scheduler 3 resources (PBS, SGE and Condor) In Detail… The GridWay Project

17. 17/29 GridWay DSA Group GridWay DSA Group AstroGrid-D, German Astronomy Community Grid Collaborative management of supercomputing resources & astronomy-specific resources Grid-level meta-scheduler (GRAM interface) 22 resources @ 5 sites, 800 CPUs In Detail… The GridWay Project

18. 18/29 GridWay DSA Group GridWay DSA Group SGE Cluster Users PBS Cluster gLite GridWay gLite Services: BDII, GRAM, GridFTP EGEE Resource Broker DRMAA interface VO Schedulers GridWay Users Biomed Fusion EGEE RB Massive Ray Tracing CD-HIT workflow In Detail… The GridWay Project

19. 19/29 GridWay DSA Group GridWay DSA Group Contents 1.What is GridWay? 2.A Global Vision 3.The GridWay Project 4.Uniform Access Methods

20. 20/29 GridWay DSA Group GridWay DSA Group Uniform Grid Access Methods Similar to that found in LRM Systems –Easy to learn –Intuitive Example: List resources Example: Check job status Command Line Interface HID PRIO OS ARCH MHZ %CPU MEM(F/T) DISK(F/T) N(U/F/T) LRMS HOSTNAME 0 1 Linux2.6.17-2-6 x86 3216 0 44/2027 76742/118812 0/0/2 Fork cygnus.dacya.ucm.es 1 1 0 0 0/0 0/0 0/0/0 orion.dacya.ucm.es 2 1 Linux2.6.18-4-a x86_6 2211 100 819/1003 77083/77844 0/2/4 PBS hydrus.dacya.ucm.es 3 1 Linux2.6.17-2-6 x86 3216 163 1393/2027 101257/118812 0/2/2 Fork draco.dacya.ucm.es 4 1 Linux2.6.18-4-a x86_6 2211 66 943/1003 72485/77844 0/5/5 SGE aquila.dacya.ucm.es $ gwhost 0 HID PRIO OS ARCH MHZ %CPU MEM(F/T) DISK(F/T) N(U/F/T) LRMS HOSTNAME 0 1 Linux2.6.17-2-6 x86 3216 0 50/2027 76393/118812 0/0/2 Fork cygnus.dacya.ucm.es QUEUENAME SL(F/T) WALLT CPUT COUNT MAXR MAXQ STATUS DISPATCH PRIORITY default 0/2 0 -1 0 -1 0 enabled NULL 0 $ gwps USER JID DM EM START END EXEC XFER EXIT NAME HOST gwtutorial00 0 done ---- 20:16:28 20:18:16 0:00:55 0:00:08 0 stdin aquila.dacya.ucm.es/SGE tinova 1 done ---- 12:26:46 12:31:15 0:03:55 0:00:08 0 stdin hydrus.dacya.ucm.es/PBS tinova 2 pend ---- 12:38:38 --:--:-- 0:00:00 0:00:00 -- t.jt --

21. 21/29 GridWay DSA Group GridWay DSA Group Uniform Grid Access Methods OGF DRMAA Distributed Resource Management Application API –http://www.drmaa.org/ Open Grid Forum Standard Homogeneous interface to different Distributed Resource Managers (DRM): –SGE –Condor –PBS/Torque –GridWay –C –JAVA –Perl (coming soon!) –Ruby (coming soon!) –Python (coming soon!)

22. 22/29 GridWay DSA Group GridWay DSA Group Uniform Grid Access Methods Application Widgets (i.e. Microsoft Excel)

23. 23/29 GridWay DSA Group GridWay DSA Group Uniform Grid Access Methods SGE Transfer Queues

24. 24/29 GridWay DSA Group GridWay DSA Group Uniform Grid Access Methods GridGateWay SGE Cluster Users Globus GridWay Infrastructure PBS Cluster Applications Globus Middleware SGE ClusterPBS Cluster GridWay Globus gLite Access to different infrastructures with the same adapters EGEE managed as other resource Delegate identity/ “VO” certificates In-house/provider gateway Regional infrastructure

25. 25/29 GridWay DSA Group GridWay DSA Group Uniform Grid Access Methods GridWay provides many easy ways to access the Grid but… … no web portal access system has been developed yet. Some portal solutions have shown interest in GridWay: GridSphere P-Grade EngineFrame We offer: Collaboration for implementation/integration of GridWay based web portal “Our two Forint Cents” Look for us at Demo Booths!

26. 26/29 GridWay DSA Group GridWay DSA Group Thank you for your attention! Uniform Grid Access with GridWay