1. Interoperability Achieved by GADU in using multiple Grids. OSG, Teragrid and ANL Jazz Presented by: Dinanath Sulakhe Mathematics and Computer Science Division Argonne National Laboratory Computational Institute University of Chicago

2. GADU Applications… Its all about Comparative analysis Insights of Biology are gained by Comparative Analysis: Unknown genes are compared against known. Unknown genes are compared against known. Similar genes tend to perform same functions. Similar genes tend to perform same functions. Comparative analysis to know what is same and different between two strains of an Organism: Example: What is different a organism living Boiling temperature such as 108 deg Celsius and the one living in extreme freezing conditions. Example: What is different a organism living Boiling temperature such as 108 deg Celsius and the one living in extreme freezing conditions. Difference between Pathogenic and non-pathogenic organisms. Mycobecterium Tuberculosis is a Pathogen causing TB, is only 12 genes different from the non-pathogenic BCG used as vaccine against TB. Mycobecterium Tuberculosis is a Pathogen causing TB, is only 12 genes different from the non-pathogenic BCG used as vaccine against TB. Tools BLAST, Blocks, Chisel, Interpro etc.. An embarrassingly parallel workload. An embarrassingly parallel workload.

3. GADU’s evolution.. GADU Just evolved into what it is today. Chiba City at Argonne. Jazz Cluster at Argonne. Grid2003 to OSG Teragrid All of them togeather.

4. Some Results and Highlights GADU can successfully use OSG and Teragrid resources simultaneously. GADU can successfully use OSG and Teragrid resources simultaneously. Individual clusters such as ANL Jazz is also used parallely. Individual clusters such as ANL Jazz is also used parallely. Site selection and scheduling across multiple grids. Easily add a new site into the pool of sites. StatusSite NameSite TestMaxNodesGridcat ASGC_OSG18199Pass FNAL_FERMIGRID12 Pass FNAL_GPFARM266749Pass GRASE-CCR-U21142112Pass NebraskaFAIL_TIMEOUT252Pass OSG_LIGO_PSU28312Pass Purdue-ITaP131224Pass Purdue-Physics1463Pass STAR-BNLFAIL_TIMEOUT672Pass UFlorida-PG279268Pass UMATLASFAIL_TIMEOUT771Pass UTA_DPCC18154Inactive UWMadisonCMSFAIL_TIMEOUT90Pass grow-UNI-PFAIL_TIMEOUT17Pass TG_UC44316NONE TG_NCSA551000NONE TG_PURDUEFAIL_FTP1024NONE Last Run.. ( Last week) Ran 38830 BLAST Jobs 70% OSG 30% Teragrid

5. Grid Resources.. Open Science Grid and Teragrid. Authentication. OSG OSG OSG : GADU VOMS Server. DOE Grid Certificates are automatically picked by the Sites. TeraGrid TeraGrid Individual Accounts via Allocations. Manually adding DOE Grid certificates to each site. (gx-map). Application Deployment. OSG OSG OSG variables, $OSG_APP and $OSG_DATA is used to install GADU’s applications and pre-stage the databases such as NR. TeraGrid TeraGrid GADU has a Community space on each of the sites available. Applications and installed within this community space.

6. Resource Independent GADU. GADU uses Pegasus based VDS and Condor-G Globus GRAM Interface Pegasus DAGMan Condor-G tc.data Pool.config Abstract Workflow as VDL Condor Submit files Submit Host WN Job management system Gatekeeper JobManager WN Remote Resources WN Job management system Gatekeeper JobManager WN Job management system Gatekeeper JobManager WN Information Services GADU’s automated Analysis Server, expressing, executing and tracking the scientific workflows on Grid. Database Controller Query Interface

7. Resource Independent GADU. GADU uses Pegasus based VDS and Condor-G The Workflow Generator in GADU is responsible for producing a workflow suitable for execution in the Grid environment. This task is accomplished through the use of the “virtual data language” (VDL). Once the VDL for the workflow is written, VDS converts it into condor submit files and a DAG that can be submitted to the site selected by the site selector. TR FileBreaker(input filename, none nodes, output sequences[], none species) { argument = ${species}; argument = ${filename}; argument = ${nodes}; profile globus.maxwalltime = "300"; } TR BLAST( none OutPre, none evalue, input query[], none type ) { argument = ${OutPre}; argument = ${evalue}; profile globus.maxwalltime = "300"; } DV jobNo_1_1separator->FileBreaker ( filename=@{input:"inputfile.1"|rt}, nodes="5", sequences=[@{output:"job1.0":"tmp"}, @{output:"job1.1":"tmp"}, @{output:"job1.2":"tmp"}, @{output:"job1.3":"tmp"}, @{output:"job1.4":"tmp"} ], species="Aeropyrum_Pernix" ) …. VDL for BLAST workflow

8. Resource Independent GADU. 4 Million sequences Fig. Example of a Dag representing the workflow. ATGCATGCA 1000 sequences ATGCATGCA

9. Resource Independent GADU. Representing a Site and the applications on it.. #SITE Transformation PFNTYPE ANL_Jazz BLAST /soft/apps/BLAST/bin/blastallnull ANL_Jazz Blocks /soft/apps/run-Blocks.plnull ANL_Jazz Chisel /soft/apps/chisel/runChisel.plnull ANL_Jazz IPRSCAN /soft/apps/iprscan_wrapper.pl null ANL_Jazz globus-url-copy /soft/apps/packages/globus-2.2.4/bin/globus-url-copy GLOBUS_LOCATION=/soft/apps/packages/globus-2.2.4/; LD_LIBRARY_PATH=/soft/apps/packages/globus-2.2.4/lib; PATH=/soft/apps/packages/globus-2.2.4/bin pool ANL_Jazz { lrc "rls://gnare.mcs.anl.gov“ gridftp "gsiftp:// jmayor1.lcrc.anl.gov:2812/soft/apps/gadu" gridlaunch "/soft/apps/gadu/bin/kickstart" workdir "/soft/apps/gadu/vdldata" universe vanilla "jmayor1.lcrc.anl.gov:2121/jobmanager-pbs" universe globus "jmayor1.lcrc.anl.gov:2121/jobmanager-pbs" universe transfer " jmayor1.lcrc.anl.gov:2812/jobmanager-fork" } …. pool.config tc.data

10. Resource Independent GADU. GADU uses Pegasus based VDS and Condor-G Globus GRAM Interface Pegasus DAGMan Condor-G tc.data Pool.config Abstract Workflow as VDL Condor Submit files Submit Host WN Job management system Gatekeeper JobManager WN Remote Resources WN Job management system Gatekeeper JobManager WN Job management system Gatekeeper JobManager WN Information Services GADU’s automated Analysis Server, expressing, executing and tracking the scientific workflows on Grid. Database Controller Query Interface

11. Requirements... Information Services. VDS like System can to provide an Architecture independent mechanism to use different sites (Grids) Information Services at various levels Authentication – To check if the certs are valid at this site. Architecture – Is it an ia-32 cluster or an ia-64 ? Gatekeeper, GridFtp Server. Environment Variables – $OSG_APP, $TG_COMMUNITY Number of CPUs Number of Used CPUs. Number of Idle CPUs. VO (user) specific jobs running at a given site. VO (user) specific jobs sitting in QUEUE at a given site (why?) We a need standards and protocols for these Information Services and identify more information variables that needs to published by the Grids. Gridcat or MDS or something else. Currently GADU uses GridCat to collect site specific information for OSG and manually adds information for TeraGrid and Jazz. We are working on an MDS based information interface on TeraGrid. In order to automatically add a new Grid site, we need information about the site:

12. Another Big Challenge.. Site Selection. GADU has access to 60 OSG Sites and 5 TeraGrid Sites. One challenge in using the Grid reliably for high-throughput analysis is monitoring the state of all Grid sites and how well they have performed for job requests from a given submit host. We view a site as “available” if our submit host can communicate with it, if it is responding to Globus job-submission commands, and if it will run our jobs promptly, with minimal queuing delays GADU Server OSG

13. Another Big Challenge.. Site Selection. GADU has access to 60 OSG Sites and 5 TeraGrid Sites. Web Interface to Control the Selection of Sites for GADU: http://compbio.mcs.anl.gov/sulakhe/cgi-bin/site_selection_new.pl?user=dina Web Interface showing live status of usage: http://compbio.mcs.anl.gov/gaduvo/gadu_jobs.cgi Grid may not worry about this…

14. Next Steps.. Working with Teragrid Information Services group – MDS based interface. Continue to improve GADU’s implementation of Site Selection. Trying to generalize Site Selection using the Information Services such as MDS and Gridcat. Continue to deploy faster scientific applications for the Bioinformatics Group at Argonne.

15. Bioinformatics Group: Natalia Maltsev, PI Alex Rodriguez Elizabeth Glass Mark D’ Souza Mustafa Syed Yi Zhang Globus and VDS Mike Wilde Nika Nefedova Jens Voeckler Ian Foster Rick Stevens VDT Support. Condor Support. Systems at MCS. Acknowledgements Open Science Grid Thanks to Ruth Pordes and OSG team for their wonderful support TeraGrid Charlie Catlett Special thanks to David O’Neal, Joeseph Insley, and Sergiu Sanielevici