NMI Testbed GRID Utility for Virtual Organization Art Vandenberg Director, Advanced Campus Services Georgia State University 19 November 2018

NMI Testbed GRID – “virtual organization” Participating sites Overview NMI Testbed GRID – “virtual organization” Participating sites Resources for VO Catalog of grid applications Example: genome alignment for VO Plans for May-August 2004 11/19/2018

Vision – NMI Testbed GRID VO NMI Integration Testbed Program NSF #ANI-0123937 Explore grid capability – interoperability Researchers & faculty Across heterogeneous sites Integrated with enterprise middleware A utility grid using NMI components Non-specialized, open, transparent An inter-organizational model immediately calls for focus on interoperability - standards. Consider certificate structures and policies: An application solution may work wonderfully well for that application - even when researchers are drawn from multiple institutions. And one could say, “See truly inter-organizational.” But campus enterprise infrastructures would say “It’s inter-organizational when the solution can be applied equally to other applications.” And we have a way to go to be able to claim that model (such as envisioned by OGSA web services model.) 11/19/2018

Collaborative environment VO Beyond application specific grids Leverage enterprise middleware Identity management, authN, authZ... Strive for transparent access Portals Ease of use: submit, monitor, retrieve data Security policy & technology Federation of cooperating sites 11/19/2018

Your Grid is here, now 19 November 2018 In the world of Middleware, this diagram has grown over the last 4+ years. From a Network/Middleware/Application 3-level diagram to the 3-dimensional complexity shown. GRIDs is a part of the picture… 19 November 2018

We want VO utility grid to be here... 19 November 2018 We need to think of GRIDs in the context of the whole picture. Not just a part. 19 November 2018

Participating sites - the VO Testbed sites – push interoperation limits Georgia State University Texas Advanced Computing Center University of Alabama at Birmingham University of Alabama at Huntsville University of Michigan University of Southern California University of Virginia 11/19/2018

Testbed sites – interoperation challenges Site resources – VO Testbed sites – interoperation challenges GSU: Shibboleth, GridPort portal, REU & Grads, disk TACC: REU student, portal, Enterprise CA, cluster UAB: beowulf cluster, CA, Pubcookie, OGCE portal UAH: application expertise, NASA IPG Certs UMich: KX.509 & Kerberos, MGrid, ATLAS integration USC: CA, Pubcookie, Shibboleth, Linux cluster, KX.509 UVa: Bridge CA model Sites non-homogeneous – a VO challenge 11/19/2018

Catalog of grid applications Knowledge base is important REU students – Nicole Geiger, Anish Shindore Graduate Research Asst – Manish Garg NMI Testbed Sites initially Researchers, schools, projects Grid specific as well as grid potential Started as spreadsheet, now online db One activity that we think is useful is a catalog of GRID applications or potential applications. Note that Globus is collecting such info on GLOBUS TOOLKIT TM deployments. We’d like to do the same at campus level and maybe for an earlier stage of the deployment - what are applications that may be potentially grid-enabled? 11/19/2018

Catalog of grid applications Catalog of GRID Applications (current version) http://art12.gsu.edu:8080/grid_cat/index5.jsp Expanding scope beyond testbed sites 18 schools/labs, 300 researchers & counting Differentiated from Globus http://www.gpds.org Oriented to researcher, institutional level Planning clustering, visualization modality 11/19/2018

Example: Genome Alignment for VO (GSU – UAB) An opportunity for utility Grid VO Nova Ahmed, CS grad with Dr. Yi Pan, GSU dynamic programming algorithm for genome sequence alignment Initial runs on GSU shared memory hydra Limited access (grad student, shared cycles) Algorithm improvement using multi-processor cluster across a grid? 11/19/2018

The Genome Alignment Problem Alignment of DNA sequences Sequence X: TGATGGAGGT Sequence Y: GATAGG Count the matching score as 1 => matching 0 => non-matching Populate the Similarity matrix using: Observation re Similarity Matrix: Many zero values Reduction of memory possible by reducing zero value elements 11/19/2018

Improved Parallel Algorithm for Genome Alignment The parallel Method: Similarity matrix is divided among processors Processors calculate in parallel to match the partial sequence Communication is done among the processors to match the whole sequence The new Data Structure: New algorithm calculates only non-zero values of the similarity matrix Memory is dynamically allocated as needed 11/19/2018

Results on the Shared Memory Machine (Hydra) Limitations Can not allocate memory for long sequences Ex: Largest sequence to align is 2000 x 2000 Number of processors are limited Ex: For Hydra 12 processors Not scalable Performance Computation time decreases with increased number of processors 11/19/2018

Results on the Beowulf Cluster of UAB Using the beowulf cluster: Longer genome sequence can be aligned Highest sequence length can be 10,000 in the cluster Limited scalability Can increase the number of processors up to a certain limit 11/19/2018

Results via the GRID at UAB Submitting genome alignment program using Globus and MPICH Advantages: Scalable – Can add new clusters to the grid Easier job submission – Don’t need account on every node Scheduling is easier – Can submit multiple jobs at one time 11/19/2018

Future Work Genome Alignment Use MPICH-G2 (instead of MPICH) – Use the power of Grid Expand the computational resources – Combine more clusters across the Grid Develop program to align Multiple Genome Sequences (rather than two at a time) – Requiring more computation resources Use Georgia State certificate via Bridge CA Via Shibboleth protected sector CA…? 11/19/2018

Portal for NMI Testbed GRID Integration of campus authN Plans for May-August 2004 More resources Contributed from current sites (others?) Portal for NMI Testbed GRID Cf. NPACI Hotpage https://hotpage.npaci.edu/ Integration of campus authN UVa Bridge CA More applications Utility grid for grad research & education 11/19/2018

Catalog of Grid Applications Plans for May-August 2004… Documentation Web site Application docs and demos Catalog of Grid Applications Provide for self service contribution Develop clustering (SOM), visualization options (“find researchers or projects like X”) Auto-discovery of Grid researchers & apps based on reference sets (core sites)? 11/19/2018

Contact Information Art Vandenberg Avandenberg@gsu.edu NMI Testbed GRID http://www.gsu.edu/~wwwacs/GRID_Group/NMI.html 11/19/2018