1. University at BuffaloThe State University of New York CCR Center for Computational Research Grid Computing Overview Coordinate Computing Resources, People, Instruments in Dynamic Geographically-Distributed Multi-Institutional Environment Treat Computing Resources like Commodities  Compute cycles, data storage, instruments  Human communication environments No Central Control; No Trust Imaging Instruments Computational Resources Large-Scale Databases Data Acquisition Analysis Advanced Visualization Thanks to Mark Ellisman

2. University at BuffaloThe State University of New York CCR Center for Computational Research Factors Enabling the Grid Internet is Infrastructure  Increased network bandwidth and advanced services Advances in Storage Capacity  Terabyte costs less than $5,000 Internet-Aware Instruments Increased Availability of Compute Resources  Clusters, supercomputers, storage, visualization devices Advances in Application Concepts  Computational science: simulation and modeling  Collaborative environments  large and varied teams Grids Today  Moving towards production; Focus on middleware

3. University at BuffaloThe State University of New York CCR Center for Computational Research Computational Grids & Electric Power Grids Similarities/Goals of CG and EPG  Ubiquitous  Consumer is comfortable with lack of knowledge of details Differences Between CG and EPG  Wider spectrum of performance & services  Access governed by more complicated issues  Security  Performance  Socio-political factors

4. University at BuffaloThe State University of New York CCR Center for Computational Research Growth of Data and Load vs. Moore’s Law 199020002010 ESTs Combinatorial Chemistry Human Genome Pharmacogenomics Metabolic Pathways Computational Load Genome Data Moore’s Law Courtesy of Rick Stevens

5. University at BuffaloThe State University of New York CCR Center for Computational Research A Short History of the Grid Grand Challenge Problems (1980s)  NSF and DOE initiatives  “Science is a team sport”  Initiate multi-resource projects involving computation, instruments, visualization, data Evolution of Related Communities  Parallel computation  Address resource limitations  Networking  Gigabit testbed program Investigate potential testbed network architectures Explore usefulness for end-users CASA Gigabit Testbed (1990s)

6. University at BuffaloThe State University of New York CCR Center for Computational Research Globus model focuses on providing key Grid services  Resource access and management  Grid FTP  Information Service  Security services  Authentication  Authorization  Policy  Delegation  Network reservation, monitoring, control Applications Core Services Metacomputing Directory Service GRAM Globus Security Interface Heartbeat Monitor Nexus Gloperf Local Services LSF CondorMPI NQEEasy TCP Solaris IrixAIX UDP High-level Services and Tools DUROCglobusrunMPINimrod/GMPI-IOCC++ GlobusViewTestbed Status GASS The Grid as a Layered Set of Services The Globus Project (Ian Foster and Carl Kesselman)

7. University at BuffaloThe State University of New York CCR Center for Computational Research NSF Extensible TeraGrid Facility NCSA: Compute IntensiveSDSC: Data IntensivePSC: Compute Intensive IA64 Pwr4 EV68 IA32 EV7 IA64 Sun 10 TF IA-64 128 large memory nodes 230 TB Disk Storage GPFS and data mining 4 TF IA-64 DB2, Oracle Servers 500 TB Disk Storage 6 PB Tape Storage 1.1 TF Power4 6 TF EV68 71 TB Storage 0.3 TF EV7 shared-memory 150 TB Storage Server 1.25 TF IA-64 96 Viz nodes 20 TB Storage 0.4 TF IA-64 IA32 Datawulf 80 TB Storage Extensible Backplane Network LA Hub Chicago Hub IA32 Storage Server Disk Storage Cluster Shared Memory Visualization Cluster LEGEND 30 Gb/s IA64 30 Gb/s Sun ANL: VisualizationCaltech: Data collection analysis 40 Gb/s Backplane Router Figure courtesy of Rob Pennington, NCSA

8. University at BuffaloThe State University of New York CCR Center for Computational Research Critical Resources: WNY Computational & Data Grids Computational & Data Resources (CCR)  10TF Computing & 78TB Storage Instruments (HWI, RPCI)  Microarray; Diffractometer; NMR  High-Throughput Crystallization Laboratory Data Generation (HWI)  7TB per year Databases (UB-N, UB-S, BGH, CoE)  SnB; Multiple Sclerosis; Protein/Genomic

9. University at BuffaloThe State University of New York CCR Center for Computational Research BCOEB Medical/Dental Network Connections

10. University at BuffaloThe State University of New York CCR Center for Computational Research Medical/Dental BCOEB Network Connections (New)

11. University at BuffaloThe State University of New York CCR Center for Computational Research Advanced CCR Data Center (ACDC) Computational Grid Overview 300 Dual Processor 2.4 GHz Intel Xeon RedHat Linux 7.3 38.7 TB Scratch Space Joplin: Compute Cluster 75 Dual Processor 1 GHz Pentium III RedHat Linux 7.3 1.8 TB Scratch Space Nash: Compute Cluster 9 Single Processor Dell P4 Desktops School of Dental Medicine 13 Various SGI IRIX Processors Hauptman-Woodward Institute 25 Single Processor Sun Ultra5s Computer Science & EngineeringCrosby: Compute Cluster SGI Origin 3800 64 - 400 MHz IP35 IRIX 6.5.14m 360 GB Scratch Space 9 Dual Processor 1 GHz Pentium III RedHat Linux 7.3 315 GB Scratch Space Mama: Compute Cluster 16 Dual Sun Blades 47 Sun Ultra5 Solaris 8 770 GB Scratch Space Young: Compute Cluster T1 Connection Note: Network connections are 100 Mbps unless otherwise noted. 19 IRIX, RedHat, & WINNT Processors CCR RedHat, IRIX, Solaris, WINNT, etc Expanding ACDC: Grid Portal 4 Processor Dell 6650 1.6 GHz Intel Xeon RedHat Linux 9.0 66 GB Scratch Space 1 Dual Processor 250 MHz IP30 IRIX 6.5 Fogerty: Condor Flock Master

12. University at BuffaloThe State University of New York CCR Center for Computational Research ACDC Data Grid Overview 300 Dual Processor 2.4 GHz Intel Xeon RedHat Linux 7.3 38.7 TB Scratch Space Joplin: Compute Cluster 75 Dual Processor 1 GHz Pentium III RedHat Linux 7.3 1.8 TB Scratch Space Nash: Compute Cluster 4 Processor Dell 6650 1.6 GHz Intel Xeon RedHat Linux 9.0 66 GB Scratch Space ACDC: Grid PortalCrosby: Compute Cluster SGI Origin 3800 64 - 400 MHz IP35 IRIX 6.5.14m 360 GB Scratch Space 9 Dual Processor 1 GHz Pentium III RedHat Linux 7.3 315 GB Scratch Space Mama: Compute Cluster 16 Dual Sun Blades 47 Sun Ultra5 Solaris 8 770 GB Scratch Space Young: Compute Cluster Note: Network connections are 100 Mbps unless otherwise noted. 182 GB Storage 100 GB Storage 56 GB Storage 100 GB Storage 70 GB Storage Network Attached Storage 480 GB Storage Area Network 75 TB 136 GB Storage CSE Multi-Store 2 TB

13. University at BuffaloThe State University of New York CCR Center for Computational Research WNY Grid Highlights Heterogeneous Computational & Data Grid Currently in Beta with Shake-and-Bake WNY Release in March Bottom-Up General Purpose Implemenation  Ease-of-Use User Tools  Administrative Tools Back-End Intelligence  Backfill Operations  Prediction and Analysis of Resources to Run Jobs (Compute Nodes + Requisite Data)

14. University at BuffaloThe State University of New York CCR Center for Computational Research Advanced CCR Data Center (ACDC) Computational Grid Overview 300 Dual Processor 2.4 GHz Intel Xeon RedHat Linux 7.3 38.7 TB Scratch Space Joplin: Compute Cluster 75 Dual Processor 1 GHz Pentium III RedHat Linux 7.3 1.8 TB Scratch Space Nash: Compute Cluster 9 Single Processor Dell P4 Desktops School of Dental Medicine 13 Various SGI IRIX Processors Hauptman-Woodward Institute 25 Single Processor Sun Ultra5s Computer Science & EngineeringCrosby: Compute Cluster SGI Origin 3800 64 - 400 MHz IP35 IRIX 6.5.14m 360 GB Scratch Space 9 Dual Processor 1 GHz Pentium III RedHat Linux 7.3 315 GB Scratch Space Mama: Compute Cluster 16 Dual Sun Blades 47 Sun Ultra5 Solaris 8 770 GB Scratch Space Young: Compute Cluster T1 Connection Note: Network connections are 100 Mbps unless otherwise noted. 19 IRIX, RedHat, & WINNT Processors CCR RedHat, IRIX, Solaris, WINNT, etc Expanding ACDC: Grid Portal 4 Processor Dell 6650 1.6 GHz Intel Xeon RedHat Linux 9.0 66 GB Scratch Space 1 Dual Processor 250 MHz IP30 IRIX 6.5 Fogerty: Condor Flock Master

15. University at BuffaloThe State University of New York CCR Center for Computational Research Data Grid Motivation& Goal Motivation:  Large data collections are emerging as important community resources.  Data Grids inherently complements Computational Grids, which manipulate data.  A data grid denotes a large network of distributed storage resources such as archival systems, caches, and databases, which are linked logically to create a sense of global persistence. Goal:  To design and implement transparent management of data distributed across heterogeneous resources, such that the data is accessible via a uniform web interface.

16. University at BuffaloThe State University of New York CCR Center for Computational Research Data Grid Summary 544 GB Storage  Located on 6 heterogeneous ACDC-Grid resources 480 GB Storage  Located on 1 dual processor Dell PowerVault server 75,000 GB Storage (10/03)  Served by 4 – 16 processor HP GS1280 servers 2,000 GB Storage  Served by Sun Ultra-60 servers 78,024 GB Total Data Grid Storage available and accessible from the ACDC-Grid Portal 182 GB Storage 100 GB Storage 56 GB Storage 100 GB Storage 70 GB Storage 136 GB Storage Storage Area Network 75 TB Network Attached Storage 480 GB CSE Multi-Store 2 TB

17. University at BuffaloThe State University of New York CCR Center for Computational Research Grid-Based SnB Objectives Install Grid-Enabled Version of SnB Job Submission and Monitoring over Internet SnB Output Stored in Database SnB Output Mined through Internet-Based Integrated Querying Tool Serve as Template for Chem-Grid & Bio-Grid Experience with Globus and Related Tools

18. University at BuffaloThe State University of New York CCR Center for Computational Research Grid Enabled SnB Problem Statement  Use all available resources in the ACDC-Grid for determining a single molecular structure. Grid Enabling Criteria  All heterogeneous resources in the ACDC-Grid are capable of executing the SnB application.  All job results obtained from the ACDC-Grid resources are stored in a corresponding molecular structure database.  There are three modes of operation:  Continue submitting SnB application jobs until the grid-enabled SnB application determines a solution has been found, or “X” number of trials have been evaluated, or indefinitely (grid job owner determines when a solution has been found).

19. University at BuffaloThe State University of New York CCR Center for Computational Research Grid Services and Applications ACDC-Grid Computational Resources ACDC-Grid Data Resources ACDC-Grid Data Resources Applications Local Services LSF Condor MPI TCP SolarisIrix WINNT UDP High-level Services and Tools Globus Toolkit globusrun MPI NWS MPI-IO Core Services Metacomputing Directory Service GRAM Globus Security Interface GASS C, C++, Fortran, PHP Shake-and-Bake Oracle MySQL Apache PBS Maui Scheduler RedHat Linux Stork Adapted from Ian Foster and Carl Kesselman

20. University at BuffaloThe State University of New York CCR Center for Computational Research Notes Apache – web portal server PHP - used by apache server for dynamic web portal pages MDS – traditional to use MDS with LDAP but we use MDS with MYSql grid portal database to keep information of available resources (we poll every 15 mins) GRAM – Globus Resource Allocation Manager – API for requesting comptuational jobs GASS – Global Access to Secondary Storage – API for accessing files stored on various platforms Stork – Condor module for transporting job files within a flock

21. University at BuffaloThe State University of New York CCR Center for Computational Research Grid Enabled SnB Required Layered Grid Services  Grid-enabled Application Layer  Shake – and – Bake application  Apache web server  MySQL database  High-level Service Layer  Globus, NWS, PHP, Fortran, and C  Core Service Layer  Metacomputing Directory Service, Globus Security Interface, GRAM, GASS  Local Service Layer  Condor, MPI, PBS, Maui, WINNT, IRIX, Solaris, RedHat Linux

22. University at BuffaloThe State University of New York CCR Center for Computational Research Required Grid Services Applications Core Services Metacomputing Directory Service GRAM Globus Security Interface Heartbeat Monitor Nexus Gloperf Local Services LSF CondorMPI NQEEasy TCP Solaris IrixAIX UDP High-level Services and Tools DUROCglobusrunMPINimrod/GMPI-IOCC++ GlobusViewTestbed Status GASS Application Layer  Shake-and-Bake  Apache web server  MySQL database High-level Services  Globus, PHP, Fortran, C Core Services  Metacomputing Directory Service, Globus Security Interface, GRAM, GASS Local Services  Condor, MPI, PBS, Maui, WINNT, IRIX, Solaris, RedHat Linux Grid Implementation as a Layered Set of Services

23. University at BuffaloThe State University of New York CCR Center for Computational Research Grid Enabled SnB Execution  User  defines Grid-enabled SnB job using Grid Portal or SnB  supplies location of data files from Data Grid  supplies SnB mode of operation  Grid Portal  assembles required SnB data and supporting files, execution scripts, database tables.  determines available ACDC-Grid resources.  ACDC-Grid job management includes:  automatic determination of appropriate execution times, number of trials, and number/location of processors,  logging/status of concurrently executing resource jobs, &  automatic incorporation of SnB trial results into the molecular structure database.

24. University at BuffaloThe State University of New York CCR Center for Computational Research ACDC-Grid Portal

25. University at BuffaloThe State University of New York CCR Center for Computational Research ACDC-Grid Portal Login Grid Portal login screen

26. University at BuffaloThe State University of New York CCR Center for Computational Research Data Grid Capabilities Browser view of “mlgreen” user files stored in the Data Grid

27. University at BuffaloThe State University of New York CCR Center for Computational Research Data Grid Capabilities Browser view of “miller” group files published by user “rappleye”

28. University at BuffaloThe State University of New York CCR Center for Computational Research Data Grid Capabilities Browser view of “public” user files published by user “miller”

29. University at BuffaloThe State University of New York CCR Center for Computational Research Data Grid Capabilities

30. University at BuffaloThe State University of New York CCR Center for Computational Research Data Grid Capabilities

31. University at BuffaloThe State University of New York CCR Center for Computational Research Grid Portal Job Status Grid-enabled jobs can be monitored using the Grid Portal web interface dynamically.  Charts are based on:  total CPU hours, or  total jobs, or  total runtime.  Usage data for:  running jobs, or  queued jobs.  Individual or all resources.  Grouped by:  group, or  user, or  queue.

32. University at BuffaloThe State University of New York CCR Center for Computational Research Grid Portal Job Status

33. University at BuffaloThe State University of New York CCR Center for Computational Research ACDC-Grid Portal Condor Flock CondorView integrated into ACDC-Grid Portal

34. University at BuffaloThe State University of New York CCR Center for Computational Research ACDC-Grid Portal User Management user based Administrator based

35. University at BuffaloThe State University of New York CCR Center for Computational Research ACDC-Grid Portal Resource Management Administrator grants a user access to ACDC-Grid  resources,  software, and  web pages.

36. University at BuffaloThe State University of New York CCR Center for Computational Research ACDC-Grid Administration

37. University at BuffaloThe State University of New York CCR Center for Computational Research ACDC-Grid Administration

38. University at BuffaloThe State University of New York CCR Center for Computational Research Grid Enabled Data Mining Problem Statement  Use all available resources in the ACDC-Grid for executing a data mining genetic algorithm optimization of SnB parameters for molecular structures having the same space group. Grid Enabling Criteria  All heterogeneous resources in the ACDC-Grid are capable of executing the SnB application.  All job results obtained from the ACDC-Grid resources are stored in a corresponding molecular structure databases.

39. University at BuffaloThe State University of New York CCR Center for Computational Research Grid Enabled Data Mining  There are two modes of operation and two sets of stopping criteria:  Data mining jobs can be submitted in a dedicated mode (time critical), where jobs are queued on ACDC-Grid resources, or in a back fill mode (non-time critical), where jobs are submitted to ACDC-Grid resource that have unused cycles available.  There are two sets of stopping criteria:  Continue submitting SnB data mining application jobs until the grid-enabled SnB application determines optimal parameters have been found, or indefinitely (grid job owner determines when optimal parameters have been found).

40. University at BuffaloThe State University of New York CCR Center for Computational Research Grid Enabled Data Mining Grid Portal Workflow Job Manager ACDC-Grid Computational Resources Molecular Structure Database Data Mining Criteria ACDC-Grid Data Grid ACDC-Grid Data Grid

41. University at BuffaloThe State University of New York CCR Center for Computational Research SnB Molecular Structure Database Molecular Structure Database

42. University at BuffaloThe State University of New York CCR Center for Computational Research Grid Enabled Data Mining Execution Scenario  User defines a Grid-enabled data mining SnB job using the Grid Portal web interface supplying:  designate which molecular structures parameter sets to optimize,  data file metadata, and  Grid-enabled SnB mode of operation dedicated or back fill mode, and  Grid-enabled SnB stopping criteria.  The Grid Portal assembles the required SnB application data and supporting files, execution scripts, database tables, and submits jobs for parameter optimization based on the current database statistics.  ACDC-Grid job management includes:  automatic determination of appropriate execution times, number of trials, and number of processors for each available resource,  logging and status of all concurrently executing resource jobs,  automatic incorporation of SnB trial results into the molecular structure database, and  post processing of updated database for subsequent job submissions.

43. University at BuffaloThe State University of New York CCR Center for Computational Research ACDC Data Grid Database Schema ACDC-Grid Data Grid ACDC-Grid Data Grid

44. University at BuffaloThe State University of New York CCR Center for Computational Research Grid Portal Job Status ACDC-Grid Computational Resources

45. University at BuffaloThe State University of New York CCR Center for Computational Research Data Grid Overview Enable the transparent migration of data between various resources while preserving uniform access for the user.  Maintain metadata information about each file and its location in a global database table.  Currently using MySQL tables.  Periodically migrate files between machines for more optimal usage of resources.

46. University at BuffaloThe State University of New York CCR Center for Computational Research Data Grid Functionality Implement basic file management functions accessible via a platform-independent web interface. Features include:  User-friendly menus/ interface.  File Upload/ Download to and from the Data Grid Portal.  Simple web-based file editor.  Efficient search utility.  Logical display of files for a given user in three divisions (user/ group/ public).  Hierarchical vs. List-based  3 divisions: (user/ group/ public)  Sorting capability based on file metadata, i.e. filename, size, modification time, etc.

47. University at BuffaloThe State University of New York CCR Center for Computational Research Data Grid Functionality Support multiple access to files in the data grid.  Implement basic Locking and Synchronization primitives for version control. Integrate security into the data grid.  Implement basic authentication and authorization of users.  Decide and enforce policies for data access and publishing.

48. University at BuffaloThe State University of New York CCR Center for Computational Research Data Grid File Migration Migration Algorithm  File migration depends upon a number of factors:  User access time  Network capacity at time of migration  User profile  User disk quotas on various resources

49. University at BuffaloThe State University of New York CCR Center for Computational Research Data Grid File Migration We need to mine log files in order to determine  How much data to migrate in one migration cycle?  What is an appropriate migration cycle length?  What is a user’s access pattern for files?  What is the overall access pattern for particular files?

50. University at BuffaloThe State University of New York CCR Center for Computational Research Data Grid File Aging Global File Aging vs. Local File Aging  User aging attribute  Indicative of a user’s access across their own files.  Attribute of a user’s profile.  During migration time, this attribute will determine which user’s files should be migrated off of the grid portal onto a remote resource.  Function of (file age, global file aging, resource usage)

51. University at BuffaloThe State University of New York CCR Center for Computational Research Data Grid File Aging File aging attribute  Indicative of overall access to/migration activity of a particular file.  Attribute in file_management table.  Scale: 0 to 1 probability of whether or not to migrate file.  File_aging_local_param initialized to 1.  During migration time after a user has been chosen, this attribute will help determine which files of the user to migrate.  i.e. Migrate a maximum of the top 5% of user’s files in any one cycle.

52. University at BuffaloThe State University of New York CCR Center for Computational Research Data Grid File Aging For a given user, the average of the file_aging_local_param attributes of all files should be close to 1.  Operating tolerance before action is taken is within the range of 0.9 – 1.1. In this way, the user file_aging_global_param can be a function of this average.  If the average file_aging_local_param attribute > 1.1, then files of the user are being held to long before being migrated.  The file_aging_global_param value should be decreased.  If the average file_aging_local_param attribute < 0.9, then files of the user are being accessed at a higher frequency than the file_aging_global_param value.  The file_aging_global_param value should be increased.

53. University at BuffaloThe State University of New York CCR Center for Computational Research Data Grid Resource Info

54. University at BuffaloThe State University of New York CCR Center for Computational Research Data Grid Resource Info Both platforms have reduced bandwidth available for additional transfers

55. University at BuffaloThe State University of New York CCR Center for Computational Research Date Grid File Management Table

56. University at BuffaloThe State University of New York CCR Center for Computational Research Data Grid File Age File age, access time, and resource id denote:  the amount of time since a file was accessed,  when the file was accessed, and  where the file currently resides respectively.

57. University at BuffaloThe State University of New York CCR Center for Computational Research Data Grid Summary The Data Grid algorithms are continually evolving to minimize network traffic and maximize disk space utilization on a per user basis by data mining user usage and disk space requirements. 182 GB Storage 100 GB Storage 56 GB Storage 100 GB Storage 70 GB Storage 136 GB Storage Storage Area Network 75 TB Network Attached Storage 480 GB CSE Multi-store 2 TB

58. University at BuffaloThe State University of New York CCR Center for Computational Research ACDC-Grid Development/Maintenance Development Requirements  7 – Person months for Grid Services Coordinator  Including Grid and Database conceptual design and implementation  5 – Person months for Grid Services Programmer  Web portal programming  5 – Person months for System Administrator  Globus, NWS, MDS, etc. installations  3 – Person months for Database Administrator  Grid Portal Database implementation Minimum Maintenance Requirements  1 – Grid Services Coordinator  100% level of effort  1 – Grid Services Programmer  100% level of effort  1 – System Administrator  50% level of effort  1 – Database Administrator  10% level of effort

59. University at BuffaloThe State University of New York CCR Center for Computational Research Future ACDC Applications Princeton Ocean Model (POM) Genetic Algorithms for Earthquake Structural Design Bioinformatics Computational Chemistry (Q-Chem) Environmental Engineering Applications