Grid Andrew Grimshaw September, 2005

What is a Grid System? A Grid system is a collection of distributed resources connected by a network. Examples of Distributed Resources: Desktop Handheld hosts Devices with embedded processing resources such as digital cameras and phones Tera-scale supercomputers

A grid enables users to collaborate securely by sharing processing, applications, work flows and processes, and data across heterogeneous systems and administrative domains for collaboration, faster application execution, and easier access to data. What is a Grid? A grid is all about gathering together resources (CPU, data, policy, devices, …) and making them accessible to users and applications.

Grid System Characteristics of Grid systems Numerous Resources Ownership by Mutually Distrustful Organizations & Individuals Faulty Resources Different Security Requirements & Policies Required Resources are Heterogeneous Geographically Separated Different Resource Management Policies Connected by Heterogeneous, Multi-Level Networks

Characteristics of a Grid system Numerous Resources Ownership by Mutually Distrustful Organizations & Individuals Faulty Resources Different Security Requirements & Policies Required Resources are Heterogeneous Geographically Separated Different Resource Management Policies Connected by Heterogeneous, Multi-Level Networks

Requirements for robust, enterprise solutions Interoperable implementations Standards-based Simple Secure Scalable Extensible Site Autonomy Multi-Language Legacy Support Transparency in multiple dimensions Naming schemes critical Fault-tolerance & Exception Management Modular and composable Success Requires an integrated model at the foundation. OGSA Complexity Management!!

Users view Site 0Site 1Site 2Site 3 SP2 HPSS Users Grid Run programs Access Data Collaborate Provide shared services

Evolution in action Bare Metal Programming 50’s Batch OS Multi-User Timeshare 60’s to 80’s Low Level Network Programming Today Grid & WS Now & Future!

Grid Computing Scenarios Desktop Cycle Aggregation Limited acceptance in commercial enterprises Cluster Grids Single owner, department, project Single domain, file system LAN connection Campus/Enterprise Grids Multiple owners, domains Multiple file systems WAN connection Partner Grids Multiple owners, sites, domains Multiple file systems Internet connectivity Legion Grid Software – Compute and Data Grid

Grid “component” models Most grids today have a service oriented architecture where (grid) resources are endpoints with An interface – usually WSDL Some meta-data – usually resource properties A base set of functions that can be assumed, e.g., lifetimes, notification, etc. Some form of factory model Most grids today have Some form of information discovery model Some means of moving data around implicitly or explicitly Some means establishing identity and mapping between identity spaces Can operate between administrative domains Can operate through firewalls

Vendor neutral Grid buzz words GGF – Global Grid Forum OGSA – Open Grid Services Architecture EMS – Execution Management Services BES, RSS, Data ByteIO Naming RNS WS-Names (identities) Provisioning and deployment JSDL – Job Submission Description Language WS-Agreement WS-Addressing WSRF/WSN WSDM

Context Services Info Services Infra Services Security Services Rsrc Mgmt Services Execution Mgmt Services Data Services Policy Mgmt VO Mgmt Access Integration Transfer Replication Boundary Traversal Integrity Authorization Authentication WSRFWSNWSDM Event Mgmt MonitoringDiscovery Job Mgmt Logging Execution Planning Workflow Mgmt Workload Mgmt Provisioning Execution DeploymentConfigurationReservation Naming Self Mgmt Services Heterogeneity Mgmt Service Level Attainment QoS Mgmt Optimization Information Services Infrastructure Services Self Mgmt Services Security Services Resource Mgmt Services Execution Mgmt Services Data Services Context Services

UVA and Grid Pioneered grid/metasystems with Legion Integrated OGSA-like system Stress on ease of use Motto - “by default the use does not have to think” First operational in 1994 Grounds-wide virtual computer in 1996 NPACINet & DoD MSRC grid Dozens of applications

NPACINet NPACI - SDSC, UCSD, Caltech, UT, UCB, UM, UVa DoD MSRC’s - NAVO & ARL, NASA - Ames

The Global Bio Grid

Tomorrow $1000/sequnce for humans – becomes standard clinical practice Global interconnected networks – grids Provide transparent, secure, access to data, applications, and on-demand compute. Research using not just your data, but all trusted data, not just your applications, but any trusted application.

Tomorrow - Clinical Unified clinical records Data mining of integrated clinical and scientific databases

What this will facilitate Discovery of new diagnostics and predictors for prescriptive medicine Discovery of new treatments to complex genetic diseases Push back expected lifetimes and improve quality of life Deliver better health care by identifying problems earlier Wide spread epidemiological research

There are a number of “catches” So much data! So many organizations with so little trust! So much complexity!

This is where grids enter the picture

Federated access to multiple Data sources Public databases Commercial databases In-house databases, annotations, etc. Application suites (including processes and workflows) Compute resources Shared among collaborative research teams Multiple research locations Virtual organizations Heterogeneous compute environments OS, geography, administrative domains, policies Fine-grain modular security with work towards HIPAA and CFR 21 part 11 Built on evolving standards (GGF, WS-*) GBG concept

GBG Today Datagrid using Avaki DG technology UVA, NCBIO, U-Texas, Texas Tech Already operational Flat file and relational Replacing with open-source data grid Compute grid at UVA on-line IBM, 64 node, cluster as keystone Four clusters on-line 300+ lab PC’s running Windows XP