GridLab: A Grid Application Toolkit and Testbed

Sun HPC Consortium, Glasgow, July 2002 GridLab Project EU Funded (5+M€) Application and Testbed orientated Applications and grid application developers are in the center of the project

Sun HPC Consortium, Glasgow, July 2002 GridLab Members n PSNC (Poznan) - coordination n AEI (Potsdam) n ZIB (Berlin) n Univ. of Lecce n Cardiff University n Vrije Univ. (Amsterdam) n SZTAKI (Budapest) n Masaryk Univ. (Brno) n NTUA (Athens) Sun Compaq (HP) n ANL (Chicago, I. Foster) n ISI (LA, C.Kesselman) n UoWisconsin (M. Livny) Also collaborating with: n Users!!! (EU Astrophysics Network, High Performance Computing Group, Ottawa) n DFN TiKSL/GriKSL n NSF ASC Project n GrADS Project n GriPhyn/iVDGL

Sun HPC Consortium, Glasgow, July 2002 GridLab Aims Get Computational Scientists using the “Grid” and Grid services for real, everyday, production work (AEI Relativists, EU Network, Grav Wave Data Analysis, Cactus User Community). Make it easier for applications to make flexible, efficient, robust, use of the resources available to their virtual organizations. Dream up, prototype, and test new application scenarios which make adaptive, dynamic, wild, and futuristic uses of resources.

Sun HPC Consortium, Glasgow, July 2002 What GridLab Isn’t … Don’t want to develop low level Grid Infrastructure Don’t want to repeat work which has already been done (want to incorporate and assimilate it … Globus APIs, OGSA, ASC Portal (GridSphere/Orbiter), GPDK, GridPort, DataGrid, GriPhyn)

Sun HPC Consortium, Glasgow, July 2002 Solution “Grid Application Toolkit” or GAT Provides a layer between applications and emerging grid technologies. Provides an application developer orientated API, allowing the flexible use of different tools and services, as well as providing protection from developing software. “GridLab Testbed/VO” Diverse controllable environment for developing and testing applications and tools, software maintained by people who know it. End Users GAT Tool Developers Grid Infrastructure Developers GAT-API Developers

Sun HPC Consortium, Glasgow, July 2002 General Architecture

Sun HPC Consortium, Glasgow, July 2002 What is the GAT ? Set of application developer APIs for Grid tools, services and software libraries, (and example implementations) that support the development of grid-enabled aplications (open source!) Usable from any high level “application” (any generic code, Cactus, Triana, Portals, Scripts, …) More or less … Set of calls GAT_ToolOrService(arguments) Your chosen tools/services: Resource broker, information server, application manager, grid monitoring, data manager, notification, etc, etc Set of APIs for dealing with the GAT (registration, information, errors, fault tolerance)

Sun HPC Consortium, Glasgow, July 2002 GAT Architecture Applications GAT-API GridLab Services Grid Core Services Cactus, Triana, Generic Codes, Portals, Scripts High level calls based on functionality GAT_FileCopy, GAT_ResourceFind Resource Broker, Monitoring System, Data Manager GRAM, GridFTP, GridSSH, MDS but not limited to Globus! (Triana and JXTA)

Sun HPC Consortium, Glasgow, July 2002 Proposed GAT Implementation Applications GAT Engine GAT Adaptors Grid Infrastructure (GAT/GridLab+base/core grid services) Cactus, Triana, Generic Codes, Portals, Scripts Layer providing access to services Resource brokers, basic grid services (GRAM, GSIFTP, MDS) Library containing registration routines and empty implementations GAT-API

Sun HPC Consortium, Glasgow, July 2002 GAT Example Machine A Gridftp Gridscp Machine B Gridscp “Application” (Cactus Migration Tool, User Portal, Grid Script) wants to move a file between two other machines How to move the file? Available software User authentification Disk properties (disksize, user quota, inodes) Network bandwidth

Sun HPC Consortium, Glasgow, July 2002 GAT Example GAT_FileCopy(GAT, “ThisMachine:ThisFile”, ”ThatMachine:ThatFile”) GAT_GFTPAdaptor IGAT_FileCopy IGAT_FileMove IGAT_MkDir GAT API call from Cactus Application, or Portal, or Python script GridFTP Adaptor says it knows how to both Copy and Move files between machines GridSSH Adaptor only knows how to Copy files GAT_GSCPAdaptor IGAT_FileCopy GAT Engine Engine decides (by different means) how to respect this request

Sun HPC Consortium, Glasgow, July 2002 Implementation GridLab is an early OGSA adopter, however We start with the web services implementation (all the GridLab services will be web services based and in the next step OGSA compliant) Several implementations of GAT will be developed

Sun HPC Consortium, Glasgow, July 2002 Workpackages WP1: Grid Application Toolkit (AEI) This is a key component of GridLab - link between Grid middleware and applications, usable by any conforming application or middleware component. Requiring input from, and connecting to, most other workpackages and components. WP2: Cactus Grid Application Toolkit (AEI) Provides an extended GAT interface for Cactus, a very general toolkit framework supporting different Grid applications, from astrophysics to chemical engineering. Cactus will be one of the primary application drivers for the GAT, and the project generally. WP3: Work-flow Application Toolkit (CARDIFF) Will develop Grid capabilities for a widely used dataflow programming environment, Triana, used in gravitational wave and other data analysis areas.

Sun HPC Consortium, Glasgow, July 2002 Workpackages (cont.) WP4: Grid Portals (AEI) Will be highly application driven, aimed at providing uniform, flexible and intuitive user access to Grid resources from anywhere, as well as administration tools for maintaining a Grid environment. WP5: Testbed management (MU) Will administrate and maintain an active development testbed across roughly a dozen EU sites (leveraging the work of the EGrid), deploying technologies as they are developed by the project. This workpackage will also coordinate with sites in the USA-based NCSA Alliance and others to test and develop interoperability. WP6: Security (PSNC) Will develop the required security mechanisms and will ensure the integration of all the technologies developed under other WPs, taking into account the various local security requirements and state of the art solutions.

Sun HPC Consortium, Glasgow, July 2002 Workpackages (cont.) WP7: Adaptive Application Components (VU) develops a set of components and APIs to be plugged into the toolkit, for example to take monitoring information and implement basic techniques for short-term forecasting and behavior adaptation/optimization. WP8: Data Handling and Visualization (ZIB) will provide Grid aware techniques for data management, analysis, and visualization, needed especially for applications that make use of multiple sites in a dynamic, time dependent manner, leaving data unpredictably scattered across the Grid. WP9: Resource Management (PSNC) will develop resource need estimators, resource brokers, and other tools, for both Grid users and the applications themselves to make intelligent decisions about which Grid resources should be used at any instant in the lifetime of a simulation.

Sun HPC Consortium, Glasgow, July 2002 Workpackages (cont.) WP10: Information Services (ISUFI) will extend existing Grid middleware toolkits with dynamic features needed by applications to select appropriate Grid resources and to provide simulation information to collaborative user groups. WP11: Monitoring (SZTAKI) will develop new components that will fit in the general Grid monitoring architecture to support application steering, adaptive monitoring, and automatic analysis and prediction of performance data. WP12: Access for mobile users (ZIB) will develop and test Grid access and monitoring technologies through a variety of mobile devices,

Sun HPC Consortium, Glasgow, July 2002 Workpackages (cont.) WP13: Information Dissemination and Exploitation (PSNC) will ensure the active dissemination of the project results through a variety of channels, including active participation in international organizations (e.g. GGF), co-development with other Grid projects in the USA and EU, participation in international conferences, training programs, instruction of GridLab technologies into various communities, and introduction into the commercial vendor world. WP14: Project Management (PSNC) day-to-day scientific, financial and administrative management of the project, including careful orchestration and monitoring of work across groups, major project decisions, liaisons with external projects and with the international advisory board, reporting

Sun HPC Consortium, Glasgow, July 2002 More info / summary You’re welcome to join our testbed We are open for collaborations!