GridLab: A Grid Application Toolkit and Testbed Jarosław Nabrzyski GridLab Project Manager Poznań Supercomputing and Networking Center, Poland

MANs Meeting, Warsaw, NASK, September 2002 GridLab Project EU Funded (5+M€) Application and Testbed orientated Applications and grid application developers are in the center of the project

MANs Meeting, Warsaw, NASK, September 2002 GridLab Members n PSNC (Poznań) - coordination n AEI (Potsdam) n ZIB (Berlin) n Univ. of Lecce n Cardiff University n Vrije Univ. (Amsterdam) n SZTAKI (Budapest) n Masaryk Univ. (Brno) n NTUA (Athens) Sun Compaq (HP) n ANL (Chicago, I. Foster) n ISI (LA, C.Kesselman) n UoWisconsin (M. Livny) Also collaborating with: n Users!!! (EU Astrophysics Network, High Performance Computing Group, Ottawa) n DFN TiKSL/GriKSL n NSF ASC Project n GrADS Project n GriPhyn/iVDGL

MANs Meeting, Warsaw, NASK, September 2002 GridLab on a map

MANs Meeting, Warsaw, NASK, September 2002 GridLab Aims Get Computational Scientists using the “Grid” and Grid services for real, everyday, production work (AEI Relativists, EU Network, Grav Wave Data Analysis, Cactus User Community). Make it easier for applications to make flexible, efficient, robust, use of the resources available to their virtual organizations. Dream up, prototype, and test new application scenarios which make adaptive, dynamic, wild, and futuristic uses of resources.

MANs Meeting, Warsaw, NASK, September 2002 What GridLab Isn’t … Don’t want to develop low level Grid Infrastructure Don’t want to repeat work which has already been done (want to incorporate and assimilate it … Globus APIs, OGSA, ASC Portal (GridSphere/Orbiter), GPDK, GridPort, DataGrid, GriPhyn)

MANs Meeting, Warsaw, NASK, September 2002 GridLab end user requirements Give us an application oriented environment, Applications running on resources of one or more virtual organisations, Flexible, easy-to-use, simple interfaces to resources, jobs, and data (including compiling, tracking jobs, cataloguing data) Give us means to make efficient and effective use of resources, Fault tolerance is a key requirement, The ability to work in a disconnected environment, Mobile working,

MANs Meeting, Warsaw, NASK, September 2002 GridLab end user requirements (cont) The ability to run in environments as minimalistic as laptops with no grid infrastructure to fully deployed Virtual Organisations Complexity should be hidden as much as possible, Provide a collaborative infrastructure, The infrastructure must cater for all classes of applications, from lightweight to largescale, The infrastructure must provide capabilities to customise choice of service implementation (e.g. using efficiency, reliability, first succeeding, all).

MANs Meeting, Warsaw, NASK, September 2002 Solution “Grid Application Toolkit” or GAT Provides a layer between applications and emerging grid technologies. Provides an application developer orientated API, allowing the flexible use of different tools and services, as well as providing protection from developing software. “GridLab Testbed/VO” Diverse controllable environment for developing and testing applications and tools, software maintained by people who know it. End Users GAT Tool Developers Grid Infrastructure Developers GAT-API Developers

MANs Meeting, Warsaw, NASK, September 2002 General Architecture

MANs Meeting, Warsaw, NASK, September 2002 What is the GAT ? Set of application developer APIs for Grid tools, services and software libraries, (and example implementations) that support the development of grid-enabled aplications (open source!) Usable from any high level “application” (any generic code, Cactus, Triana, Portals, Scripts, …) More or less … Set of calls GAT_ToolOrService(arguments) Your chosen tools/services: Resource broker, information server, application manager, grid monitoring, data manager, notification, etc, etc Set of APIs for dealing with the GAT (registration, information, errors, fault tolerance)

MANs Meeting, Warsaw, NASK, September 2002 GAT Architecture Applications GAT-API GridLab Services Grid Core Services Cactus, Triana, Generic Codes, Portals, Scripts High level calls based on functionality GAT_FileCopy, GAT_ResourceFind Resource Broker, Monitoring System, Data Manager GRAM, GridFTP, GridSSH, MDS but not limited to Globus! (Triana and JXTA)

MANs Meeting, Warsaw, NASK, September 2002 Proposed GAT Implementation Applications GAT Engine GAT Adaptors Grid Infrastructure (GAT/GridLab+base/core grid services) Cactus, Triana, Generic Codes, Portals, Scripts Layer providing access to services Resource brokers, basic grid services (GRAM, GSIFTP, MDS) Library containing registration routines and empty implementations GAT-API

MANs Meeting, Warsaw, NASK, September 2002 GAT Example Machine A Gridftp Gridscp Machine B Gridscp “Application” (Cactus Migration Tool, User Portal, Grid Script) wants to move a file between two other machines How to move the file? Available software User authentification Disk properties (disksize, user quota, inodes) Network bandwidth

MANs Meeting, Warsaw, NASK, September 2002 GAT Example GAT_FileCopy(GAT, “ThisMachine:ThisFile”, ”ThatMachine:ThatFile”) GAT_GFTPAdaptor IGAT_FileCopy IGAT_FileMove IGAT_MkDir GAT API call from Cactus Application, or Portal, or Python script GridFTP Adaptor says it knows how to both Copy and Move files between machines GridSSH Adaptor only knows how to Copy files GAT_GSCPAdaptor IGAT_FileCopy GAT Engine Engine decides (by different means) how to respect this request

MANs Meeting, Warsaw, NASK, September 2002 Implementation GridLab is an early OGSA adopter, however We start with the web services implementation (all the GridLab services will be web services based and in the next step OGSA compliant) Several implementations of GAT will be developed

MANs Meeting, Warsaw, NASK, September 2002 Workpackages WP1: Grid Application Toolkit (AEI) This is a key component of GridLab - link between Grid middleware and applications, usable by any conforming application or middleware component. Requiring input from, and connecting to, most other workpackages and components. WP2: Cactus Grid Application Toolkit (AEI) Provides an extended GAT interface for Cactus, a very general toolkit framework supporting different Grid applications, from astrophysics to chemical engineering. Cactus will be one of the primary application drivers for the GAT, and the project generally. WP3: Work-flow Application Toolkit (CARDIFF) Will develop Grid capabilities for a widely used dataflow programming environment, Triana, used in gravitational wave and other data analysis areas.

MANs Meeting, Warsaw, NASK, September 2002 Workpackages (cont.) WP4: Grid Portals (AEI) Will be highly application driven, aimed at providing uniform, flexible and intuitive user access to Grid resources from anywhere, as well as administration tools for maintaining a Grid environment. WP5: Testbed management (MU) Will administrate and maintain an active development testbed across roughly a dozen EU sites (leveraging the work of the EGrid), deploying technologies as they are developed by the project. This workpackage will also coordinate with sites in the USA-based NCSA Alliance and others to test and develop interoperability. WP6: Security (PSNC) Will develop the required security mechanisms and will ensure the integration of all the technologies developed under other WPs, taking into account the various local security requirements and state of the art solutions.

MANs Meeting, Warsaw, NASK, September 2002 Workpackages (cont.) WP7: Adaptive Application Components (VU) develops a set of components and APIs to be plugged into the toolkit, for example to take monitoring information and implement basic techniques for short-term forecasting and behavior adaptation/optimization. WP8: Data Handling and Visualization (ZIB) will provide Grid aware techniques for data management, analysis, and visualization, needed especially for applications that make use of multiple sites in a dynamic, time dependent manner, leaving data unpredictably scattered across the Grid. WP9: Resource Management (PSNC) will develop resource need estimators, resource brokers, and other tools, for both Grid users and the applications themselves to make intelligent decisions about which Grid resources should be used at any instant in the lifetime of a simulation.

MANs Meeting, Warsaw, NASK, September 2002 Workpackages (cont.) WP10: Information Services (ISUFI) will extend existing Grid middleware toolkits with dynamic features needed by applications to select appropriate Grid resources and to provide simulation information to collaborative user groups. WP11: Monitoring (SZTAKI) will develop new components that will fit in the general Grid monitoring architecture to support application steering, adaptive monitoring, and automatic analysis and prediction of performance data. WP12: Access for mobile users (PSNC) will develop and test Grid access and monitoring technologies through a variety of mobile devices,

MANs Meeting, Warsaw, NASK, September 2002 Workpackages (cont.) WP13: Information Dissemination and Exploitation (PSNC) will ensure the active dissemination of the project results through a variety of channels, including active participation in international organizations (e.g. GGF), co-development with other Grid projects in the USA and EU, participation in international conferences, training programs, instruction of GridLab technologies into various communities, and introduction into the commercial vendor world. WP14: Project Management (PSNC) day-to-day scientific, financial and administrative management of the project, including careful orchestration and monitoring of work across groups, major project decisions, liaisons with external projects and with the international advisory board, reporting

MANs Meeting, Warsaw, NASK, September 2002 More info / summary You’re welcome to join our testbed We are open for collaborations!