General Intro to GridLab Jarek Nabrzyski et al. Poznań Supercomputing and Networking Center

GridLab Conference, Eger, 31 March-1 April 2003 GridLab Project Funded by the EU (5+ M€), January 2002 – December 2004 Application and Testbed oriented Cactus Code, Triana Workflow, all the other applications that want to be Grid-enabled Main goal: to develop a Grid Application Toolkit (GAT) and set of grid services and tools...: resource management (GRMS), data management, monitoring, adaptive components, mobile user support, security services, portals,... and test them on a real testbed with real applications

GridLab Conference, Eger, 31 March-1 April 2003 GridLab Members n PSNC (Poznan) - coordination n AEI (Potsdam) n ZIB (Berlin) n Univ. of Lecce n Cardiff University n Vrije Univ. (Amsterdam) n SZTAKI (Budapest) n Masaryk Univ. (Brno) n NTUA (Athens) Sun Microsystems Compaq (HP) n ANL (Chicago, I. Foster) n ISI (LA, C.Kesselman) n UoWisconsin (M. Livny) collaborating with: Users! EU Astrophysics Network, DFN TiKSL/GriKSL NSF ASC Project other Grid projects Globus, Condor, GrADS, PROGRESS, GriPhyn/iVDGL, CrossGrid and all the other European Grid Projects (GRIDSTART) other...

GridLab Conference, Eger, 31 March-1 April 2003 GridLab Applications n Cactus ( n Triana ( n other...

GridLab Conference, Eger, 31 March-1 April 2003 GridLab Aims Get Computational Scientists using the “Grid” and Grid services for real, everyday, production work (AEI Relativists, EU Network, Grav Wave Data Analysis, Cactus User Community), Make it easier for applications to make flexible, efficient, robust, use of the resources available to their virtual organizations Dream up, prototype, and test new application scenarios which make adaptive, dynamic, wild, and futuristic uses of resources.

GridLab Conference, Eger, 31 March-1 April 2003 What GridLab isn’t We are not developing low level Grid infrastructure, We do not want to repeat work which has already been done (want to incorporate and assimilate it …) Globus APIs, OGSA, ASC Portal (GridSphere/Orbiter), GPDK, GridPort, DataGrid, GriPhyn,...

GridLab Conference, Eger, 31 March-1 April 2003 GridLab end user requirements Application oriented environment, Applications on one or more virtual organisations, Flexible, easy-to-use, simple interfaces resources, jobs, and data (including compiling, tracking jobs, cataloguing data), Means to make efficient and effective use of resources, Robustness smart adaptivity, complete control and fail safety are available on all levels, The ability to work in a disconnected environment, Mobile working,

GridLab Conference, Eger, 31 March-1 April 2003 GridLab end user requirements From laptops to fully deployed Virtual Organisations, Complexity hidden as much as possible, Collaborative infrastructure, The infrastructure for all classes of applications The infrastructure must provide capabilities to customise choice of service implementation (e.g. using efficiency, reliability, first succeeding, all)

GridLab Conference, Eger, 31 March-1 April 2003 Solution... GAT – a layer between apps and emerging grid technologies GridLab testbed/VO Close cooperation between developers and deployers End Users GAT Tool Developers Grid Infrastructure Developers GAT-API Developers

GridLab Conference, Eger, 31 March-1 April 2003 The Grid is complex … Monitoring Resource Management Information Security Data Management GLOBUS Application Manager Logging Notification Migration Profiling SOAPWSDLCorbaOGSAOther Other Grid Infrastructure? Application “Is there a better resource I could be using?”

GridLab Conference, Eger, 31 March-1 April 2003 …need to make it easier to use GAT Application “Is there a better resource I could be using?” GAT_FindResource( ) The Grid

GridLab Conference, Eger, 31 March-1 April 2003 GridLab Architecture

GridLab Conference, Eger, 31 March-1 April 2003 GridLab Architecture

GridLab Conference, Eger, 31 March-1 April 2003 Grid Application Toolkit The GAT provides functionality through a carefully constructed set of generic high-level APIs, through which an application will be able to call the underlying grid services, Set of application developer APIs for Grid tools, services and software libraries, (and example implementations) that support the development of grid-enabled applications (open source!) Usable from any high level “application” (any generic code, Cactus, Triana, Portals, Scripts, …)

GridLab Conference, Eger, 31 March-1 April 2003 GAT More or less … Set of calls GAT_ToolOrService(arguments) Your chosen tools/services: resource broker, information server, application manager, grid monitoring, data manager, notification, etc. Set of APIs for dealing with the GAT (registration, information, errors, fault tolerance)

GridLab Conference, Eger, 31 March-1 April 2003 GAT Engine When an application makes a GAT-API call, the engine searches through an internal database of adaptors for the requested capability and calls it

GridLab Conference, Eger, 31 March-1 April 2003 GAT Adaptor Interface between GAT Engine and one or more capabilities Translates user requests to appropriate interface syntax for a capability provider Active adaptors change dynamically Includes “security context” Return appropriate error codes Examples OGSA adaptor (provides many capabilities) Globus adaptor (directly talk to gatekeepers) Adaptors for each GridLab service provider “Local” adaptors (GAT_MoveFile => “cp”, GATFindResource => “localhost”)

GridLab Conference, Eger, 31 March-1 April 2003 GAT: AIM Abstract Grid capabilities (services) from the application developer. Application developer concentrates on the functionality as needed by the application. Hide complexity. Provides a layer (buffer zone) between applications and the Grid.

GridLab Conference, Eger, 31 March-1 April 2003 GAT Example Machine A Gridftp Gridscp Machine B Gridscp “Application” (Cactus Migration Tool, User Portal, Grid Script) wants to move a file between two other machines How to move the file? Available software User authentification Disk properties (disksize, user quota, inodes) Network bandwidth

GridLab Conference, Eger, 31 March-1 April 2003 GAT Example GAT_FileCopy(GAT, “ThisMachine:ThisFile”, ”ThatMachine:ThatFile”) GAT_GFTPAdaptor IGAT_FileCopy IGAT_FileMove IGAT_MkDir GAT API call from Cactus Application, or Portal, or Python script GridFTP Adaptor says it knows how to both Copy and Move files between machines GridSSH Adaptor only knows how to Copy files GAT_GSCPAdaptor IGAT_FileCopy GAT Engine Engine decides (by different means) how to respect this request

GridLab Conference, Eger, 31 March-1 April 2003 The Same Application … Application GAT Application GAT Application GAT Laptop The Grid Super Computer No network! Firewall issues!

GridLab Conference, Eger, 31 March-1 April 2003 GridLab services Software environment for Grid-enabling scientific applications GridLab services, third party services and various core-grid services will be supported by GAT In the advent of the Open Grid Service Architecture (OGSA), GridLab's architecture will revolve around the notion of services, all the GridLab services will be OGSA compliant currently all the services are Web Services based roadmap for Web Services to OGSA transformation is being prepared (6-8 months from now)

GridLab Conference, Eger, 31 March-1 April 2003 GridLab services A primary aim of this project is to produce a GridLab GAT containing a set of high quality services which provide a complete environment for Grid- enabling generic applications GridLab services: implement common (strict) security use common service conneciton protocols (WSDL/OGSA) are built primarily for Globus infrastructure

GridLab Conference, Eger, 31 March-1 April 2003 What are the GL services? Authorisation Service (WP6) Adaptive Services (WP7) Data Management Services (WP8) Resource Management System (GRMS) (WP9) Information Services (WP10) Monitoring Services (WP11) Mobile User Support (WP12)

GridLab Conference, Eger, 31 March-1 April 2003 Security (WP6) Security WP focuses right now on the Authorization Service (AS) The main requirement is flexibility The AS is about to provide universal way of defining security policy for the whole Grid, independent of technologies used at lower levels It should be able to implement most security models for Grids and use many different scenarios at the same time

GridLab Conference, Eger, 31 March-1 April 2003 Security (WP6) – current status The Core AS Component is currently under heavy development The basic prototype should be available for initial tests in Spring 2003 At this time, an experiment aimed at verification of various scenarios for communication between GridLab services will be conducted (integration of portal with GRMS) Upon this experience, the proposal for GridLab security model will be presented

GridLab Conference, Eger, 31 March-1 April 2003 Adaptive Components (WP7) Adaptive Components Service (ACS) and the Local Adaptive Components (LAC). ACS provides an interface to query the adaptive system. It currently supports calls to: rank resources estimate transfer time estimate usage (of some given metric) LAC uses the monitoring system (shown in blue), to continuously collect data about the resource and applications running on it (load information, queue lengths, network bandwidth to other machines, etc.).

GridLab Conference, Eger, 31 March-1 April 2003 Data Management Services (WP8) replica catalog prototype was ready at Zakopane meeting data movement/copy service also since Zakopane meeting supports reliable gridftp file transfer is gsi enabled with authentication and delegation scalable and fault-tolerant replica catalog in M12, based on ongoing research at ZIB Soap Host A Host B Host C Host D automatic load-balance, fail-over between replica catalogs external access via SOAP and OGSA internal communication via more efficient protocol (Corba)

GridLab Conference, Eger, 31 March-1 April 2003 Resource Management (WP9) Job Receiver Jobs Queue Broker Execution Unit Resource Discovery Scheduler Resource Reservation Prediction Unit File Transfer Unit MDS Data Management Executable Repository GRAM Authorization System Adaptive Workflow Manager QoS/SLA Negotiation GRMS Third party systems n Chimera n DataGrid n Crossgrid n...

GridLab Conference, Eger, 31 March-1 April 2003 Information Services (WP10) Client Information Service (OGSA) SOAP over GSI Users Software Services Firewall V.O. C. A. Cluster Job Queues MDS GSI-SASL

GridLab Conference, Eger, 31 March-1 April 2003 Monitoring System (WP11) Implements GGF’s GMA architecture Fast and robust Small resource usage Can monitor hosts and jobs Can deliver event notifications

GridLab Conference, Eger, 31 March-1 April 2003 Mobile User Support (WP12) Grid Services Network Environment / Grid Portal Applications User Mobile device

GridLab Conference, Eger, 31 March-1 April 2003 Services - Summary GridLab Services are the primary capability providers for GAT They can be used by other projects and production environments GridLab can also accomodate services from other projects, when necessary First pilot implementations are ready and have been used for several demos including SC02, where GridLab Team working with GGTC won or co-won 2 Challenge Competitions

GridLab Conference, Eger, 31 March-1 April 2003 GridLab Testbed Managed by Masaryk University, Czech Republic Operational since March 2002 Most of the machines are not dedicated, except one cluster at PSNC Built on existing middleware: Globus Toolkit 2.0 MDS 2 PKI/GSI for secure access Provide functional Grid environment for development and testing of GATs, services and their components Production stability a goal No long production jobs A core team of the Global Grid Testbed Collaboration

GridLab Conference, Eger, 31 March-1 April 2003 Highlights The project is up and running! All partners are now contributing at contractual level Project administrative and managerial structure established with minimum resources All EU deliverables (30, 700+ pages) submitted in time for the review First testbed delivered with real production demos First (very successful) demos have been given

GridLab Conference, Eger, 31 March-1 April 2003 Project Schedule Project started on 1st of January 2001 Requirements documents: Month 3-4 Design documents: Month 6-12 (continously updated) Testbed running: Month 3 First prototypes: Month 12 (actual: 9) First release: Month Second release: Month 28-36

GridLab Conference, Eger, 31 March-1 April 2003 More info / summary Bring your application and test it with the GAT and our services.

GridLab Conference, Eger, 31 March-1 April 2003 The goal of this meeting „Let’s make the most advanced grid in the world” Michael Russell, AEI