Meeting with Sun Microsystems at PSNC: Exploitation, 13 May 2004 GridLab 2003/4 „Steady leadership in changing times!” Jarek Nabrzyski Project Coordinator Poznan Supercomputing and Networking Center

GridLab Project Funded by the EU (5+ M€), January 2002 – December 2004 Application and Testbed oriented Cactus Code, Triana Workflow, all the other applications that want to be Grid-enabled Main goal: to develop a Grid Application Toolkit (GAT) and set of grid services and tools: Resource management (GRMS), Data management (GDMS), Monitoring (Mercury) and information services, Adaptive components (Pythia), Mobile user support and remote visualization, Security services (GAS), Portals (GridSphere),... and test them on a real testbed with real applications

GridLab Members n PSNC (Poznan) - coordination n AEI (Potsdam) n ZIB (Berlin) n Univ. of Lecce n Cardiff University n Vrije Univ. (Amsterdam) n SZTAKI (Budapest) n Masaryk Univ. (Brno) n NTUA (Athens) Sun Microsystems GmbH HP France n ANL (Chicago, I. Foster) n ISI (LA, C.Kesselman) n UoWisconsin (M. Livny) collaborating with: Users! EU Astrophysics Network, DFN TiKSL/GriKSL NSF ASC Project other Grid projects Globus, Condor, GrADS, PROGRESS, Clusterix GriPhyn/iVDGL, European Grid Projects (GRIDSTART) KISTI, Canadian Grid, LSU…

Review Issues and Concerns It is GridLab’s first direct contact with the new PO and 2 out of 3 reviewers are new to the project, Difficult to review a project of this size and complexity in several hours, Decision to present the project from the integration angle, Problems with synchronization of deliverables with the last- minute amendment (details in the WP presentations) Presentations: Short time per workpackage, short time for the demo, Innovative GAT, the portal and services presented in a practical demo during SC2003 in Phoenix The more features we add (to every single WP out of 12) the more complex the integration gets

GridLab Aims Get Computational Scientists using the “Grid” and Grid services for real, everyday, production work (AEI Relativists, EU Network, Grav Wave Data Analysis, Cactus User Community,...), Make it easier for applications to make flexible, efficient, robust, use of the resources available to their virtual organizations Dream up, prototype, and test new application scenarios which make adaptive, dynamic, wild, and futuristic uses of resources.

What GridLab isn’t We are not developing low level Grid infrastructure - we do more actually, we want to be infrastructure independent, We do not want to repeat work which has already been done (want to incorporate and assimilate it …) Globus APIs, OGSA/OGSI/…, ASC Portal (GridSphere/Orbiter), GPDK, GridPort, DataGrid, GriPhyn,...

Project Goals for 2003 Integration of all the GridLab software, “Zakopane Scenario” driving all the developments and integration, First release of GridLab software scheduled for December 2003, Make the testbed more stable and ready for any demo action and/or development, SC2003 was chosen to demo the GridLab software, Strong dissemination of the project in order to get attention of interesting Grid applications, Start the GAT and other GridLab standardization work under GGF, Strengthen management of the project, with clear responsibilities to ensure better control of resources, fast identification of the risks, ensure progress, GridLab Open Source License, Prepare the plan for exploitation of the results, Adjust the Technical Description of Work to the current needs of the project

What have we done during last 8 months? Response to the reviewers report sent to all Open Source License GT2 vs. GT3 issues (finally it was wise decision) Strategy for dissemination in 2004 Deliverables, some delays here (see details in the WP presentations) Contract amendment is on the way GridLab meeting in Eger (April) and Olomouc (October) More developments and integration More and more WPs brought to the scenario Today most of the WPs were integrated with each other SC2003 in Phoenix (2 big „demos”, many „small” demos) Painful demo, but successful: GRMS + portal + GAT application + Testbed + WP5 information service + Adaptive with some last minute problems with other services „The devil is in the details” - the most true sentence.

Major achievements Integration efforts and devotion of project staff, Meeting in Eger in March/April, meeting in Olomouc in October, Many small integration meetings (Poznan-Berlin-Potsdam, Potsdam- Budapest, Potsdam-Prague, Cardiff, GGF meetings and at other confs.), Getting a major attention towards the GAT of many application communities, GRIDSTART, GGF, application communities and 6FPP (EGEE, DEISA, HPC-Europa, …) Expectations are high! Many successful implementations of services Single GridLab services being used by other projects, Most of the services are fully integrated with each other, but can be configured as stand alone GAT can connect to most of the services. All will be available in a month.

Major achievement (cont.) GridLab middleware being exploited by other projects and production testbeds GridSphere: tens of projects (Grid Infrastructure Project, GEON, HPC-Europa, Physics Portal Development Project, …) GRMS (Clusterix, SGI Grid, Progress, Canadian Grid) Mercury Monitor (DataGrid, P-GRADE) WP8 Visualization - GriKSL Open Source License GridSphere has it’s own Open Source license

Major achievements Standards, collaboration Global Grid Forum Positions in GGF Alexander Reinefeld - GGF Advisory Board Jarek Nabrzyski – step down from the SC and AD for Apps and now take care of the GSA RG), Ed Seidel, Thilo Kielman chairing the Apps RG Thilo Kileman chairing Grid CPR WG Andre Merzky chairing the Persistent Archive WG and secretary of Data Area and Persistent Environments GAT architecture put at the Apps and APM groups Many WG talks about GridLab Putting the GAT to the BOF in October 2003 Very successful Participation in Gridstart! GridLab chairs at least half of the TWGs Other Projects: GriPhyN, GRIKSL, CrossGrid, Reality Grid (and some others) have a certain level of collaboration with GridLab Companies: Grid Systems, Platform Computing, IBM, HP, Sun

Major achievements Dissemination Improved the Several new papers accepted for books, journals and conferences Many talks have been given at major conferences and events Events organized GridLab Conference and Workshop in Eger, March-April 2003 Grid Portals Conference, August 2003, Edinburgh Grid Application Workshop in Czestochowa, September 2003 GridLab Conference and Workshop in Olomouc, October 2003 Grid Resource Management Workshop, Poznan, October 2003 Workshops at international conferences and symposia Tutorials on major conferences, also via AccessGrid 2 GridLab brochures In 2004 we will produce 14 different GridLab brochures featuring all the services, tools and advanced scenarios (see first three issues)

Book on Grid Resource Management 4 GridLab chapters: Application requirements for resource brokering Multicriteria resource management (GRMS) Triana peer-to-peer RM Workflow management ISI/Argonne/Wicsonsin contributions Published in November by Kluwer Academic Publishers

Technical Progress GAT API specification and GAT adaptor developer’s guide GAT tested by Cactus and a GAT subset implementation for Triana (GAP) Strenghten the collaboration with ISI, ANL, Wisconsin New Portal Framework (GridSphere) - 1st release in 2003 (2nd in 2004) - being used by tens of projects! Participation in the Grid Technology Repository (Jarek Nabrzyski serves as the Int. Advisory Board member) Strong participation in the GRIDSTART project (GridLab was one of the main contributors to the Technical Roadmap) Services ready and interoperable in spite of changing requirements GAT ready for standardization process, already under the community comment process

Technical Progress (cont.) Extending the GridLab scenario Successful meeting in Eger (new application users involved, new developments) and in Olomouc (integration) Problems/Issues GAT development delayed - underestimated efforts needed for the GAT development Should be fully ready in a month Will be revisited and refactored! User’s input is needed. Good docs is needed. Hiring a new person, experienced with Cactus development to speed up the CGAT integration. GAT documentation is behind! We are shifting efforts a bit to speed up the process. Some requirements and scenarios have changed dramatically: new ones appeared (WP7) Some unexpected problems with hackers on one of the test machines

Issues GAT development delayed - underestimated efforts needed for the GAT development Should be fully ready in a month Will be revisited and refactored! User’s input is needed. Good docs is needed. Hiring a new person, experienced with Cactus development to speed up the CGAT integration. GAT documentation is behind! We are shifting efforts a bit to speed up the process. Some deliverables are delayed New Annex1 caused some problems GAT development delayed (CGAT, GAT documentation) Some requirements and scenarios have changed dramatically: new ones appeared (WP7) Some unexpected problems with hackers on one of the test machines

Plans for 2004 Continue the development of GAT and services more info in the following presentations More complicated scenarios collaborative environments, submitting, controling and steering jobs from mobile devices, more dynamic behavior of applications Two more GridLab Workshop Meetings (Lecce: May, Zakopane: early December) Organizing the GridLab/GT3.2/WSRF integration meeting withUS partners (to ensure GAT compatibility) Preparing for the Supercomputing demos GGF BOF (GAT) and finally GGF WG GGF Scheduling Architecture Working Group now started (GRMS) Exploitation of the project results Close work with GridLab’s commercial partners Global Grid Application Alliance with GridLab’s leadership GGF activities (long term) GridSuit (based on GridLab plus Gridstart Open Source, PSNC plus partners) Open Source + commercial support European Grid Support Centre (PSNC)

Summary GAT is important We will put yet more effort into the GAT GAT will be revisited after finalizing the 1st release (in one month) Refactoring Lecce Meeting to bring application people, TWG8, tutorial, exploitation Documentation! GGF/Gridstart There was a great interest and we are talking to many groups The project works better than ever! Project is unique in this sense (this might be an impression of ours only) Integration was much more difficult than expected We had to deal with many (too many) unexpected problems in 3rd party software.

Collaboration with ISI

Collaboration with Argonne OGSA C bindings GridLab/GT3.2 Inegration Workshop (4 Argonne GT3 developers/experts are coming to help us with the integration) Help with developing the GT3.2 adaptors GAT review

Collaboration with Condor Group

Condor cont.

Meeting with Sun Microsystems at PSNC: Exploitation, 13 May 2004 Exploitation Plan Jarek Nabrzyski

Pillars of exploitation Close work with GridLab’s commercial partners Global Grid Application Alliance with GridLab’s leadership GGF activities (long term) GridSuite (based on GridLab plus Gridstart Open Source, PSNC plus partners) European Grid Support Centre (PSNC, HP) European Grid Technology Broker

Exploitation of the results through commercial partners HP Grid Resource Toolkit. HP investigates which GL software can be adopted. Distribution through server/software distribution Sun This meeting! Actions to be taken: GridLab will work closely with the companies to ensure that the companies evaluate the software and make right decisions on which software is supposed to be part of their distributions Responsible: WP managers will provide software plus documentation to a contact person at the vendor side.

Global Grid Application Alliance Global Europe - GridLab US - Ed Seidel Asia Pacific - both Satoshis Korea - KISTI Tasks Bring the GAT and services to the community Train people to make then use the GAT People will write adaptors for their favorite services Work on the new scenarios that the Alliance will bring Work on portable applications on a global scale using the GridLab technologies Actions: The initial meeting of the founders took already place SC will be responsible for the actions: meetings soon to be scheduled

GGF, GRIDSTART activities SAGA GSA and more will be continued beyond the timeframe of the project Gridstart TWGs will be used to work with other Grid projects in order to deploy the GAT and GridLab tools in these projects TWG chairs (from GridLab) are responsible to make it happen (GridLab WP managers, SC members)

GridSuite at PSNC GridSuite (based on GridLab plus Gridstart open source, PSNC plus partners) - Grid out-of-the-box complete solution Open Source + commercial support PSNC has declared to hire a team of 6-8 persons to build the GridSuite this year PSNC will work with GridLab partners to provide an expertise to build the GridSuite and to design a business plan for the effort. Release roadmap is under way. This might be a background for the commercial company selling the Grid services for European industry Actions to be taken: PSNC will discuss this with the GridLab Steering Committee soon, This will be more open to other GridLab partners (not limited to PSNC) Strategy is being developed by PSNC now Initial money from the European infrastructure funds (under way) and PSNC’s own funds.

European Grid Support Centre European Grid Support Centre (HP, PSNC plus UK eScience, CERN, PDC) Institutions’ own funds The task is to support different user groups (currently DataGrid, EGEE, GridLab, NorduGrid) The tools and the procedures are now being established. The ground for GridLab’s support for its users and infrastructure builders, Currently PSNC and HP from GridLab only. More will be involved with increase of the number of users. Actions: Build the procedures and assure that GridLab user’s know about the Center and use its services, To be discussed by the SC in more details in 6 weeks time, Extension of the group with a time and load (currently one person half- time at PSNC, similarly at HP, 3-4 persons at PSNC soon).

Exploitation by partners - general strategy Every GL partner will work with the users that the SC agrees to work with closely, In such a case particular institution will make sure that appropriate resources are assigned to work with the user in order to deploy GridLab software in the user’s environment, The appropriate resolutions will be worked out in details and decided by the Steering Committee by the existence of the GAT release, no later than by the end of April. At the beginning we will focus on strategic users to ensure the added value to the project. We will work closely with the GridLab application communities first of all (Cactus and Triana) on the GAT usage.

Beyond the project Seek for local funding by every partner Survive the gap after the GridLab Continue work, bring more users, get new requirements, analyze scenarios, work with the GGAA (Global Grid Application Alliance)