Test and Integration Current State Santiago González de la Hoz (on behalf of Integration Team) Santiago.Gonzalez@ific.uv.es Instituto de Física Corpuscular Consejo Superior de Investigaciones Científicas, CSIC Universitat de València Valencia, SPAIN http://gridportal.fzk.de/websites/crossgrid/iteam/

Overview Goals For this purpose: Implement the second-third tools-services and first application prototypes running on the international CrossGrid testbed For this purpose: A closed collaboration among WP1, WP2, WP3 and WP4 has been achieved (Jesus is leading the whole CrossGrid integration effort). Since integration meeting in Poznan (July 2003) The CrossGrid Integration team was formed. WP2 and WP3 middleware (prototype 1) and a selective applications (prototype 0) were deployed on CrossGrid testbed Integration Team (WP4, WP3, WP2 and WP1) is working to support of WP1 application prototype 1 Production testbed is based on LCG-1 middleware Development testbed based on EDG2.x middleware Now in Nicosia (January 2004): Integration Status middleware Integration among tasks demo Work for next months

Testbed evolution (see Jorge’s talk) Production testbed: From EDG 1.4.11 to LCG-1 LCG-1 is: VDT 1.1.8-9 (Globus 2.2.4) Information System (MDS) Selected software from EDG 2.0 Workload Management System (RB) EDG Data Management (RLS, LRC, …) GLUE Schema 1.1 + LCG extensions RB obtains the list of the existing gatekeepers from the MDS information tree It uses the lcgpbs job manager. Remove the lcgpbs jobmanager and configure a plain pbs jobmanager to be compatible with MPI. Development testbed EDG 2.x release using LCFGng and RH7.3 The functionality of the Globus MDS has been replaced by RGMA

Integration Work (Integration Team) Related task Developer WP4 integrator Status 1.1 Biomedical appl. D. Shamonin A.Tirado Prototyping 1.2 Flooding appl. Branislav Simo J.Astalos Tested at several sites 1.3 HEP application D.Rodriguez, C.Martinez-Rivero, I. González, M. Bluj R.Marco Installed and tested 1.4.a Meteo WAM-COAMPS J. Gajeuski, M. Mogielnicki A.Paddee 1.4.b Meteo SOM A.Cofiño, J.M. Gutierrez R. Marco Tested at CSIC 1.4.c Air Pollution J.Mouriño J. Fontán Tested at CESGA 2.2 MARMOT B.Krammer Installed and tested at CSIC(IFCA) 2.3 GridBench G.Tsouloupas C.Kanellopoulos Installed and tested at AUTH 2.4.a GPM W. Funika P.Nycyk Installed and tested at CYFRONET 2.4.b PPC F.Rivera C.Fernández Installed and tested at CESGA 3.1.a RAS + MD M. Plociennik P.Wolniewicz Installed and tested at PSNC 3.1.b Portal Y.Perros, M. Kokkosoulis J.Marco Partially installed at CSIC (IFCA) 3.2 Scheduler M.A.Senar A.Fernandez, J.Sanchez Prototype tested at CSIC (IFIC) 3.3.a OCM-G R.Wismuller A.Garcia 3.3.b SANTA-G B.Coghlan, S.Kenny M.David, D.O’Callaghan Installed and tested at TCD/LIP 3.3.c Jiro Monitoring K. Balos M.Hardt 3.3.d Global A.Padee 3.4 Data Access Lukasz Dutka S.Gonzalez Installed at CSIC(IFIC)

Integration work Integration Document It is based on the contributions received from integration team people more or less each month. The document covers the following issues: Couple of lines saying what is the software released (including version) Pre-Form points (based in developers guide) Code in CVS and how is organized Compliance of QA rules Packaging granularity Level of autobuild experience Explicit dependency list Test-request (based in test and validation proposed by task 4.4) Documentation: installation/management manual, user manual, development manual Unit test Compatibility with other software Security issues Usability Previously reported issues Plans for demo Status Use of other CrossGrid components Being Updating in Cyprus (Appendix Deliverable D4.6)

Integration between Application and Services&Tools Prototypes Interactive Job Scenario (see S. Beco’s talk) To run interactive job for the simplified X# HEP application X# middleware based on the EDG-2.0.18 plus HEP (Task 1.3) executable Computes a series of histogram for Gaussian Distribution User interacts with the RAS (Task 3.1) via the X# MD (VNC mechanism is put in place in order to allow users to submit jobs and see the histogram) In Cyprus demo, they don’t have a web service that allows the user to submit the interactive job. User launches the interactive submission service using a command line

Integration between Application and Services&Tools Prototypes Application: WP1.1 Interactivity using MD Working in integrating the application on the testbed Submissions of jobs outside Amsterdam cluster: Solver executable needs environment variables to run at other testbed sites

Integration between Application and Services&Tools Prototypes Application: WP1.2 Running without problems with the MD and the Portal (Task 3.1) Integration done with 3.1 (MD) and 3.2 (Scheduling Agent) Demo presents: jobs submission visualization results Integration with their portal based on Jetspeed with new components (workflow, collaborative tools) Problems with job submission (authorities) Data Management lacks some required functionalities???? Integration with Marmot (WP2) is OK Working with MD new version Demo: Marmot + OCM-G (G-PM) + Portal

Integration between Application and Services&Tools Prototypes Application: WP1.3 Running with MPICH-P4, submission time was long Fixed using Replica Manager Submit MPI jobs from the UI to the new EDG-2.x Resource Broker (To show MPICH-G2, in contact with Task 3.2). Testing Santa-G possibilities Integrated with the MD DEMO: Using Replica Manager Inclusion of graphics in Portal

Integration between Application and Services&Tools Prototypes Application: WP1.4 Atmospheric and Wave models: Integrated with the MD and working to be integrated with Portal Running in four processors in a local cluster. Air pollution: Integrated with the MD (old configuration) Trying integrated the data coming from atmospheric prediction Portal part was integrated for Poznan They have installed their application in Valencia’s testbed and it runs ok. DEMO: They plan use the MD + PPC

Integration between Application and Services&Tools Prototypes WP2 MARMOT: Being integrated with MD Integrated with WP 1.2 (Flooding application) GridBenchmarking: Progress with the GUI in the MD They use applications for performance predictions Benchmarking using different locations Same Kernel as PPC (Vertlq) In contact with Task 3.2 people about MPI jobs (Interested on MPICH-G2) Tutorial ready Being integrated with JIMS Interested in Santa-G

Integration between Application and Services&Tools Prototypes WP2 GP-M (OCM-G): Show monitoring of Flooding applications with GPM tool PPC (Performance Predictions): The GUI on MD is working fine Problem: they need monitoring information at WN Working in this (Discarded NWS solution, waiting for JIMS) Tutorial ready Demo: They will show the tool

Integration between Application and Services&Tools Prototypes WP3 Task 3.1 a) RAS and MD: RAS machine and services installed in Poznan, Lisbon and Nicosia (problems with JSS) MD installed in Pawel’s Laptop (to assure functionality) Medical, Air pollution and Flooding applications used as pilots to present functionality MD presents MPI jobs in distributed environment Interactivity issue is presented by Task 3.1, 3.2 and 1.3 (HEP) Task 3.1 b) Portal: Two portlets added to the portal (job-get-output, job-get-info) Medical, HEP, Flooding and Air pollution applications ready with the portal to be shown in Demo Show the generic portlets with all edg-commands

Integration between Application and Services&Tools Prototypes WP3 Task 3.2: New RB (EDG2.x and the same used in LCG) Ready to work with MPICH-P4 Integrated with RAS and MD Functionality is rather the same as presented in Poznan, the effort was dedicated to make it available under LCG1 and EDG2.x Finishing MPICH-G2 support Training application running HEP Problems related to connectivity of WN’s.

Integration between Application and Services&Tools Prototypes WP3 Task 3.3.4 (Postprocessing, past 3.3 subtask) RPM ready in CVS??? It’s being autobuild Install in two clusters in Warsaw (but it must be installed on all sites driven by the scheduler) The input data are taken from ganglia and iptables modules: But it has to be integrated with infrastructure monitoring tools developed in CrossGrid (Santa-G and Jiro). Work to be done during integration meeting in Cyprus. Task 3.3 OCM-G: Investigating which applications are suitable for monitoring submitted from the MD (MD plug-in to support monitoring) Demo shows integration of quite number of X# elements (MD, APPS, and G-PM)

Integration between Application and Services&Tools Prototypes WP3 SANTA-G: RPM’s available Working on Demo with HEP application and snort tool Santa-G works fine with EDG2.x RGM-A JIMS: It is working in local cluster at Cyfronet RPM’s finished but…. Using web service commercial tool Input to performance predictions has been discussed Task 3.4 Functionality exploited with LCG-1 and EDG2.x They are using four storage systems: (Cyfronet, Lisbon, Slovakia, and Karlsruhe) Working in Integration with MD Demo: Estimation real access cost to these storage systems

Agenda 9:00 Integration Status Demos 10:30 Coffee Break Integration setup at Cyprus (Wei 5’) General description of integration work (Santi, 10’) Testbed support for integration (Jorge, 10’) Repository and tools (Marcus, 10’) Demos 9:30 Flooding (Jan and Bettina, 60’) Marmot + OCM-G + G-PM + Discussion 10:30 Coffee Break 11:00 Medical applications (Alfredo, 30’) 11:30 HEP (Celso, 15’) 11:45 Meteo Part (Carlos,15’) 12:30 Interactivity (Stefano, 30’) 13:00-14:00 Lunch 14:00 GridBenchmarking + JIMS (George, 25’) 14:25 Data Access Optimization (Lukasz, 15’) 14:40 Santa-G ( Stuart, 15’) 14:55 Postprocessing (Adam, 15’) 15:10 MD+Portal (20’) 15:30 Help Desk (Farida, 15’) 15:45 Discussion IST Demo Review