EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Application porting in EGEE Gergely Sipos MTA SZTAKI Hungarian Academy of Sciences 3 rd EGEE User Forum, Clermont-Ferrand

Enabling Grids for E-sciencE EGEE-II INFSO-RI Application Porting in EGEE 2 Outline Application porting in EGEE-II and EGEE-III –Groups –Methods –Infrastructures Porting tools and experiences –P-GRADE Portal –GridWay –GANGA and DIANE –GENIUS Problems, solutions Conclusions

Enabling Grids for E-sciencE EGEE-II INFSO-RI Application Porting in EGEE 3 Acknowledgements Input and slides from –Peter Kacsuk, SZTAKI –Ignacio M. Llorente, José Luis Vázquez-Poletti, UCM –Roberto Barbera, Antonio Calanducci, INFN –Isabel Campos, Gabriel Amorós, CSIC –Patricia Mendez Lorenzo, CERN

Enabling Grids for E-sciencE EGEE-II INFSO-RI Application Porting in EGEE 4 Application porting, “gridification” Porting legacy applications to EGEE –Legacy: some code that runs on a local infrastructure –Not development from scratch Includes –Consultancy –Customized training –Technology support – / telephone help desk –Work together Generic –Independent from VO, application domain, project

Enabling Grids for E-sciencE EGEE-II INFSO-RI Application Porting in EGEE 5 Application porting in EGEE-II NA4 – Application Identification and Support –Application incubation: generate interest  NA2 Dissemination, NA3 Training, NA4 GILDA –Deployment as a local application  Supported by a single resource centre  Porting support from Your institute Your discipline Your region –Transition to a supported VO  Porting support in large VOs Porting expertise developed at several centers independently –Serving colleagues, local users, disciplines Generic porting is unfounded effort now

Enabling Grids for E-sciencE EGEE-II INFSO-RI Application Porting in EGEE 6 Generic application porting groups MTA SZTAKI, Budapest –Grid Application Support Center (GASuC)   Peter Kacsuk  Gergely Sipos INFN, Catania –GILDA Team  Roberto Barbera UCM, Madrid –Distributed Systems Architecture Research Group  Ignacio M. Llorente CERN, Geneva –IT-Grid Support group  Patricia Méndez Lorenzo CSIC, Santander –Institute of Physics of Cantabria  Isabel Campos Plasencia

Enabling Grids for E-sciencE EGEE-II INFSO-RI Application Porting in EGEE 7 Grid Application Support Group

Enabling Grids for E-sciencE EGEE-II INFSO-RI Application Porting in EGEE 8 Application porting in EGEE-III Start on the 1 st of May for 2 years, key project objectives –Expand/optimise existing EGEE infrastructure, include more resources and user communities –Prepare migration from a project-based model to a sustainable federated infrastructure based on National Grid Initiatives NA4 support areas –Application porting support 261 PM –Direct user support 232 PM (~ GGUS) –VO support 70 PM Named partners, financial support for generic application porting –MTA SZTAKI (Hungary) 126 PM –UCM (Spain) 48 PM –INFN (Italy) 36 PM –ASGC (Taiwan) 24 PM –CEA (France) 11 PM –CSIC (Spain) 10 PM –UM (Australia) 6 PM

Enabling Grids for E-sciencE EGEE-II INFSO-RI Application Porting in EGEE 9 Application porting support process Support group contacted –Directly, or as a result of dissemination (e.g.NA2, NA3, NA4) Survey, interviews to learn about the current and targeted use case, the nature of the problem Technical analysis. Outcome: –Recommendations for methods, concepts, tools, infrastructure –Roadmap Arrange access to infrastructure and tools Actual porting: porting group helps, but this is basically your task –Training –VO specific support –NA4 managerial support Feedback, documentation, dissemination Support process finished

Enabling Grids for E-sciencE EGEE-II INFSO-RI Application Porting in EGEE 10 Survey & interviews Current use case? Executable? –Sequential, MPI, script, … Input/output size and nature? Frequency of execution? Deadline for a single run? Loose or tightly coupled jobs? Reason, motivation of porting? User interface expectations … To build up a common understanding

Enabling Grids for E-sciencE EGEE-II INFSO-RI Application Porting in EGEE 11 Application description template

Enabling Grids for E-sciencE EGEE-II INFSO-RI Application Porting in EGEE 12 Technical analysis Re-structuring the code to the grid (job and data management) –Identifying „a grid job”  Type of executable, required libraries and environments  Sandboxing or installing on SE? –Identify „an execution”  Single job  Parameter study (parametric jobs)  Workflow  Parameter study workflow  Custom (e.g. loops over some data) Additional services –Monitoring –User interfaces –Meta-data management –Etc. Specification: what and how should be used + Roadmap

Enabling Grids for E-sciencE EGEE-II INFSO-RI Application Porting in EGEE 13 Getting access to the infrastructure Part 1: certificates Getting to the point where the user is able to submit a job is hard! –Grid CAs are sometime perferc filters There is solution: –A user friendly CA: ASGC CA, Taiwan, Asia-Pacific  Proper tutorials on webpage  Request process: Fill out a web form, click a button  Keys are generated directly into the browser  No need to ssh and openssl -a –b –c -d param1 param2 param3 Application porting group helps obtain certificate from the CA Even better: –Certificate obtained during training event  At least 3 days long training is required –Example: Debrecen, Hungary,  Certificate from NIIF CA, Hungary  Registration to SEEGRID VO (Catch-all VO of Southern Eastern Europe)  3 rd day: Using a real grid with real certificate –Teach rather one tool but reach the level of a real user

Enabling Grids for E-sciencE EGEE-II INFSO-RI Application Porting in EGEE 14 Port directly to production VO when possible –CIC portal: –Registration information is there. Fine, –What services can I reach then? This lacks even for some of the largest VOs! –There is solution. See for GILDA VOhttps://gilda.ct.infn.it/testbed.html “In-house” or regional porting VOs when necessary: –INFN – GILDA –CERN – GEAR VO –SZTAKI – VOCE, SEEGRID, GILDA –UCM – GridIMadrid, DSA Group VO –... EGEE-III: NA4 VO, dedicated for application porting –Currently under setup –Production infrastructure –Sites from NA4 partners –Support for MPI, Short deadline jobs, Hydra, AMGA, … –Not large, but reliable –SAM tests to help migration to real VO Setting up a new VO for the application?  NA4 VO managers group Getting access to the infrastructure Part 2: VO

Enabling Grids for E-sciencE EGEE-II INFSO-RI Application Porting in EGEE 15 Manual porting vs. automated porting Manual –Writing scripts, wrapper programs from scratch –Quite often a necessary step  to test new concepts Automated porting –Interfacing with gLite/LCG services through high level tools –Solutions for problems common in many grid application –NA4 RESPECT program  P-GRADE Portal SZTAKI Porting group  GridWay UCM Porting group –Other tools  GANGA and DIANE ganga.web.cern.ch/ganga cern.ch/dianeganga.web.cern.ch/gangacern.ch/diane CERN Porting group  GENIUS Portal genius.ct.infn.itgenius.ct.infn.it GILDA Porting group

Enabling Grids for E-sciencE EGEE-II INFSO-RI Application Porting in EGEE 16 Typical goals of porting Porting for myself –Demand for large capacity  Porting from another grid  Coming from a local environment Porting for my group/VO/project –Customized UIs might required –Typically motivated by a grid project Porting to operate grid application as a service –Application developer is not the end user –End user should know nothing about grids Porting Execution Porting Customization Execution Porting Customization Execution Where does the porting phase end, where does customization phase begin?

Enabling Grids for E-sciencE EGEE-II INFSO-RI Application Porting in EGEE 17 Porting tools Finding the right tools is not easy… P-GRADE Portal –SZTAKI Porting group GridWay –UCM Porting group GANGA and DIANE ganga.web.cern.ch/ganga cern.ch/diane ganga.web.cern.ch/gangacern.ch/diane –CERN Porting group GENIUS Portal genius.ct.infn.itgenius.ct.infn.it –INFN Porting group

Enabling Grids for E-sciencE EGEE-II INFSO-RI Application Porting in EGEE 18 General purpose grid application developer and user environment Based on GridSphere web portal framework To port parameter studies, workflows and workflow based parametric studies Open source, Community coordination: MTA SZTAKI Supported grid middleware services: ServiceEGEE VOs (LCG/gLite)Globus grids Job executionComputing ElementGRAM File storageStorage Element, File catalogGridFTP server Certificate managementMyProxy server, VOMS Information systemBDIIMDS-2, MDS-4 BrokeringWorkload Management System Application monitoring & visualization PROVE Grid interoperability at the workflow level  Migration between VOs & Grids P-GRADE Portal

Enabling Grids for E-sciencE EGEE-II INFSO-RI Application Porting in EGEE 19 APPLICATION CERTIFICATE PERFORMANCE EXECUTION    DESIGN MANAGEMENT ANALYSIS ON GRIDS Application porting lifecycle

Enabling Grids for E-sciencE EGEE-II INFSO-RI Application Porting in EGEE 20 Porting without code modification Nodes represent batch jobs to be executed on a Computing Element Ports represent input/output files locally read by executables Arcs represent file transfer and rename operations to be performed by P-GRADE Semantics of the workflow: –A job can be submitted if all of its input files are available –Responsibility of the built-in workflow manager Executables works with local files P-GRADE can transfer files from/to Storage Elements

Enabling Grids for E-sciencE EGEE-II INFSO-RI Application Porting in EGEE 21 Parallel processing efficiently – Parameter study execution of the workflow (Multiple workflows simultaneously) – Parallel execution among workflow nodes (different jobs on different CEs) Multiple jobs run parallel Each job can be a parallel program – Parallel component inside a workflow node (MPI job on a CE) Multiple instances of the workflow process different data sets

Enabling Grids for E-sciencE EGEE-II INFSO-RI Application Porting in EGEE 22 Parallel processing efficiently – Parameter study execution of the workflow (Multiple workflows simultaneously) – Parallel execution among workflow nodes (different jobs on different CEs) – Parallel component inside a workflow node (MPI job on a CE) Input Gen Result Coll – Generate input data files into Storage Elements – Collect and evaulate output files from Storage Elements

Enabling Grids for E-sciencE EGEE-II INFSO-RI Application Porting in EGEE 23 Pros, cons, examples Suited to parameter studies, workflows and the mixture of them Not simply a web portal –Development since 2003, >100 manyears, Support and use by open alliance –Customization may require the review of large source code Easy access: –Abstract GUIs  Use the grid without knowing how it works (almost) –Multi-grid portal: Connected to Biomed, Compchem, VOCE, SEEGRID, GILDA, HunGRID  Any other VO can register  contact P-GRADE Portal team –Source is now available Examples at –  Success stories, Current applicationswww.lpds.sztaki.hu/gasuc  Computational chemistry, Physics, Logistics, Life sciences, … –portal.p-grade.hu  Training, Documentsportal.p-grade.hu Meet the team Demo booth D19 and Training on Thursday

Enabling Grids for E-sciencE EGEE-II INFSO-RI Application Porting in EGEE 24 GridWay Execution manager for large set of similar jobs –Client side or central installation Includes a broker I/O staging from GridWay –I/O size per job is limited User transparent porting between Globus and EGEE grids Command line or API interface SGE Cluster Users PBS Cluster gLite GridWay gLite Some VO EGEE RB GridWay.org Job Template

Enabling Grids for E-sciencE EGEE-II INFSO-RI Application Porting in EGEE 25 CD-HIT: GridWay case study Job Template creates processes Job launches Waits/Resolves dependencies retrieves output files Application Job GridWay Compares protein DB entries Eliminates redundancies User leaves input database divisions in a local specified location User executes a program which interacts with GridWay through DRMAA API WridWay acts as a fault tolerant execution layer

Enabling Grids for E-sciencE EGEE-II INFSO-RI Application Porting in EGEE 26 Pros, cons, examples Suited for parameter study jobs I/O per job is limited –Storage Element, LFC access by your code For certain type of applications –Better job success rate than with RB –Higher throughput than with RB Application examples –Fusion, Bioinformatics, Computational Astrobiology, … Easy to interface from higher level GUI –DRMAA API

Enabling Grids for E-sciencE EGEE-II INFSO-RI Application Porting in EGEE 27 GANGA and DIANE ganga.web.cern.ch/ganga cern.ch/diane PBS LSF PANDA SGE LocalPC jobs GANGA execution manager for large set of master-worker jobs (Parameter study) –User defined data slitter and combiner –Transparent job migration from local PC to cluster or to grid –I/O staging from master server  I/O size per job is limited –Command line scripting or graphical interface to define GANGA jobs  Object oriented view of jobs DIANE is a fault tolerance and resource optimization layer –Master installed on the UI and submits DIANE worker as grid job –Worker pulls user job from master

Enabling Grids for E-sciencE EGEE-II INFSO-RI Application Porting in EGEE 28 Users and customized usage Originally for HEP –Large parameter studies –Fault tolerance –Cmd line is fine Other communities –Application specific interfaces e.g. GAP jobs Target selection Compound selection HARP Garfield

Enabling Grids for E-sciencE EGEE-II INFSO-RI Application Porting in EGEE 29 Pros, cons, examples Suited for parameter studies –User can define data splitter and collector logic Customization, interfacing from higher level tools can be an effort Used by several communities CERN remains unfounded from porting support Meet the team Tuesday Monitoring, Accounting & Support Increased productivity for emerging Grid applications: the application support system / Andrew Maier et al. Demo D04

Enabling Grids for E-sciencE EGEE-II INFSO-RI Application Porting in EGEE 30 GENIUS Grid Portal GUI mapped to gLite cmd line –Write JDL, Submit JDL, Check status, Download result GUI for Storage access TRIANA integration: execute DAG workflows Contact for a CVS

Enabling Grids for E-sciencE EGEE-II INFSO-RI Application Porting in EGEE 31 Application case studies Intensively used at „Gridification schools” –EELA project schools    GENIUS applications –EUMEDGrid schools   10 GENIUS applications –First International Grid School for Industrial Application  –e-Learning, Climate, Biomed, Optimization

Enabling Grids for E-sciencE EGEE-II INFSO-RI Application Porting in EGEE 32 Pros, cons, examples Suited for single jobs and workflows –JDL understanding still required. Big and helpful developer group –Experience with porting Customization can be an effort –Getting access to source is limited –Documentation? Meet the team Demo booth D18

Enabling Grids for E-sciencE EGEE-II INFSO-RI Application Porting in EGEE 33 Application porting and training Efficiency of training –Where do people disappear? Collaboration on regular surveys –Improve overall user support cycle Collaboration on porting schools: –First part dedicated to learn grid concepts and tools, getting access –Second part dedicated to application porting  Working in groups  Reporting, regular presentations –Concept piloted by INFN  EELA Schools, EUMEDGrid School, Industrial applications school, EUChinaGrid school –Also adapted by SZTAKI  Porting sessions at regular EGEE training events

Enabling Grids for E-sciencE EGEE-II INFSO-RI Application Porting in EGEE 34 Visiting a porting group Overcome initial difficulties of porting together –Obtain a grid user certificate  Different CAs, different rules –Find information about VOs  CIC Portal is good, but not enough –Registering at a VO  Certificate into browser –Where to find manuals, tutorials, reusable examples –How to distinguish application errors from grid errors –How to handle middleware and infrastructure problems –What to do if your code is erroneous Make you a confident grid user In EGEE-III –Community building budget  Application porting expert can visit you

Enabling Grids for E-sciencE EGEE-II INFSO-RI Application Porting in EGEE 35 Current problems MPI is still a problem for several communities –Not enough MPI sites –Heterogenous MPI configurations –Please share experiences, solutions, tricks, tips with porting group and the MPI Technical Coordination Group  Stephen Childs –Recently certified MPI gLite patch will hopefully help Case studies –Technical description –Useful for training, useful for application developers, useful for self sustainability –Catalyze a gLite user community webpage One site in many VOs –How many CPUs do I have then? Not every SE can be accessed from every CE –CE selection can be transparent. SE selection cannot

Enabling Grids for E-sciencE EGEE-II INFSO-RI Application Porting in EGEE 36 Conclusions Hard to survive the first few weeks on the grid –Generic porting support helps – Please disseminate this! –Do not let potential users disappear! EGEE is for everybody Porting support is improving –Organized effort in EGEE-III –Network of centers –Better feedback, better dissemination of results Existing tools save you time –Easy porting to grid, customization is what takes time –Established, off the shelf solutions for  Parameter studies  Workflows  Parameter study workflows

EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Thank you! Questions