1. INFSO-RI-508833 Enabling Grids for E-sciencE www.eu-egee.org GLOBAL GRID USER SUPPORT THE MODEL AND EXPERIENCE IN LCG/EGEE Gilles Mathieu(1), Torsten Antoni(2), Flavia Donno(3), Helmut Dres(2), Alistair Mills(3), Philippa Strange(4), David Spence(4), Min Tsai(5), Marco Verlato(6) (1) IN2P3/CNRS Computing Centre, Lyon – France (2) Forschungzentrum Karlsruhe, Karlsruhe – Germany (3) CERN, European Organization for Nuclear Research, Geneva – Switzerland (4) Rutherford Appleton Laboratory, CCLRC, Chilton – England (5) Academia Sinica, Tapei – Taiwan (6) Istituto Nazionale Fisica Nucleare (INFN), Padova - Italy

2. Enabling Grids for E-sciencE INFSO-RI-508833 G. Mathieu et al., CHEP06 Mumbai 13-17 February 2006 2 Outline Motivation GGUS: history and general description/architecture Services offered GGUS supporters: first and second line support Interface to the Regional Operations Centres (ROCs) GGUS for Grid Operations: CIC-on-duty Some performance number and statistics Conclusions and future work

3. Enabling Grids for E-sciencE INFSO-RI-508833 G. Mathieu et al., CHEP06 Mumbai 13-17 February 2006 3 Motivation A single access point for support A portal with a well structured information and updated documentation Knowledgeable experts Correct, complete and responsive support Tools to help resolve problems –search engines –monitoring applications –resources status Examples, templates, specific distributions for software of interest Interface with other Grid support systems Connection with developers, deployment, operation teams Assistance during production use of the grid infrastructure

4. Enabling Grids for E-sciencE INFSO-RI-508833 G. Mathieu et al., CHEP06 Mumbai 13-17 February 2006 4 Central Application (GGUS) Deployment Support Middleware Support Network Support Operations Support TPM ROC 1 ROC 10 ROC… VO Support Interface Webportal The Support Model “Regional Support with Central Coordination" The ROCs and VOs and the other project wide groups such as the Core Infrastructure Center (CIC), middleware groups (JRA), network groups (NA), service groups (SA) areCICJRANA connected via a central integration platform provided by GGUS. Regional Support units User Support units Technical Support units

5. Enabling Grids for E-sciencE INFSO-RI-508833 G. Mathieu et al., CHEP06 Mumbai 13-17 February 2006 5 The GGUS System

6. Enabling Grids for E-sciencE INFSO-RI-508833 G. Mathieu et al., CHEP06 Mumbai 13-17 February 2006 6 GGUS Portal: user services Browseable tickets Search through solved tickets Useful links (Wiki FAQ) Broadcast tools Latest News GGUS Search Engine Updated documentation (Wiki FAQ)

7. Enabling Grids for E-sciencE INFSO-RI-508833 G. Mathieu et al., CHEP06 Mumbai 13-17 February 2006 7 GGUS Portal: Search engine GGUSSearchEngine Ongoing work to make it faster and to search through a widerset of docs and DBs

8. Enabling Grids for E-sciencE INFSO-RI-508833 G. Mathieu et al., CHEP06 Mumbai 13-17 February 2006 8 GGUS Portal:… and more Hot line with supporters available to answer to problems VO specific supporting teams for middleware integration problems Links to specific useful tools and middleware Examples and templates Links to dedicated Virtual Organization portals that gather resources and monitoring information, contact mailing lists, etc. Modular system to accommodate further needs GGUS has shown to meet user needs

9. Enabling Grids for E-sciencE INFSO-RI-508833 G. Mathieu et al., CHEP06 Mumbai 13-17 February 2006 9 TPM Grid experts GGUS Supporters VO-TPM VO experts User First line support VO Support Units Middleware Support Units Deployment Support Units Operations Support ROC Support Units Network Support Second line support

10. Enabling Grids for E-sciencE INFSO-RI-508833 G. Mathieu et al., CHEP06 Mumbai 13-17 February 2006 10 Operations support Purpose/role –Detect problems by monitoring the grid –Report them by creating and assigning GGUS tickets –Provide help and follow-up on problems Operations Support teams : “CIC on duty” –Currently 6 teams (CERN, France, Italy, UK, Russia, Taiwan) –Weekly shift CIC/GGUS interface –Based on Web services at GGUS side –“CIC-on-duty dashboard”: graphic user interface for operators, hosted at IN2P3 Computing Centre (Lyon, France)

11. Enabling Grids for E-sciencE INFSO-RI-508833 G. Mathieu et al., CHEP06 Mumbai 13-17 February 2006 11 FZK, Karlsruhe, GermanyIN2P3-CC, Lyon, France CIC  ROC Basic Workflow CIC PORTAL CIC-on-duty dashboard UK FRGERIT … Regional Support Units Operator on duty - Create() - Set(ticket) SOAP -Get(ticket) - Get_all() Ticket Central Helpdesk CIC Helpdesk WSDL GGUS Problem detection & reporting Ticket follow-up GGUS Interface

12. Enabling Grids for E-sciencE INFSO-RI-508833 G. Mathieu et al., CHEP06 Mumbai 13-17 February 2006 12 GGUS  ROC Basic Workflow Web Portal GGUS System Operations support ROC-1 Helpdesk ROC-1 Interface Ticket solved Ticket assignment to ROC-1 SU-1 SU-2 SU-N ROC-X Helpdesk ROC-X Interface SU-1 SU-2 SU-N Ticket re-assigned TPM

13. Enabling Grids for E-sciencE INFSO-RI-508833 G. Mathieu et al., CHEP06 Mumbai 13-17 February 2006 13 ROCs Interface: how it works First Interface between ROC_Italy Helpdesk and GGUS – ready since November ’04 –in ‘production’ since March ‘05 Based on Web Services at GGUS side, several advantages: –sample code available for PHP/Perl/Python and other computing languages –very fast: 600-1000 service requests/sec on the GGUS Servers – easy to adapt Based on e-mail at local side (importing tool) XML exchange format Tickets fields mapping between the two systems http://infnforge.cnaf.infn.it/eticketimp/

14. Enabling Grids for E-sciencE INFSO-RI-508833 G. Mathieu et al., CHEP06 Mumbai 13-17 February 2006 14 Performance statistics September October A peak of 80 tickets per day has been reached. The system can handle up to 1400 tickets a week. November 2005: 315 tickets

15. Enabling Grids for E-sciencE INFSO-RI-508833 G. Mathieu et al., CHEP06 Mumbai 13-17 February 2006 15 Conclusions “Regional Support with Central Coordination" The GGUS model of “Regional Support with Central Coordination" has shown to be a workable and scalable solution for User support, given the feedback received from the users. GGUShas improved The functionality and usability of the GGUS system has improved in the last months, thanks to the help of the ROCs and the experience acquired. GGUS/CIC GGUS/CIC interface has made the system a central mechanism for Grid Operations and monitoring. interfaces with the ROCs The existent interfaces with the ROCs are quite practical and make the system function as one. do not knowrealistic figure The ticket traffic is increasing. We still do not know what a realistic figure would be for the number of ticket to be expected. However, the system can be dimensioned appropriately with more TPMs and support units. metrics established A lot of metrics established to measure the performance of the system. plan to offer resilience to system and network failures GGUS is working on a plan to offer resilience to system and network failures. We need more specialized supporters We need more specialized supporters and dedicated man power in order to guarantee good quality of service. Training sessions for supporters are organized by the GGUS Teams and recording made available off-line.