EGEE/LCG Operation Workshop 24th-26th May 2005 A report on operation support, open issues and statistics Marco Verlato INFN – Sezione di Padova www.eu-egee.org EGEE is a project funded by the European Union under contract IST-2003-508833

Outline History since last Operation Workshop EGEE User Support Framework Grid.it helpdesk support infrastructure usage report interface to GGUS ROC Integration ROC SE helpdesk overview ROC Russia, SW, GER-CH snapshots statistics Some Issues EGEE/LCG Operation Workshop – May 24-26, 2005 - 2

History Nov. 04: Outcome from User Support Task Force, Grid.it support infrastructure and pilot interface to/from GGUS presented at 1st EGEE/LCG Operation Workshop Nov. 04: Pilot GGUS-Grid.it helpdesk interface live demo at EGEE-2 Conference Dec. 04: Grid.it helpdesk code and interface documentation made available to other ROCs Jan. 05: E(gee, or xecutive) Support Commettee kick off at FZK, WP definition and mandate Mar. 05: Support on Duty start, GGUS/ Grid.it / Cic-on-duty helpdesks fully interfaced May 05: GGUS enhanced, SE and RU helpdesks interfaced EGEE/LCG Operation Workshop – May 24-26, 2005 - 3

EGEE User Support: requirements Support requests range from: Grid Services and Sites faults Problems with installation/configuration “How do I …?” Problems with applications Bugs Requirements for extra features Users may be Site Admins, VO application users, VO managers, … they all prefer a single point of contact for Grid problems User Support / Operation Support / VO Support are different but with a lot of overlap Different sets of experts and levels of support EGEE/LCG Operation Workshop – May 24-26, 2005 - 4

EGEE User Support: infrastructure The ROCs, VOs and the other project wide groups such as the Core Infrastructure Center (CIC), middleware groups (JRA), and network groups (NA), will be connected via a central integration platform provided by GGUS. This central helpdesk keeps track of all service requests and assigns them to the appropriate support groups. In this way, formal communication between all support groups is possible. To enable this, each group has to build only one interface between its internal support structure and the central GGUS application. EGEE/LCG Operation Workshop – May 24-26, 2005 - 5

EGEE User Support: interfaces Using the local Helpdesk Systems in conjunction with a central integration platform at GGUS Resource Center 1(RC) ... Resource Center N(RC) Local User Support Application Regional Operations Center (ROC) Third level support: Generic deployment Grid Middleware Report Problem The User Interface VO support Use the Webview Report Problem Central GGUS Application Interface CIC EGEE/LCG Operation Workshop – May 24-26, 2005 - 6

EGEE User Support: Responsible Units First Level Support GGUS team SOD (ROC experts rotation) Second Level Support CIC-on-duty ROC_Asia/Pacific ROC_CE ROC_CERN ROC_France ROC_GER/CH ROC_Italy ROC_North ROC_Russia ROC_SE ROC_SW ROC_UK/Ireland VOSupport (atlas,magic,biomed,compass,babar,cdf,alice,lhcb,cms,d0) Third Level Support (filled with experts provided by ROCs) Grid Deployment Castor Generic Deployment Manual Installation Pre-production system VO management/VOMS Grid Middleware d-Cache Data Management GLUE GridICE Information System/GIP/BDII R-GMA Security Management Workload Management ROC Helpdesks EGEE/LCG Operation Workshop – May 24-26, 2005 - 7

The Grid.it portal 31 RCs ~1400 CPUs ~120 TB 21 VOs +DAG+MPI +DGAS http://grid-it.cnaf.infn.it 31 RCs ~1400 CPUs ~120 TB 21 VOs +DAG+MPI +DGAS EGEE/LCG Operation Workshop – May 24-26, 2005 - 8

Deployment Status EGEE/LCG Operation Workshop – May 24-26, 2005 - 9

Services and Sites Monitoring EGEE/LCG Operation Workshop – May 24-26, 2005 - 10

Grid.it Helpdesk EGEE/LCG Operation Workshop – May 24-26, 2005 - 11

Trouble Ticketing System The trouble ticketing system is based on OneOrZero Helpdesk tool (www.oneorzero.com), coded in PHP, using MySQL, customizable, free Replaced with Xoops / xHelp tool soon Access allowed to registered members approved by administrators: End-users: they create the tickets describing problems or suggestions Supporters: fix the problems, or redirect somewhere else Site Managers: act as supporters for a given RC, and exchange tickets with Operatives for operational issues Operatives: people of ROC/CIC Central Management Team, Release & Deployment Team and Ticketing System Team itself, exchange tickets with Site Managers and Supporters EGEE/LCG Operation Workshop – May 24-26, 2005 - 12

ROC Support Units ~ 40 people + site managers EGEE/LCG Operation Workshop – May 24-26, 2005 - 13

Weekly shifts 4 people a day weekly rotating 8.30-19.30 working hours 11x5 coverage ICQ channel Mainly busy with Operations EGEE/LCG Operation Workshop – May 24-26, 2005 - 14

Usage Report Statistics for last 6 months of operations ~25 tickets a week on average EGEE/LCG Operation Workshop – May 24-26, 2005 - 15

Usage Report Grid services Operative teams Grid sites VO applications EGEE/LCG Operation Workshop – May 24-26, 2005 - 16

Interface to GGUS http://infnforge.cnaf.infn.it/eticketimp/ First Interface between Grid.it Helpdesk and GGUS ready since November 04 and in ‘production’ since March 05 Based on Web Services at GGUS side, several advantages: sample code available for PHP / Perl and other computing languages very fast: 600-1000 service requests/sec on the GGUS Servers easy to adapt Based on e-mail at Grid.it side (importer tool) XML exchange format http://infnforge.cnaf.infn.it/eticketimp/ EGEE/LCG Operation Workshop – May 24-26, 2005 - 17

Interface to GGUS EGEE/LCG Operation Workshop – May 24-26, 2005 - 18

GGUSROC Basic Workflow ROC Helpdesk GGUS System XML Mail GGUS/SOD Web Portal SUPP Unit CMT Ticket assignment CIC-on-duty CIC Interface SUPP Unit X Ticket solved notification SUPP Unit Y Web services EGEE/LCG Operation Workshop – May 24-26, 2005 - 19

ROC Integration All ROCs were asked to create/enable their Support Structure to be integrated with GGUS: providing a contact to their helpdesk system providing a well defined structure behind their helpdesk system providing a list of experts committed to VO support and 3th level support filling the corresponding GGUS Responsible Units Some ROCs set up an helpdesk system interfaced to GGUS following the Grid.it example using OneOrZero SE: ready in production since April 25th RU: work started in April, interface in production since May 23th SW: almost ready EGEE/LCG Operation Workshop – May 24-26, 2005 - 20

ROC SE Helpdesk Overview (slides from Alexandru Stanciu) Oneorzero v1.6 http://helpdesk.egee-see.org helpdesk is hosted by ICI, RO it's a new release (March) which has functional enhancements and new features over the 1.4 and 1.5 series Integration with GGUS made based on INFN example, but with local customizations In production since 25 April Decentralized structure EGEE/LCG Operation Workshop – May 24-26, 2005 - 21

ROC SE Helpdesk Structure Two kinds of support groups Per country support groups: BG, CY, GR, IL, RO Specialized support groups: 14 support groups ( Site Certification, VOMS, MyProxy, etc. ) Each site has a supporter account in the country group where the site belongs: i.e. for GR there are: GR01AUTH, GR02UoM, GR03HEPNTUA, GR04FORTHICS, GR05DEMOKRITOS, HG01GRNET supporter accounts Each site account is registered with the mailing list as site contact Helpdesk administration is distributed each country has one admin managing the user registration process Generic GGUS support group is used for the interface: members should manage the workflow of tickets coming from GGUS reassign them to the right support group and supporter ROC Central Support group to coordinate helpdesk operations Operations coordination support group for the overall management of operations at the ROC level EGEE/LCG Operation Workshop – May 24-26, 2005 - 22

ROC SE Ticket Categories EGEE/LCG Operation Workshop – May 24-26, 2005 - 23

ROC SE Ticket Statistics EGEE/LCG Operation Workshop – May 24-26, 2005 - 24

ROC SE Helpdesk Stats Currently 22 individual supporters and 17 site accounts registered Over 80 tickets in our database Support through helpdesk is provided on a “best effort” basis Helpdesk is mostly used for operations support most tickets are trouble tickets concerning sites EGEE/LCG Operation Workshop – May 24-26, 2005 - 25

ROC Integration: Russia (thanks to Valeriy Kirichenko) EGEE/LCG Operation Workshop – May 24-26, 2005 - 26

ROC Integration: Russia 4 supporters from ITEP + 1 helpdesk admin 10.00-18.00 working hours 8x5 coverage should have an answer in 2 working days (or send to GGUS or CERN) Supporters from other Russian sites may register Full integration with helpdesks of other Russian sites within Fall EGEE/LCG Operation Workshop – May 24-26, 2005 - 27

ROC Integration: SW (slides from M. Kaci , F. Fassi, J. Salt) Links to : Home FAQ Ticket Documents Repositories Training EGEE/LCG Operation Workshop – May 24-26, 2005 - 28

ROC Integration: SW username/ password needed Powered by OneOrZero v1.4 RC2 Red Lava http://helpdesk.oneorzero.com EGEE/LCG Operation Workshop – May 24-26, 2005 - 29

ROC Integration Some ROCs had different helpdesks inside their federation: CE & NE: helpdesk based on RT open to local users since April, plan to be interfaced to GGUS by end of May, support structure and responsibilities defined within their ROC, tickets expected to be answered in a reasonable time FR: home developed helpdesk, interface to GGUS by end of May GER-CH: helpdesk based on Remedy, interface to GGUS ready by June 8th UK-I: helpdesk based on Footprint, plan to be interfaced to GGUS by end of July All ROCs will have their Support System ready and interfaced to GGUS by end of July Open issue: what about ROC_CERN and ROC_Asia/Pacific? EGEE/LCG Operation Workshop – May 24-26, 2005 - 30

ROC Integration: GER/CH (slide from Sven Hermann) ROC User Support GER/CH based on web application similar to GGUS 1:1 ticket exchange with GGUS implemented portal currently tested; going into operation in June'05 ROC Operations Support GER/CH Handle tickets created in GGUS Support group changes every two weeks 2-3 people per RC involved Mo – Fr, 9:00 – 17:00 about 15 tickets/month FZK 06/06/2005 - 19/06/2005 23/24 62/63 DESY 23/05/2005 - 05/06/2005 21/22 60/61 GSI 09/05/2005 - 22/05/2005 19/20 58/59 FhG 25/04/2005 - 08/05/2005 17/18 56/57 11/04/2005 - 24/04/2005 15/16 54/55 On Duty Site / Contact Date Calendar Week Project Week EGEE/LCG Operation Workshop – May 24-26, 2005 - 31

ROC Integration: some numbers Even if most ROC helpdesks are not interfaced to GGUS yet, ROC supports units are reached with mailing lists: ROC # tickets # open oldest CE 29 2 1 day France 3 1 month GER-CH 33 Italy 54 5 days NE 10 5 Russia 13 2 months SE 36 4 SW 31 12 UK-I 58 15 TOTAL 309 60 Statistic available since half March More than 90% coming from CIC-on-duty CIC-on-duty rate: ~ # 50/week 1st Level rates: GGUS ~ # 20/week SOD ~ # 4/week EGEE/LCG Operation Workshop – May 24-26, 2005 - 32

Some Issues Distributed EGEE User/Operation Support Infrastructure is progressing, but: tickets must be solved within an acceptable timeframe, otherwise we’ll not attract users simply forwarding to ROCs may delay solution increase ROC experts participation to 1st Level Support / SOD might help: most of people at ROCs involved in deploying / troubleshooting the Grid can more easily solve tickets at once real responsive people must be placed behind the Support Units looking into user tickets is time consuming, but resources at ROCs now are mainly busy with Operations they can handle 1-2 tickets/day but not 50 tickets/day ROC resources needs to be re-allocated / re-organized / enhanced / committed to User Support Workflows, Monitoring, Reporting, Escalation procedures … see Alistair’s talk about Service Operation Challenge (SOC) Integration effort useless if at the end we are not able to provide a reliable service EGEE/LCG Operation Workshop – May 24-26, 2005 - 33