1. INFSO-RI-508833 Enabling Grids for E-sciencE www.eu-egee.org The role of the Virtual Organization Ticket Processing Manager Guido Negri INFN – CNAF Italy

2. Enabling Grids for E-sciencE INFSO-RI-508833 G. Negri, GGUS Training for supporters, 10-11 November 2005, Karlsruhe 2 Content of this seminar Problem reporting and support workflow The VO TPM team The VO TPM duties Informations to solve tickets Handling tickets Open issues Conclusions Many things have already been said in previous talks. We will see them from the Virtual Organization support point of view

3. Enabling Grids for E-sciencE INFSO-RI-508833 G. Negri, GGUS Training for supporters, 10-11 November 2005, Karlsruhe 3 www.ggus.org -user-support@ggus.org helpdesk@ggus.org ROC 1 RC 1RC X… Local Helpdesk Problem Problem reporting Users can make a support request via their Regional Operations' Center (ROC) or theirROC Virtual Organisation (VO). Central Application (GGUS) Within GGUS there is an internal support structure for all support requests. TPM GGUS GGUS

4. Enabling Grids for E-sciencE INFSO-RI-508833 G. Negri, GGUS Training for supporters, 10-11 November 2005, Karlsruhe 4 Mail to - user- support@ggus.org Central Application (GGUS) VO Support Units Middleware Support Units Deployment Support Units Operations Support ROC Support Units Network Support VO-specific TPM Grid+VO experts - Solves - Classifies - Monitors Automatic Ticket Creation It is addressed to TPM and VO TPM Support Workflow For VO users and VO specific problems

5. Enabling Grids for E-sciencE INFSO-RI-508833 G. Negri, GGUS Training for supporters, 10-11 November 2005, Karlsruhe 5 Mail to - user- support@ggus.org Central Application (GGUS) VO Support Units Automatic Ticket Creation It is addressed to TPM and VO TPM Support Workflow For VO users and VO specific problems Some VOs have chosen to have the tickets sent to GGUS processed first by the TPM team. Then the TPM team can reassign them to VO TPM if the problem is VO specific. Some other VOs, like ATLAS, want the tickets sent to both TPM and VO TPM for processing. TPM VO TPM

6. Enabling Grids for E-sciencE INFSO-RI-508833 G. Negri, GGUS Training for supporters, 10-11 November 2005, Karlsruhe 6 Local Helpdesk Central Application (GGUS) Automatic Ticket Creation VO Support Units Middleware Support Units Deployment Support Units Operations Support ROC Support Units Network Support - Solves - Classifies - Monitors TPM or VO TPM Local Helpdesk Local Problem? RC Support Workflow

7. Enabling Grids for E-sciencE INFSO-RI-508833 G. Negri, GGUS Training for supporters, 10-11 November 2005, Karlsruhe 7 The VO Ticket Processing Managers The VO TPM: People with experience in both generic Grid problems and VO specific software Receive VO specific tickets at the same time or after the generic TPM depending on VO They have the same duties as a generic TPM If a problem is really due to VO software they use the VO support structures to solve the problem TPMSupport VO Support

8. Enabling Grids for E-sciencE INFSO-RI-508833 G. Negri, GGUS Training for supporters, 10-11 November 2005, Karlsruhe 8 The VO special support units The VO Special Support Units: This are special units of support specific to the VO. They can be specific mailing lists or Savannah VO specific ticketing systems. It is the responsibility of the VO TPM to use the VO special Support Units to solve the user’s problem. It is the responsibility of the VO TPM to feed the answer received back into the GGUS system. VO Specific Wiki pages can be filed under request of the VO TPM. The procedure is the same as for User’s Wiki pages (see TPM previous talk). Central Application (GGUS) VO Support Units It is important to receive the answer in a timely way in order not to discourage users from using the system. It is the duty of the VO TPM to stimulate supporters to provide the answer as soon as possible and to notify the user of possible delays/problems.

9. Enabling Grids for E-sciencE INFSO-RI-508833 G. Negri, GGUS Training for supporters, 10-11 November 2005, Karlsruhe 9 The VO TPM team Frederic Brochu - PH/UAT Simone Campana - IT/GD Alessandro De Salvo - PH/UAT Flavia Donno - PH/UCMbrochu@hep.phy.cam.ac.ukSimone.Campana@cern.chAlessandro.de.Salvo@cern.ch Guido Negri - Other Flavia.Donno@cern.ch Silvia Resconi - PH/UATguido.negri@cern.ch Oxana Smirnova - PH/UAT Luca Vaccarossa - Other Silvia.Resconi@cern.chOxana.Smirnova@cern.chluca.vaccarossa@mi.infn.it TPMSupport VO Support GGUS VO TPM responsible

10. Enabling Grids for E-sciencE INFSO-RI-508833 G. Negri, GGUS Training for supporters, 10-11 November 2005, Karlsruhe 10 Frederic Brochu - PH/UAT Simone Campana - IT/GD Alessandro De Salvo - PH/UAT Flavia Donno - PH/UCMbrochu@hep.phy.cam.ac.ukSimone.Campana@cern.chAlessandro.de.Salvo@cern.ch Guido Negri - Other Flavia.Donno@cern.ch Silvia Resconi - PH/UATguido.negri@cern.ch Oxana Smirnova - PH/UAT Luca Vaccarossa - Other Silvia.Resconi@cern.chOxana.Smirnova@cern.chluca.vaccarossa@mi.infn.it The VO TPM team When a ticket is assigned to the VO TPM team, the entire list of people registered as VO TPM for a specific VO receives notification of the ticket. One of the person in the group who feels to be expert for the problem reported by the user should assign the ticket to him/herself using the “Assign to” field and process the ticket as already seen. If nobody from the group picks up the ticket, then the VO TPM responsible after the first escalation e-mail should assign the ticket to him/herself and try to process it.

11. Enabling Grids for E-sciencE INFSO-RI-508833 G. Negri, GGUS Training for supporters, 10-11 November 2005, Karlsruhe 11 Who is behind a VO Team ? To find out who is behind a VO TPM team you can follow this link.

12. Enabling Grids for E-sciencE INFSO-RI-508833 G. Negri, GGUS Training for supporters, 10-11 November 2005, Karlsruhe 12 Who is behind a VO Team ? For each VO, the people belonging to the VO TPM are registered in a CERN-managed e-mail list: -grid-support@cern.ch

13. Enabling Grids for E-sciencE INFSO-RI-508833 G. Negri, GGUS Training for supporters, 10-11 November 2005, Karlsruhe 13 The Ticket Processing Managers TPMs are notified via e-mail about user tickets. TPMSupport VO Support on submit ticketon modify ticketon solve ticket submitterGGUS TPM VO-Admins if vo-specific GGUS TPM RespUnit (if assigned) RespUnit (if updated) „assign to one person“ „involve others“ TPM submitter „assign to one person“ „involve others“ Who gets email notification after which action?

14. Enabling Grids for E-sciencE INFSO-RI-508833 G. Negri, GGUS Training for supporters, 10-11 November 2005, Karlsruhe 14 The VO TPM duties What are the duties and tasks ?  A VO TPM commits to process GGUS tickets  Close simple trouble tickets  Notify users about the status of their tickets  Ensure that other tickets are sent to the correct place for processing  React to alarms that tickets have not been processed  Ensure that the knowledge base and Wiki pages are enhanced by the responses associated with tickets;  Forward problem to VO specific support mailing lists or support tools. TPMSupport VO Support

15. Enabling Grids for E-sciencE INFSO-RI-508833 G. Negri, GGUS Training for supporters, 10-11 November 2005, Karlsruhe 15 Solve simple tickets: some example An ATLAS user reports an inconsistency in ATLAS LCG File catalog, due to a premature interruption of a data management command a VO TPM assigns the ticket to himself and, after checking the inconsistency, updates the catalog (using simple data management commands) and closes the ticket, reporting what has been done in this case, a basic knowledge of the data management architecture is needed

16. Enabling Grids for E-sciencE INFSO-RI-508833 G. Negri, GGUS Training for supporters, 10-11 November 2005, Karlsruhe 16 Information to solve tickets The procedures for handling tickets are the subject of this tutorial FAQ and Documentation can be easily found in the web pages of the GGUS portal Basic documents can be found also in EGEE SA1 Documentation web page at http://egee-docs.web.cern.ch/egee- docs/list.php?dir=.\support\documentation \pdf\& and, in particular, documents 1100, 1200, 1300, 4200

17. Enabling Grids for E-sciencE INFSO-RI-508833 G. Negri, GGUS Training for supporters, 10-11 November 2005, Karlsruhe 17 Information to solve tickets: VO specific In order to handle simple tickets, LCG Middleware documentation is available on the GGUS documentation page As for ATLAS specific documentation, most of grid users requests can be faced by checking ATLAS Wiki pages at https://uimon.cern.ch/twiki/bin/view/Atlas/ WebHome or ATLAS Offline Computing web page at http://atlas.web.cern.ch/Atlas/GROUPS/S OFTWARE/OO/ VO Support Staff are also welcome to modify (when needed) the VO TPM FAQ document available at http://egee-docs.web.cern.ch/egee- docs/list.php?dir=.\support\documentation \pdf\&

18. Enabling Grids for E-sciencE INFSO-RI-508833 G. Negri, GGUS Training for supporters, 10-11 November 2005, Karlsruhe 18 The GGUS Search engine GGUSSearchEngine Ongoing work to make it faster and to search through a wide set of docs and DBs Can be used by users and supporters. Here is an example!

19. Enabling Grids for E-sciencE INFSO-RI-508833 G. Negri, GGUS Training for supporters, 10-11 November 2005, Karlsruhe 19 Notify users Users need to be notified as soon as the TPM processes the ticket Users need to be notified for tickets that take long time to be solved In case the supporter is waiting for a user reply, a user notification should be sent to remind the user to answer to the request.

20. Enabling Grids for E-sciencE INFSO-RI-508833 G. Negri, GGUS Training for supporters, 10-11 November 2005, Karlsruhe 20 Escalation tickets E-Mail escalation on doing nothing Responsible Unit Who gets email notification after which action? depending on priority of ticket less urgent - every 8 hours urgent- every 4 hours very urgent- every 3 hours top priority - every 2 hours Mo-Fr 6:00-18:00 UTC considered to adapt it to the timezone of the according responsible units.

21. Enabling Grids for E-sciencE INFSO-RI-508833 G. Negri, GGUS Training for supporters, 10-11 November 2005, Karlsruhe 21 Assigning tickets to specific supporters All supporters are behind mailing lists. These are CERN mailing lists that are generally called: -grid-support@cern.ch Please check http://ggus.pi.infn.it/supporters.html The list of supporters can be checked using SIMBA (http://www.cern.ch). You needhttp://www.cern.ch a CERN NICE account. For TPM VOs, the lists are published through the GGUS portal. Soon, the list of all supporters will be available. When the problem in Savannah is solved the user is notified (manual action at the moment  )

22. Enabling Grids for E-sciencE INFSO-RI-508833 G. Negri, GGUS Training for supporters, 10-11 November 2005, Karlsruhe 22 Assigning tickets to specific supporters When tickets present problems that depend on some specific matter that goes beyond the knowledge of the first line VO support team, then the ticket must be assigned to a specialized support team inside the VO collaboration ATLAS (and most VOs) has a generic access point for user support on specific subjects of its software or activities: it’s a Savannah portal, located at https://savannah.cern.ch/bugs/?func=additem&group=atlas-bugs If the actual nature of the problem is understood by the VO TPM, then the ticket can be addressed to the appropriate component project (the list of them all, for ATLAS Offline Computing, is at https://savannah.cern.ch/search/? words=atlas&type_of_search=soft &exact=1

23. Enabling Grids for E-sciencE INFSO-RI-508833 G. Negri, GGUS Training for supporters, 10-11 November 2005, Karlsruhe 23 Wiki pages The TPM supporters can trigger the compilation of User Guides and User FAQ Wiki pages. When the solution to a problem is of general interest and the problem can occur frequently, the TPM can ask GGUS to file Wiki Pages. TPMs should provide a very explanatory solution in the solution field together with a clear specification of the problem. It should be also specified in which Wiki page the problem should be filed. ATLAS Support Wiki page will be in place very soon (~4 days) providing documentation both for users and supporters, links to support projects and organizations and howtos

24. Enabling Grids for E-sciencE INFSO-RI-508833 G. Negri, GGUS Training for supporters, 10-11 November 2005, Karlsruhe 24 Asking for help A TPM is always assisted during normal operations. If you have technical questions concerning the Grid middleware and how to solve a problem, you can always send e-mail to tpm-grid-support@cern.ch and ask more experienced TPMs to help you with the solution of a ticket.tpm-grid-support@cern.ch If instead you have questions about procedures (“How can I assign a ticket to ROC_US?”, “The user says that he has attached the log of his job but I cannot see it. What should I do ?”, “how can I search all tickets for which a Wiki compilation has been triggered ?”, etc), then you can send e-mail to support@ggus.org.support@ggus.org +49 724 782 8383 For urgent problems (“I cannot access the GGUS portal and therefore I cannot proceed with my TPM shift”), you can call the GGUS hotline: +49 724 782 8383. You can also ask other specialized supporters (second line support) to help you understand how to correctly assign tickets, as you have seen in one of the examples given in this tutorial.

25. Enabling Grids for E-sciencE INFSO-RI-508833 G. Negri, GGUS Training for supporters, 10-11 November 2005, Karlsruhe 25 Open issues VO TPM should try to get a bit more involved in the development plans of the system, so that simple procedures for handling tickets can be clear and unambiguous Documentation and FAQs documents are clear and complete, but probably training (like this tutorial, but also e-training) could hugely simplify the organizational efforts of newcomers in the TPM teams If you have suggestions, comments, problems to report, please send e-mail to tpm-grid-support@cern.ch ANDtpm-grid-support@cern.ch support@ggus.org You can also use the GGUS/ESC (GGUS coordination project-eu-egee-sa1-esc@cern.ch body) mailing list: project-eu-egee-sa1-esc@cern.ch

26. Enabling Grids for E-sciencE INFSO-RI-508833 G. Negri, GGUS Training for supporters, 10-11 November 2005, Karlsruhe 26 The TPM Supporters TPMSupport VO Support VO TPMsVO TPMs: First line supporters for VOs Specialized VO Support Specialized VO Support: People knowledgeable about the specialized VO Software To join ATLAS TPM Team register to GGUS, then send a mail to guido.negri@cern.ch Documentation on ATLAS software ATLAS Wiki pages at https://uimon.cern.ch/twiki/bin/view/Atlas/WebHome ATLAS Offline Computing web page at http://atlas.web.cern.ch/Atlas/GROUPS/SOFTWARE/OO/

27. Enabling Grids for E-sciencE INFSO-RI-508833 G. Negri, GGUS Training for supporters, 10-11 November 2005, Karlsruhe 27 Conclusions A VO TPM team needs planned shifts in order to ba able to handle tickets in a timely manner, it needs good documentation and a careful training The system has to be properly advertised by the VO TPM team: users should be encouraged to use this single access point to support, in order to have a more controlled environment and to acquire a better knowledge of what’s working fine and of what, on the contrary, can create confusion or problems to end users More specialized supporters are welcome to enlarge the community of TPMs with some more technical knowledge