Graeme Stewart: ATLAS Computing WLCG Workshop, Prague ATLAS Suspension and Downtime Procedures Graeme Stewart (for ATLAS Central Operations Team)

Graeme Stewart: ATLAS Computing WLCG Workshop, Prague 2009 Production lFor ATLAS production we run shift teams who look at the state of sites nThere’s almost 24 hour coverage, but not 24 hour expert support 2 Number 1 problem site Number 2 problem site

Graeme Stewart: ATLAS Computing WLCG Workshop, Prague 2009 Offline lIf the problems are diagnosed as site issues then nAn ATLAS eLog entry is created nA GGUS ticket is sent to the site nThe site is set offline if the problems are serious enough lIf the problem is well understood and resolved quickly by the site then the site will usually be set directly online again

Graeme Stewart: ATLAS Computing WLCG Workshop, Prague 2009 Suspension and Revalidation lIf the problem is extended, not well explained or the site have done, e.g., a major upgrade, then their queues can be put into test status while they are revalidated nTest status queues can pull only test jobs from PanDA! nTest jobs are usually small event generations, but do do a full chain test: [Insert pictute of test jobs here] lIf the site runs all it’s test jobs successfully it will be set online

Graeme Stewart: ATLAS Computing WLCG Workshop, Prague 2009 Production System Functional Tests lIn addition to these very targeted site specific tests we run weekly ‘production system functional tests’ for the whole cloud nJobs are similar short evgens to the site targeted tests lThese are very useful, e.g., after a Tier-1 downtime lGood test of whole cloud, especially if there is little or no production

Graeme Stewart: ATLAS Computing WLCG Workshop, Prague 2009 User Analysis: Ganga Robot lFor user analysis jobs there is a similar ‘standard candle’ analysis job sent every day through the ganga framework lIf your site fails 3 tests in a row it’s blacklisted for ATLAS user analysis Tests of different sw releasesTests of different storage areas

Graeme Stewart: ATLAS Computing WLCG Workshop, Prague 2009 DDM Dashboard lShifters also monitor the ATLAS DDM Dashboard nThis monitors file transfer success rates across the grid

Graeme Stewart: ATLAS Computing WLCG Workshop, Prague 2009 Drill Down to Sites Scheduled Downtime

Graeme Stewart: ATLAS Computing WLCG Workshop, Prague 2009 Data Distribution Tests lATLAS also runs weekly Distributed Data Management Functional Tests These tests distribute a small amount of /dev/random data to each ATLAS site according to the ATLAS computing model nAs these run all the time then they test the system’s functionality even when there are no other activities l[Insert plots here when available…]

Graeme Stewart: ATLAS Computing WLCG Workshop, Prague 2009 Failing Site? lThe procedure is the usual one nE-log (for us) nGGUS (for site) lIf the problem would affect MC production the site will also be taken offline for production nBut often a broken SE means you can’t get input data anyway… lIf the problem is very grave then the site will be removed from the DDM site services machine and/or the subscription engine will be stopped to that cloud (e.g., T1 problems) nThis prevents any transfer attempts to that site at all nIt is a manual operation  So we don’t like to do it, because it’s easy to forget that a site/cloud was removed lAfter a period of suspension a cloud/site must succeed in DDM functional tests for 36 hours before being allowed to take ATLAS data

Graeme Stewart: ATLAS Computing WLCG Workshop, Prague 2009 How do we know about downtimes? lGoing to the GOC to see if a site is in downtime is far to slow for shifters triaging dozens of problems lThere is a feed from the GOC to an ATLAS Grid Downtime Calendar lProblems: nExtensions not shown nDowntimes can be marked for secondary services

Graeme Stewart: ATLAS Computing WLCG Workshop, Prague 2009 Communications lFrom us to you nWe primarily use GGUS tickets  Direct ticketing of sites is generally used and is much preferred by us nWe also use our cloud contacts  E.g., requests to change space token setup nAnd we have operational mailing lists which sites should sign up to  In particular: nAnd weekly operations meetings, several jamborees a year  To which sites are welcome and encouraged to come lFrom you to us nYou can use ggus tickets  But responses may be slower as the ticket needs to be routed to the correct ATLAS responsibles nPlease do use your ATLAS cloud contacts  You should know who they are! nOr ask a question  On a mailing list  In a meeting