Mu2e FY15 and FY16 Computing Needs Rob Kutschke Scientific Computing Portfolio Management Team (SC-PMT) Review 4 March 2015

Scientific Goals for FY15 and FY16 Obtain CD3c and begin construction Planning date for DOE CD3c review is March 2016 Implies computing done by November 15, 2015 Big questions for computing: Is the neutron flux on the CRV system low enough to allow the scintillator based design to work? Are the weak spots in the CRV system acceptable? One iteration of the full simulation, with the most recent design, to obtain resolutions, sensitivities and backgrounds, all with systematic errors. At least the same size as the TDR sample. After CD3c Execute the plan: detector commissioning FY20 Computing: final optiization, calibration, algorithm improvement 3/4/15Kutschke | Mu2e FY15/FY16 Computing Needs2

Large Scale or out of ordinary computing needed to complete these goals - 1 Before Oct 1, 2015 we need about 14 million CPU hours –Need to get much of this opportunistically (including offsite) Already testing offsite running Thanks to the SCD team working with us. –Much of the work is low IO and is well suited for offsite. We will need about 210 TB of tape to hold all of the files generated by the above –We have an option to reduce this substantially Not clear that we have the people to implement it in a timely fashion. –Many files will use small file aggregation –Existing bluearc data disk (82 TB) OK. 3/4/15Kutschke | Mu2e FY15/FY16 Computing Needs3

Large Scale or out of ordinary computing needed to complete these goals - 2 Making premixed background events: –To do complete events we need a CPU with 8GB memory/core –Proof of principle on FermiCloud We found it awkward to use. Advertised to be easier with jobsub_client – not yet tested. Expect that jobs needing large memory/core will become more frequent as the experiment proceeds: –It will remain a minor use case for a long time to come. –But convenience has value 3/4/15Kutschke | Mu2e FY15/FY16 Computing Needs4

Did you meet your FY14 Scientific Goals? Yes In Feb Mu2e received a positive recommendation for CD2/3b Expect signatures on CD2/3b document today! Computing was NOT a rate limiting factor 100E9 events for CRV deadtime study using G4beamline 3E9 events with full simulation to study physics sensitiviity 3/4/15Kutschke | Mu2e FY15/FY16 Computing Needs5

Needs from Service Areas (1) Grid and Cloud Computing –To end of FY15 we need 14 M-hours to be ready for CD3c –Exisiting allocation integrates to 4.3 M-hours To achieve this we will need significant offsite running. Peak vs integral: if we can get it faster, that would be great. –Using jobsub_client; testing offsite OSG running Networked Storage –Existing allocation is OK for work leading to CD3c. –Would like finer control of qutotas on bluearc Physics and Detector Simulation –Geant4, CRY both needed for CD3c 3/4/15Kutschke | Mu2e FY15/FY16 Computing Needs6

Needs from Service Areas (2) Scientific Data Storage and Access –Enstore: estimate incremental tape usage: 2015: 210 TB - for CD3c 2016: 200 TB - mock data/alignment challenges 2017: 300 TB - mock data/alignment challenges –dCache – little operational experience, just a guess. The following are integral: 2015: 50 TB R/W and 50 TB scratch 2016: same 2017: 100 TB R/W and 50 TB scratch 3/4/15Kutschke | Mu2e FY15/FY16 Computing Needs7

Needs from Service Areas (3) CVMFS –Just commisioned it. Soon to be a critical service. Scientific Data Management –SAM and SAM Web: critical –FTS Just getting experience with it Have dumped 20 TB from bluearc IFDH, GRIDFTP –Heavy use of both IFDH_ART –Will soon test running art using SAM for input and output –We have use cases for which this will not work. We know how to script them using IFDH 3/4/15Kutschke | Mu2e FY15/FY16 Computing Needs8

Needs from Service Areas (4) Scientific Frameworks –art: critical Interactive GPCF –5 nodes are enough for now and the next few years Experiment control room –N/A for 2015/16. First commissioning data in 2020 Jenkins –Recently commissioned: becoming critical –Continuous integration / nightly build/ release builds –Expect to increase testing in all 3 modes. 3/4/15Kutschke | Mu2e FY15/FY16 Computing Needs9

Needs from Service Areas (5) Central Web Hosting –Mu2e homepage and web site: critical DAQ and Engineering –artdaq: it’s in our baseline –Kurt Biery, Mark Bowden and their groups are on project. Document Management –DocDB: critical redmine –Critical for git repos, wiki, issue tracker cdcvs –Some of Mu2e still use cvs hosted on cvs for their work svn –No lab hosted repos but we access an svn repo at LBL 3/4/15Kutschke | Mu2e FY15/FY16 Computing Needs10

Needs from Service Areas (6) ECL –In use for offsite test beam work –Expect it to grow exponentially during construction UPS/UPD –Critical Readytalk –Critical xrootd –Not yet but we know it’s there. 3/4/15Kutschke | Mu2e FY15/FY16 Computing Needs11

Needs from Service Areas (7) Construction databases –Discussions started with Igor. –Mu2e point person is Kevin Lynch (CUNY) –Plan is to build from the NOvA experience –Mu2e has some in-house people for GUI development but I don’t yet know if we have enough. –Need to be ready for start of construction in FY2016 Conditions database –Guess we will want to start discussions with Igor in late 2016 or early –Very likely can start with work done for other experiments. 3/4/15Kutschke | Mu2e FY15/FY16 Computing Needs12

Needs from Service Areas (8) Consulting –G4: critical, mostly Krzysztof Genser –art: critical, concerned that the art team may be shrinking Production Operators –They have been valuable and plan to use them for the simulation campaign this summer. Discussion Forum –Hypernews Up and running in a sandbox Waiting for Shibboleth to be installed so that we can open it up for offsite people Then testing starts for real. Thanks for setting this up. 3/4/15Kutschke | Mu2e FY15/FY16 Computing Needs13

TSW/EOP Status and Plans Are your TSWs signed and up to date? –To the best of my knowledge we have no TSWs or EOPs. If not, do they need revision? –According to Program Planning, these need be in draft form by the CD3c review (est. March 2016). –They do not need to be signed until CD4 ( Q1 FY21) 3/4/15Kutschke | Mu2e FY15/FY16 Computing Needs14

Future Directions (Challenges and R&D) Will your SOPs change significantly in the future (new phase of the experiment, new running conditions, etc.)? – focus of computing effort will switch to calibrations, alignment, refinement of reco algorithms. –Some simulations will continue. Are future R&D projects in the pipeline? –None that I know of. Are additional resources going to be required to meet them? –No 3/4/15Kutschke | Mu2e FY15/FY16 Computing Needs15

Additional Projects/Comments (1) In 2015 Mu2e Project contributed 0.5 FTE of funding to CS –Will be 0 for 2016 and beyond –But we still need work done by CS and we do not yet have an operations budget. We need to learn how this works. Event display help –We have several event displays –Each does a good for a narrow purpose –It would be great help to have access to a graphics consultant to integrate and improve these. We need a good general programmer/scripter –Not available from the collaboration except for people who are more valuable elsewhere (too far from data). 3/4/15Kutschke | Mu2e FY15/FY16 Computing Needs16

Additional Projects/Comments (2) SCD documentation is aimed at experts and is generally very good. –We greatly appreciate it. –It would be much easier to onboard new people there were more material aimed at beginners. –For example: What is a grid? How does it work? 3/4/15Kutschke | Mu2e FY15/FY16 Computing Needs17

Schedule 2/6/15Kutschke | Introduction18 FY15 FY16 FY17 FY18 FY19 FY20 FY21 FY22 CD-2/3b Project Complete Detector Construction Accelerator and Beamline Construction Solenoid Infrastructure KPPs Satisfied PS Fabrication and QA PS/DS Final Design Accelerator Commissioning (off Project) CD-3c Fabricate and QA Superconductor Detector Hall Construction Cosmic Ray System Test 24 months of schedule float PO issued for TS Module Fabrication Fabricate and QA TS Modules, Assemble TS DS Fabrication and QA PS Installation TS Installation DS Installation Critical Path Solenoid power/cryo hookup Solenoid Commissioning