Heavy Ion Physics Program of CMS Proposal for Offline Computing Charles F. Maguire Vanderbilt University for the US CMS HI Collaboration April 25, 2009 CMS-HI Meeting at Santa Fe

CMS-HI Meeting at Santa Fe Outline Impact of CMS-HI Research Plan CMS-HI Compute Model Guiding principles Actual implementation Computing Requirements (within a known budget constraint) Wide area networking Compute power and local area networking Disk and tape storage Capital Cost and Personnel Summary Offline Organization, Operations, and Oversight April 25, 2009 CMS-HI Meeting at Santa Fe

Impact of the CMS-HI Research Plan CMS-HI Research Program Goals Focus is on the unique advantages of the CMS detector for detecting high pT jets, Z0 bosons, quarkonia, D and B mesons Early studies at low luminosity will concentrate on establishing the global properties of heavy ion central collisions at the LHC Later high luminosity runs with sophisticated high level triggering will allow for in-depth rare probe investigations of strongly interacting matter Projected luminosity growth Only one HI run is known for certain, at the end of 2010 After 2010 run LHC may shut down for an extended period Conditioning work needed to achieve design beam energy This proposal assumes a simple 3-year luminosity growth model Computer hardware resource acquisition is tailored to the model April 25, 2009 CMS-HI Meeting at Santa Fe

Impact of the CMS-HI Research Plan Projected HI Luminosity and Data Acquistion for the LHC 2010-2012 CMS-HI Run Ave. L (cm-2s-1) Uptime (s) Events taken Raw data (TB) 2010 (FY11) 2.5 x 1025 105 1.0 x 107 22 2011 (FY12) 2.5 x 1026 5 x 105 2.5 x 107 110 2012 (FY13) 5.0 x 1026 106 5.0 x 107 225 Caveats 1) First year running may achieve greater luminosity and uptime resulting in factors of 2 or 3 more events taken than assumed here 2) Second year running may not occur in 2011, but may shift to 2012 3) Third year running is the planned “nominal” year case when the CMS DAQ writes at the design 225 MB/s for the planned 106 s of HI running April 25, 2009 CMS-HI Meeting at Santa Fe

CMS-HI Meeting at Santa Fe CMS-HI Compute Model CMS-HI Compute Model Guiding Principles CMS-HI computing will follow, as much as feasible, the existing design and framework of CMS computing TDR (2005) CMS-HI community is much too small to embark on independent software development outside the mainstream of rest of CMS Size of CMS-HI community also mandates that we adapt the CMS multi-tiered computing grid to be optimum for our work CMS-HI Compute Model Implementation Raw data is transferred to and tape-archived at Vanderbilt site Reconstruction passes will also be done at Vanderbilt site Some reconstruction output will be copied to overseas sites (Moscow, Paris, Budapest, Seoul) as is practical Analysis passes will be done by all CMS-HI institutions using CMS’s remote job batch submission system (CRAB) Simulation production and support will be done at MIT site April 25, 2009 CMS-HI Meeting at Santa Fe

Computing Requirements: Wide Area Networking Nominal Year Running Specifications for HI CMS DAQ writes at 225 MB/s for 106 s = 225 TB Calibration and fast reconstruction at Tier0 = 75 TB Total nominal year data transfer from Tier0 = 300 TB Note: DAQ may be able to write faster eventually Nominal Year Raw Data Transport Scenario Tier0 holds raw data briefly (days): calibrations, preliminary reco Data are written to Tier0 tape archive, not designed for re-reads Above mandates a continuous transfer of data to a remote site 300 TB x 8 bits/Byte / (30 days x 24 hours/day x 3600 sec/hour) = 0.93 Gbps DC rate (no outages) Same rate calculation as for pp data except pp runs for ~5 month A safety margin must be provided for outages, e.g. 4 Gbps burst April 25, 2009 CMS-HI Meeting at Santa Fe

Computing Requirements: Wide Area Networking Nominal Year Raw Data Transport Scenario Tier0 criteria mandate a continuous transfer of data to a remote site 300 TB x 8 bits/Byte / (30 days x 24 hours/day x 3600 sec/hour) = 0.93 Gbps DC rate (no outages) A safety margin must be provided for outages, e.g. 4 Gbps burst Plan is to transfer this raw data to the Vanderbilt tape archive site Raw Data Transport Network Proposal for CMS-HI CMS-HEP and ATLAS will use USLHCNet to FNAL and BNL FNAL estimates that CMS-HI traffic will be ~2% of all USLHCNet Propose to use USLHCNet with modest pro-rated cost to DOE-NP Have explored Internet2 alternatives to the use of USLHCNet CMS management strongly discourages use of a new raw data path A new raw data path would have to be solely supported by CMS-HI It is not obvious that there would be any cost savings to DOE-NP April 25, 2009 CMS-HI Meeting at Santa Fe

Computing Requirements: Annual Compute Power Budget for All Tasks Annual Compute Tasks for Available Compute Power Reconstruction passes (CMS standard is 2) Analysis passes (scaled to take 50% of reconstruction time) Simulation production and analysis (takes 50% of reco time) Simulation event samples at 5% of real event totals (RHIC experience) Simulation event generation and processing takes 10x that of real event Constraint: Accomplish All Processing in One Year Offline processing must keep up with the annual data streams Would like to process all the data within one year, on average Essential to have analysis guidance for future running Computer Power 12 Month Allocation A single, complete reconstruction pass will take 4 months This proposal will allow for 1.5 reconstruction passes = 6 months Analysis passes of these reconstruction passes in 3 months Simulation production and analysis in 3 months April 25, 2009 CMS-HI Meeting at Santa Fe

Computing Requirements: Compute Time for Single Reconstruction Pass Nominal year DAQ HI Bandwidth Partition and Associated CPU Times Channel BW (MB/s) Size/Evt (MB) Rate (Hz) CPU time/evt (s) Annual total events Annual CPU time (s) Min Bias 33.75 2.5 13.5 100 1.35 x 107 1.35 x 109 Jet-100 24.75 5.8 4.27 450 4.27 x 106 1.92 x 109 Jet-75 27.00 5.7 4.74 4.74 x 106 2.13 x 109 Jet-50 27.50 5.4 5.00 5.00 x 106 2.25 x 109 J/y 67.50 4.9 13.78 1000 1.38 x 107 1.38 x 1010 Y 2.25 0.46 4.59 x 105 4.59 x 108 e-g-10 40.50 6.98 6.98 x 106 3.14 x 109 Ultra-per 1.0 2.25 x 106 1.25 x 109 Sum 225 51 x 106 Evts/yr 2.53 x 1010 seconds/yr CPU times from the jet-g simulation study, and scaled to a 1600 SpecInt2000 processor April 25, 2009 CMS-HI Meeting at Santa Fe

Computing Requirements: Net Compute Power Requirement Determination of the Total CPU Number Ncpu A complete reconstruction pass takes 25 x 109 seconds (scaled to a 1600 SpecInt2000 processor) A single reconstruction pass must be completed in 4 months Assume that there is an effective duty cycle of 80% The US CMS-HI Compute Center is Set for 3,000 CPUs Real data reconstruction and analyses will consume 75% of the annual integrated compute power, i.e. ~2,250 CPUs Simulation production and analyses will consume 25% of the annual integrated compute power, i.e. ~750 CPUs A satellite simulation center is proposed at the MIT Tier2 CMS faciltiy, taking advantage of that local support and opportunistic cycles April 25, 2009 CMS-HI Meeting at Santa Fe

Computing Requirements: Annual Allocations of CMS-HI Compute Power Distribution of Compute Power For 3,000 CPUs Processing Task Duration (months) Total CPU Use (CPU-months) Partial reconstruction (50% of events) 2 6,000 Partial analysis pass (50% of events) 1 3,000 Complete reconstruction pass 4 12,000 Complete analysis pass Simulation generation and analysis 3 9,000 Total 12 36,000 Simulation generation and analysis can be done asynchronously with real data analysis, i.e. 750 CPUs x 12 months = 9,000 CPU-months Comparison of CMS-HI Compute Center with CMS-HEP Compute Centers 1) CMS-HI with 3000 CPUs @ 1600 SpecInt2000 = 4.8 MSpecInt2000 2) CMS-HEP with 7 Tier1 and 36 Tier2 had a quota of 49.2 MSpecInt2000 (from TDR-2005) 3) 10% relative size scales with 10% of running and raw data output April 25, 2009 CMS-HI Meeting at Santa Fe

Computing Requirements: Local Disk Storage Considerations Disk Storage Categories (there is never enough disk space) Raw data buffer for transfer from Tier0, staging to tape archive Reconstruction (RECO) output Analysis Object Data (AOD) output Physics Analysis Group (PAG) output Simulation production (MC) Each category class assigned a relative scale factor Disk Storage Acquisition Timelines Model dependent according to luminosity growth Need to minimize tape re-reads, most "popular" files kept on disk Model dependent according to pace of publication RHIC experience: deadlines by which older data are removed April 25, 2009 CMS-HI Meeting at Santa Fe

Computing Requirements: Local Disk Storage Annual Allocations Annual Disk Storage Allocated to Real Data Reconstruction and Analysis FY Events (Millions) Raw Data Buffer (TB) RECO (TB) AOD (TB) PAG (TB) Total (TB) 2010 20 2011 10 50 30 6 3 89 2012 25 75 21 11 157 2013 51 153 46 23 271 2014 55 44 302 Assumptions for the Allocation Growth of Real Data Needs 1) In 2010 there is no real data but a 20 TB buffer is allocated for transfer testing All other disk storage needs are met out of simulation allocations on next page 2) Event growth in 2011-2013 as per model on page 3 3) Relative sizes of RECO, AOD, and PAG categories according to present experience April 25, 2009 CMS-HI Meeting at Santa Fe

Computing Requirements: Local Disk Storage Annual Allocations Annual Disk Storage Allocated to Simulation Production and Analysis FY Events (Millions) Event Size (TB) Raw (TB) RECO (TB) AOD PAG (TB) Total 2010 0.5 15 3.0 1.0 0.2 0.1 19 2011 30 5.9 2.0 0.4 38 2012 1.25 37 7.4 2.5 0.7 48 2013 2.55 75 5.1 1.3 97 2014 1.6 0.8 98 Assumptions for the Allocation Growth of Simulation Needs 1) GEANT (MC) event size is taken to be five times a real event size 2) MC event number scales as 5% of real event number in 2012 and beyond 3) Relative sizes of RECO, AOD, and PAG categories according to present experience 4) Total real data + MC storage after 5 years = 400 TB April 25, 2009 CMS-HI Meeting at Santa Fe

Computing Requirements: Local Disk Storage Allocation Overview Storage Allocation Comparisons and Risk Analysis CMS-HI proposes to have 400 TB after 5 years when steady state data production is 300 TB per year CMS-HEP TDR-2005 proposed 15.6 PB combined Tier1 + Tier2 Ratio is 0.026 for disk storage as compared with 0.10 for CPUs PHENIX had 800 TB disk storage at RCF in 2007 when 600 TB of raw data was written, i.e. same ratio as proposed by CMS-HI Very painful for allocating disk space among users and production PHENIX has access to significant disk space at remote sites (CCJ, CCF, Vanderbilt, ...) for real data production and simulations CMS-HI is not counting on signifcant disk space at other US sites Serious Concern of Insufficient Disk Storage Allocation May need to change balance of CPUs/Disk/Tape in outer years Possibility of significant disk storage overseas (network rates?) May have access to other disk resources (REDDNet from NSF) April 25, 2009 CMS-HI Meeting at Santa Fe

Computing Requirements: Tape Storage Allocations Tape Storage Categories and Annual Allocations Tape storage largely follows annual data production statistics First priority is archiving securely the raw data for analysis Access to tape will be limited to archiving and production teams Guidance from experiences at RHIC's HPSS Unfettered, unsynchronized read requests to tape drives leads to strangulation FY Real Evts. (Millions) Raw (TB) RECO (TB) AOD PAG (TB) Real (TB) MC (TB) Real+MC (TB) D (TB) 2010 0.0 39 2011 10 60 30 6 3 99 34 172 133 2012 25 150 75 15 7.5 248 48 467 295 2013 51 300 115 23 11.5 449 97 1013 546 2014 1559 April 25, 2009 CMS-HI Meeting at Santa Fe

Capital Cost and Personnel Summary Real Data Processing and Tape Archiving Center at Vanderbilt Category FY10 FY11 FY12 FY13 FY14 Total CPU cores 440 456 480 360 288 2024 Total CPUs 896 1376 1736 Disk (TB) 20 70 110 30 300 Total Disk 90 160 270 Tape (TB) 40 130 540 560 1570 Total Tape 170 470 1010 CPU cost $151,800 $125,400 $120,000 $81,000 $64,800 $543,000 Disk cost $10,000 $31,500 $28,000 $38,500 $9,000 $117,000 Tape cost $66,500 $23,085 $58,449 $60,720 $61,650 $270,404 Total hard. $228,300 $179,985 $206,449 $180,220 $135,450 $930,404 Staff (DOE cost) $119,534 $155,394 $161,609 $201,688 $244,715 $882,940 Total cost $347,834 $335,379 $368,059 $381,908 $380,165 $1,813,344 April 25, 2009 CMS-HI Meeting at Santa Fe

Capital Cost and Personnel Summary Simulation Production and Analysis Center at MIT Category FY10 FY11 FY12 FY13 FY14 Total CPU cores 152 160 120 96 680 Total CPUs 304 464 584 Disk (TB) 20 10 50 100 Total Disk 40 CPU cost $52,440 $41,800 $40,000 $27,000 $21,600 $182,000 Disk cost $10,000 $9,000 $4,000 $17,500 $0 $40,500 Total hard. $62,440 $50,800 $44,000 $44,500 $222,500 Staff (DOE cost) $37,040 $37,280 $39,400 $40,600 $197,120 Total cost $99,480 $89,080 $83,400 $85,100 $63,400 $420,460 MIT group experience providing simulation production for all CMS-HI institutions April 25, 2009 CMS-HI Meeting at Santa Fe

Capital Cost and Personnel Summary Cumulative Cost Comparisons Category FY10 FY11 FY12 FY13 FY14 Total Real data center at Vanderbilt $347,834 $335,379 $368,059 $381,908 $380,165 $1,813,344 Simulation center at MIT $99,480 $89,080 $83,400 $85,100 $63,400 $420,460 Vanderbilt + MIT Total $447,314 $424,459 $451,459 $467,008 $443,565 $2,249,560 Single center at Vanderbilt $440,157 $417,257 $440,380 $460,023 $436,725 $2,198,452 The small cost difference between having a single center at Vanderbilt and two separate centers at Vanderbilt and MIT is due to the slight difference in FTE charges at the two institutions. April 25, 2009 CMS-HI Meeting at Santa Fe

Organization, Operations, and Oversight Impact of CMS-HI Research Plan CMS-HI Compute Model Integration into existing CMS compute model Service role to the total CMS-HI community Computing Requirements Wide area networking Compute power and local area networking Disk and tape storage Capital Cost and Personnel Summary Offline Organization, Operations, and Oversight April 25, 2009 CMS-HI Meeting at Santa Fe