Computing Overview Topics here: CSA lessons (briefly) PADA DESY CMS meeting 20.12.07 Topics here: CSA lessons (briefly) PADA CCRC’08 = CSA08 Matthias Kasemann December 2007 December 20, 2007 DESY CMS meeting

CSA07 Goals Test and validate the components the CMS Computing Model in a simultaneous exercise the Tier-0, Tier-1 and Tier-2 workflows Test the CMS software: particularly the reconstruction and HLT packages Test the CMS production systems at 50% scale of expected 2008 operation workflow management, data management, facilities, transfers Test the computing facilities and mass storage systems. Demonstrate that data will transfer between production and analysis sites in a timely way. Test the Alignment and Calibration stream (AlcaReco) Produce, deliver and store AODs + skims for analysis by physics groups December 20, 2007 DESY CMS meeting

Prompt Reconstruction CSA07 Workflows Prompt Reconstruction CASTOR HLT TIER-0 CAF Calibration Express-Stream Analysis 300MB/s Re-Reco Skims TIER-1 TIER-1 TIER-1 TIER-1 20-200MB/s ~10MB/s Simulation Analysis TIER-2 TIER-2 TIER-2 TIER-2 December 20, 2007 DESY CMS meeting

Preparing for the CSA07 (Jul-Sep) CMSSW - software releases organized by offline team Releases are tested by data operations teams Distributed and installed to the sites (This is not an easy process.) Steps for preparing data for physics (pre-challenge workflows): Generation and Simulation with Geant4 (at the Tier-2 centers) Digitization Digi2RAW - format change to look like data input to HLT HLT processing Data are split into 7 Primary Data Sets (PDS) based on the HLT information This was a big addition in CSA07. The data samples more accurately reflect what will come from the detector, but are harder to produce. December 20, 2007 DESY CMS meeting

Preparing for the CSA07 (Jul-Sep) Planned Workflows for the Challenge: Reconstruction - HLT + RECO output (~1 MB) AOD production - (200 kB) Skims for physics analysis at the Tier-1 centers Re-Reco (and redoing AOD production/skims) at the Tier-1 centers Analysis at the Tier-2 centers Lessons from CSA07 preparations: It turned out that there was insufficient time for testing the components since some of the components were coming at the latest moment.. CSA08: We have to devote more time for testing the components December 20, 2007 DESY CMS meeting

December 20, 2007 DESY CMS meeting

MC Production summary Substantial (… more resources used) December 20, 2007 DESY CMS meeting

CSA07 Issues and Lessons There are clearly areas that are going to need development Need to work on the CMSSW application Reduce the number of workflows (in 170, 180 and 200) Reduce the memory footprint to increase the number of events we can run and increase the available resources Goal: CMSSW applications should stay in 1 GB memory Several area should be improved Access and manipulation of IOV constant (over Xmas) HLT data model (on going) New huge increase in memory seen in 170 to be address immediately (mainly in DPG’s code) December 20, 2007 DESY CMS meeting

CSA07 Issues and Lessons Increase the speed of IO on mass storage test using new ROOT version Improve our testing and validation procedures for the applications and workflows. Reduce event size RAW/DIGI and RECO size AOD size Mini-workshop with physics and DPG Groups on Jan 28/29 (CERN) Two task forces has been created in order to prepare this workshop RECO Task Force chair (Shahram Rahatlou) Analysis Task Force chair (Roberto Tenchini) FW support for handing of RAW, RECO versus FEVT (This is foreseen for version 2_0_0) December 20, 2007 DESY CMS meeting

CSA07 Issues and Lessons Need to work on the CMS Tools Augment the Production tools to be able to better handle continuous operations Roll back to known good points. Modify workflows more simply Increase the speed of Bookkeeping System under specific load conditions Optimize the data transfers in PhEDEx for data availability Improve the analysis Tool (CRAB) Planning a Workshop in January 21-25 2008 (Lyon) http://indico.cern.ch/conferenceTimeTable.py?confId=24877 Goals: Review Data and Workload management components Improvement integration (communication) between operation and development teams Will include also Tier0 components Define work plan for 2008 December 20, 2007 DESY CMS meeting

CSA07 Issues and Lessons Facility Lessons: We learned a lot about operating Castor and dCache under load Need to improve the rate of file opens Need to decrease the rate of errors. Need to improve the scalability of some components Need to work on the stability of services at CERN and Tier-1 centers Need to work on the transfer quality when the farms are under heavy processing load General lessons: Much work is needed to achieve simultaneous, sustainable and stable operations December 20, 2007 DESY CMS meeting

PADA: processing and data access taskforce Draft Mandate: Integrate developments and services to bring our centers and services to production quality for processing and analysis The Processing And Data Access Task Force is an initiative in the Integration Program Designed to transition services developed in Offline to Operations Elements of integration and testing for Production, Analysis, and Data Management tools Designed to ensure services and sites used in operations are production quality Elements in the commissioning program for links and sites Verify that items identified in the CSA07 are solved Development work is primarily in offline, but verification in Integration Plan is: To build on the expertise of the distributed MC production teams, extend scope We need the expertise in proximity of the centers to help us here For 2008 we want to make this a recognized service contribution in the MoA scheme, Initial time frame: 1 year until we have seen the first data We need to define steps, milestones, recruit people, hope for MC-OPS, DDT, .... December 20, 2007 DESY CMS meeting

PADA tasks + schedule December 20, 2007 DESY CMS meeting

Final Check before Data taking starts: CCRC’08 = CSA08CMS A combined challenge by all Experiments must be used to demonstrate the readiness of the WLCG Computing infrastructure before start of data taking at a scale comparable to the data taking in 2008. CMS fully supports the plan, to execute this CCRC in two phases: a set of functional tests in February 2008 the final challenge in May 2008 at 100% scale, starting with the readout of the experiment We must do this challenge as WLCG collaboration: Centers and Experiments together Combined planning has started: Mailing list created: wlcg-ccrc08@cern.ch Agenda pages: Phone conference every Monday afternoon Monthly session in pre-GDB meeting December 20, 2007 DESY CMS meeting

CMS CCRC’08 Schedule Phase 1 - February 2008: (proposed: 4-29.2.2008) Possible scenario: blocks of functional tests, Try to reach 2008 scale for tests at… Phase 2: - May 2008: (proposed: 5-30.5.2008) Full workflows at all centers executed simultaneously by all 4 LHC experiments Use data from cosmics data run, add artificial load to reach 100% Duration of challenge: 1 week setup, 4 weeks challenge December 20, 2007 DESY CMS meeting

1) Detector Installation, Commissioning & Operation 2) Preparation of Software, Computing & Physics Analysis Aug Sep Oct Nov Dec Jan Feb Mar Apr May S/w Release 1_6 (CSA07) V36 Schedule (Nov’07) CSA07 Cooldown of Magnet: Test S/w Release 1_7 (CCR_0T, HLT Validation) Tracker Insertion 2007 Physics Analyses First Results Out CMS Cosmic Run CCR_0T Several short periods Dec-Mar) Last Heavy Element Lowered Test Magnet at low current S/w Release 1_8 (Lessons of ‘07) Functional Tests CSA08 (CCRC) Beam-pipe Closed and Baked-out S/w Release 2_0 (CCR_4T, Production startup MC samples) 1 EE endcap Installed, Pixels installed MC Production for Startup Cosmic Run CCR_4T CSA08 (CCRC) Combined Computing Readiness Challenge Master Contingency 2nd ECAL Endcap Ready for Installation end Jun’08 December 20, 2007 DESY CMS meeting

CCRC’08 Phase 1: February 2008 Goals for CMS: Verify solutions to CSA07 issues and lessons - don’t repeat CSA07 where solution is not ready Attempt to reach ‘08 scale on individual tests - don’t repeat CSA07 where no increase in scale possible Guiding principles: CCRC’08-Phase 1 will be a Computing&Software challenge no coupling of other deliverables to CCRC’08/1 tests Cosmics run, MC production and Physics Analysis have priority and cannot be interrupted by CCRC’08 tests for long We defined blocks of tests, which stress a specific service or workflow Tests should be as independent from each other as possible Tests should be done in parallel where possible Individual test is considered successful if sustained for (3-5) days Where full ‘08 scale is not possible (hardware) tests are scaled down to hardware limit December 20, 2007 DESY CMS meeting

Status of preparation Need to provide requirements to sites Draft plan of of tests available Goals of tests agreed by computing Area or responsibility and coordination is defined Tests are to be coordinated and agreed within CMS Many tests are to be scheduled together with other VO’s (ATLAS, …) to reach scale of stress test CCRC’08 meeting next Thursday, 20.12, 16:00 Plan details of tests Propose metrics Define constraints and specify resources required Propose schedule of test Need to provide requirements to sites December 20, 2007 DESY CMS meeting

Planned blocks of tests Data recording at CERN 1a) readout from P5, use HLT, w. stream definition, use Storage Manager, transfer to T0, perform repacking, write to CASTOR Goal: verify dataflow for CMS 1b) CASTOR data archiving test Goal: verify CASTOR performance at full CMS and ATLAS rate 2. Processing at T0 at high rate Goal: verify T0 performance under CMS + ATLAS load 3. CERN data export to T1 at full / high rate Goal: verify CERN export and T1 import performance under CMS + ATLAS load 4. T1 data handling and processing Goal: verify full CMS T1 re-processing workflow in presence of ATLAS load at T1’s December 20, 2007 DESY CMS meeting

Planned blocks of tests 5 Data Transfer performance tests Goal: verify T1/T2 export + import performance under CMS +ATLAS load 5.1 T1 - T1 data transfer at real rates 5.2 T1 - T2 data transfer at real rates 5.3 T2 - T1 data transfer at real rates Data Transfer tests (5.1-3) should be done individually and then together Monte Carlo Production and Analysis at Tier-2’s Goal: Verify Pile-up and FastSim MC Production Goal: Scale tests of Analysis Jobs Data transfer request system tests Goal: verify Data Transfer and SE’s at Tier-2 centers 8. CAF tests: (depending on CAF infrastructure schedule) Goal: verify basics CMS use cases at scale December 20, 2007 DESY CMS meeting

Summary (1/2) In CSA07 a lot was learned and a lot was achieved.. We hit most of metrics - but separately and intermittently Several steps accomplished simultaneously Many workflow steps hit metric routinely Now work on accomplishing all steps simultaneously, and providing stability in a sustainable way. Global connectivity between T1-T2 sites is still an important issue. The DDT task force has been successful in increasing the # of working links. This effort must continue and work must be done to automate the process of testing/commissioning the links. We still have to increase the number of people involved in facilities, commissioning and operations. Some recent actions: New (2nd) L2 appointed to lead facility operations (based at CERN) New Production And Data Access (PADA) Task Force starting - will include some of the people from DDT task force and MC production teams. December 20, 2007 DESY CMS meeting

Summary (2/2) ~ 200M Events processed and re-processed Calibration, MC production, Reconstruction, skimming, merging all tested successfully. Still need time to test the analysis model. CSA07 Goals for providing data for physics will be accomplished … albeit delayed due to schedule slips Processing continues to complete the data samples for physics and detector studies. We are keeping the challenge infrastructure alive and trying to keep it stable, going forward... Continue to support global detector commissioning and physics studies. We have to prepare for the ‘Combined Computing Readiness Challenge’, CCRC’08  CSA08 Without testing the software and infrastructure we are not prepared… December 20, 2007 DESY CMS meeting