1. Computing Overview Topics here: CSA lessons (briefly) PADA
DESY CMS meeting
Topics here:
CSA lessons (briefly)
PADA
CCRC’08 = CSA08
Matthias Kasemann
December 2007
December 20, 2007
DESY CMS meeting

2. CSA07 Goals
Test and validate the components the CMS Computing Model in a simultaneous exercise
the Tier-0, Tier-1 and Tier-2 workflows
Test the CMS software:
particularly the reconstruction and HLT packages
Test the CMS production systems at 50% scale of expected 2008 operation
workflow management, data management, facilities, transfers
Test the computing facilities and mass storage systems.
Demonstrate that data will transfer between production and analysis sites in a timely way.
Test the Alignment and Calibration stream (AlcaReco)
Produce, deliver and store AODs + skims for analysis by physics groups
December 20, 2007
DESY CMS meeting

3. Prompt Reconstruction
CSA07 Workflows
Prompt Reconstruction
CASTOR
HLT
TIER-0
CAF
Calibration Express-Stream Analysis
300MB/s
Re-Reco
Skims
TIER-1
TIER-1
TIER-1
TIER-1
20-200MB/s
~10MB/s
Simulation
Analysis
TIER-2
TIER-2
TIER-2
TIER-2
December 20, 2007
DESY CMS meeting

4. Preparing for the CSA07 (Jul-Sep)
CMSSW - software releases organized by offline team
Releases are tested by data operations teams
Distributed and installed to the sites (This is not an easy process.)
Steps for preparing data for physics (pre-challenge workflows):
Generation and Simulation with Geant4 (at the Tier-2 centers)
Digitization
Digi2RAW - format change to look like data input to HLT
HLT processing
Data are split into 7 Primary Data Sets (PDS) based on the HLT information
This was a big addition in CSA07. The data samples more accurately reflect what will come from the detector, but are harder to produce.
December 20, 2007
DESY CMS meeting

5. Preparing for the CSA07 (Jul-Sep)
Planned Workflows for the Challenge:
Reconstruction - HLT + RECO output (~1 MB)
AOD production - (200 kB)
Skims for physics analysis at the Tier-1 centers
Re-Reco (and redoing AOD production/skims) at the Tier-1 centers
Analysis at the Tier-2 centers
Lessons from CSA07 preparations:
It turned out that there was insufficient time for testing the components since some of the components were coming at the latest moment..
CSA08: We have to devote more time for testing the components
December 20, 2007
DESY CMS meeting

6. December 20, 2007
DESY CMS meeting

7. MC Production summary Substantial (… more resources used)
December 20, 2007
DESY CMS meeting

8. CSA07 Issues and Lessons
There are clearly areas that are going to need development
Need to work on the CMSSW application
Reduce the number of workflows (in 170, 180 and 200)
Reduce the memory footprint to increase the number of events we can run and increase the available resources
Goal: CMSSW applications should stay in 1 GB memory
Several area should be improved
Access and manipulation of IOV constant (over Xmas)
HLT data model (on going)
New huge increase in memory seen in 170 to be address immediately (mainly in DPG’s code)
December 20, 2007
DESY CMS meeting

9. CSA07 Issues and Lessons Increase the speed of IO on mass storage
test using new ROOT version
Improve our testing and validation procedures for the applications and workflows.
Reduce event size
RAW/DIGI and RECO size
AOD size
Mini-workshop with physics and DPG Groups on Jan 28/29 (CERN)
Two task forces has been created in order to prepare this workshop
RECO Task Force chair (Shahram Rahatlou)
Analysis Task Force chair (Roberto Tenchini)
FW support for handing of RAW, RECO versus FEVT (This is foreseen for version 2_0_0)
December 20, 2007
DESY CMS meeting

10. CSA07 Issues and Lessons
Need to work on the CMS Tools
Augment the Production tools to be able to better handle continuous operations
Roll back to known good points.
Modify workflows more simply
Increase the speed of Bookkeeping System under specific load conditions
Optimize the data transfers in PhEDEx for data availability
Improve the analysis Tool (CRAB)
Planning a Workshop in January (Lyon)
Goals:
Review Data and Workload management components
Improvement integration (communication) between operation and development teams
Will include also Tier0 components
Define work plan for 2008
December 20, 2007
DESY CMS meeting

11. CSA07 Issues and Lessons
Facility Lessons:
We learned a lot about operating Castor and dCache under load
Need to improve the rate of file opens
Need to decrease the rate of errors.
Need to improve the scalability of some components
Need to work on the stability of services at CERN and Tier-1 centers
Need to work on the transfer quality when the farms are under heavy processing load
General lessons:
Much work is needed to achieve simultaneous, sustainable and stable operations
December 20, 2007
DESY CMS meeting

12. PADA: processing and data access taskforce
Draft Mandate:
Integrate developments and services to bring our centers and services to production quality for processing and analysis
The Processing And Data Access Task Force is an initiative in the Integration Program
Designed to transition services developed in Offline to Operations
Elements of integration and testing for Production, Analysis, and Data Management tools
Designed to ensure services and sites used in operations are production quality
Elements in the commissioning program for links and sites
Verify that items identified in the CSA07 are solved
Development work is primarily in offline, but verification in Integration
Plan is:
To build on the expertise of the distributed MC production teams, extend scope
We need the expertise in proximity of the centers to help us here
For 2008 we want to make this a recognized service contribution in the MoA scheme,
Initial time frame: 1 year until we have seen the first data
We need to define steps, milestones, recruit people, hope for MC-OPS, DDT, ....
December 20, 2007
DESY CMS meeting

13. PADA tasks + schedule
December 20, 2007
DESY CMS meeting

14. Final Check before Data taking starts: CCRC’08 = CSA08CMS
A combined challenge by all Experiments must be used to demonstrate the readiness of the WLCG Computing infrastructure before start of data taking at a scale comparable to the data taking in 2008.
CMS fully supports the plan, to execute this CCRC in two phases:
a set of functional tests in February 2008
the final challenge in May 2008 at 100% scale,
starting with the readout of the experiment
We must do this challenge as WLCG collaboration: Centers and Experiments together
Combined planning has started:
Mailing list created:
Agenda pages:
Phone conference every Monday afternoon
Monthly session in pre-GDB meeting
December 20, 2007
DESY CMS meeting

15. CMS CCRC’08 Schedule Phase 1 - February 2008: (proposed: 4-29.2.2008)
Possible scenario:
blocks of functional tests,
Try to reach 2008 scale for tests at…
Phase 2: - May 2008: (proposed: )
Full workflows at all centers executed simultaneously by all 4 LHC experiments
Use data from cosmics data run,
add artificial load to reach 100%
Duration of challenge: 1 week setup, 4 weeks challenge
December 20, 2007
DESY CMS meeting

16. 1) Detector Installation, Commissioning & Operation
2) Preparation of Software, Computing & Physics Analysis
Aug
Sep
Oct
Nov
Dec
Jan
Feb
Mar
Apr
May
S/w Release 1_6 (CSA07)
V36 Schedule (Nov’07)
CSA07
Cooldown of Magnet: Test
S/w Release 1_7 (CCR_0T, HLT Validation)
Tracker Insertion
2007 Physics Analyses
First Results Out
CMS Cosmic Run CCR_0T
Several short periods Dec-Mar)
Last Heavy Element Lowered
Test Magnet at low current
S/w Release 1_8 (Lessons of ‘07)
Functional Tests CSA08 (CCRC)
Beam-pipe Closed and Baked-out
S/w Release 2_0
(CCR_4T, Production startup MC samples)
1 EE endcap Installed, Pixels installed
MC Production for Startup
Cosmic Run CCR_4T
CSA08 (CCRC)
Combined Computing Readiness Challenge
Master Contingency
2nd ECAL Endcap Ready for Installation end Jun’08
December 20, 2007
DESY CMS meeting

17. CCRC’08 Phase 1: February 2008 Goals for CMS:
Verify solutions to CSA07 issues and lessons
- don’t repeat CSA07 where solution is not ready
Attempt to reach ‘08 scale on individual tests
- don’t repeat CSA07 where no increase in scale possible
Guiding principles:
CCRC’08-Phase 1 will be a Computing&Software challenge
no coupling of other deliverables to CCRC’08/1 tests
Cosmics run, MC production and Physics Analysis have priority and cannot be interrupted by CCRC’08 tests for long
We defined blocks of tests, which stress a specific service or workflow
Tests should be as independent from each other as possible
Tests should be done in parallel where possible
Individual test is considered successful if sustained for (3-5) days
Where full ‘08 scale is not possible (hardware) tests are scaled down to hardware limit
December 20, 2007
DESY CMS meeting

18. Status of preparation Need to provide requirements to sites
Draft plan of of tests available
Goals of tests agreed by computing
Area or responsibility and coordination is defined
Tests are to be coordinated and agreed within CMS
Many tests are to be scheduled together with other VO’s (ATLAS, …) to reach scale of stress test
CCRC’08 meeting next Thursday, 20.12, 16:00
Plan details of tests
Propose metrics
Define constraints and specify resources required
Propose schedule of test
Need to provide requirements to sites
December 20, 2007
DESY CMS meeting

19. Planned blocks of tests
Data recording at CERN
1a) readout from P5, use HLT, w. stream definition, use Storage Manager, transfer to T0, perform repacking, write to CASTOR
Goal: verify dataflow for CMS
1b) CASTOR data archiving test
Goal: verify CASTOR performance at full CMS and ATLAS rate
2. Processing at T0 at high rate
Goal: verify T0 performance under CMS + ATLAS load
3. CERN data export to T1 at full / high rate
Goal: verify CERN export and T1 import performance under CMS + ATLAS load
4. T1 data handling and processing
Goal: verify full CMS T1 re-processing workflow in presence of ATLAS load at T1’s
December 20, 2007
DESY CMS meeting

20. Planned blocks of tests
5 Data Transfer performance tests
Goal: verify T1/T2 export + import performance under CMS +ATLAS load
5.1 T1 - T1 data transfer at real rates
5.2 T1 - T2 data transfer at real rates
5.3 T2 - T1 data transfer at real rates
Data Transfer tests (5.1-3) should be done individually and then together
Monte Carlo Production and Analysis at Tier-2’s
Goal: Verify Pile-up and FastSim MC Production
Goal: Scale tests of Analysis Jobs
Data transfer request system tests
Goal: verify Data Transfer and SE’s at Tier-2 centers
8. CAF tests: (depending on CAF infrastructure schedule)
Goal: verify basics CMS use cases at scale
December 20, 2007
DESY CMS meeting

21. Summary (1/2) In CSA07 a lot was learned and a lot was achieved..
We hit most of metrics - but separately and intermittently
Several steps accomplished simultaneously
Many workflow steps hit metric routinely
Now work on accomplishing all steps simultaneously, and providing stability in a sustainable way.
Global connectivity between T1-T2 sites is still an important issue.
The DDT task force has been successful in increasing the # of working links.
This effort must continue and work must be done to automate the process of testing/commissioning the links.
We still have to increase the number of people involved in facilities, commissioning and operations.
Some recent actions: New (2nd) L2 appointed to lead facility operations (based at CERN)
New Production And Data Access (PADA) Task Force starting - will include some of the people from DDT task force and MC production teams.
December 20, 2007
DESY CMS meeting

22. Summary (2/2) ~ 200M Events processed and re-processed
Calibration, MC production, Reconstruction, skimming, merging all tested successfully.
Still need time to test the analysis model.
CSA07 Goals for providing data for physics will be accomplished … albeit delayed due to schedule slips
Processing continues to complete the data samples for physics and detector studies.
We are keeping the challenge infrastructure alive and trying to keep it stable, going forward...
Continue to support global detector commissioning and physics studies.
We have to prepare for the ‘Combined Computing Readiness Challenge’, CCRC’08  CSA08
Without testing the software and infrastructure we are not prepared…
December 20, 2007
DESY CMS meeting