Near Real Time Reconstruction of PHENIX Run7 Minimum Bias Data From RHIC Project Goals Reconstruct 10% of PHENIX min bias data from the RHIC Run7 (Spring 2007, Au+Au @ 200 GeV) in near real time at Vanderbilt University computer farm Envisioned as a demonstration project to PHENIX, CMS, VU, and DOE Speed the process of deriving physics information from the detector’s raw data Currently about one year elapses between end of data taking and end of reconstruction Run7 data taken by PHENIX may not be analyzed in time for QM’08 (February 2008) In future Vanderbilt could be reconstructing ~1/3 of PHENIX raw data in real time, with ~1/3 analyzed at CCF and CCJ (PHENIX computer centers France and Japan), and 1/3 at RHIC Computer Facility (RCF) ===> more time and more CPUs for analyses ! Project Requirements Large computer farm with ~50 TBytes of disk space, 200 CPUs Fast and automated file transport to and from the Vanderbilt computer farm Close coordination among PHENIX, Vanderbilt, and the RHIC Computer Facility Dedicated scientific staff at Vanderbilt (1 faculty, 1 PD, 3 students) Most important facet is complete automation and comprehensive monitoring tools June 15, 2007 CMS-HI at CERN

PRDFs Vanderbilt Farm 1600 CPUs, 80 TBytes disk 45TB and 200 CPUs Available for Run7 Reconstruction RCF RHIC computing facility Reconstruction 200 jobs/cycle PRDFs nanoDSTs 18 hours/job 3 FDT 45 MB/s FDT 45 MB/s 1 770 GBytes per cycle PRDFs Raw data files GridFTP to VU 30 MBytes/sec 2 nanoDSTs Reco output to RCF GridFTP 23 MB/sec Dedicated GridFTP Server Firebird 4 4.4TB of buffer disk space June 15, 2007

Actual Experience April - June 2007 Raw Data File Transport (PHENIX DAQ -> Vanderbilt) Automated GridFTP tools in use since 2005 (to CCJ, ORNL, VU, CCF) Works very well and fast enough (30 MBytes/second for current needs) Can (and should) be pushed harder New FDT scripts run for internal transfers can run at 100 MBytes/second FDT is a java-based technology supported out of CERN Throttled back to 45 MBytes/second to minimize impact elsewhere on farm May receive 37 TB of raw data (Au+Au, 200 GeV) containing 325M events by the end of June Reconstruction cycle Larger than expected output size: 70% of input size, PHENIX had previewed 35% Transport time of output limits the compute cycle to being 200 jobs to avoid pileup Cycle is 18 hours (jobs are 12 CPU-hours but new quad CPUs are only 75% efficient [memory bus] !!) Weekends, holidays are slack times --> 200 jobs are start running right away Weekdays are busier, means that there can be a 12 hour wait for the last job to start Reconstructed Output File Transport (Vanderbilt -> RCF) Most fragile component due to disk I/O competition at RCF with other PHENIX users PHENIX is severely short on disk space for this year, but better next fiscal year Developed “fault tolerant” GridFTP scripts which recover if transfers fail mid-stream Large demand to view files arriving at RCF (4 new subsystems installed for Run7) June 15, 2007 CMS-HI at CERN