Computing Strategy of the AMS-02 Experiment V. Choutko1, B. Shan2 A. Egorov1, A. Eline1 1MIT, USA 2Beihang University, China CHEP 2015, Okinawa, April 13, 2015

Outline AMS Experiment AMS Data/Computing Model Parallelization of Reconstruction Software Parallelization of Simulation Software Production Supporting Software Summary

Z, P are measured independently by the Tracker, RICH, TOF and ECAL AMS: A TeV precision, multipurpose particle physics spectrometer in space. TRD Identify e+, e- Particles and nuclei are defined by their charge (Z) and energy (E ~ P) TOF Z, E TRD TOF Tracker RICH ECAL 1 2 7-8 3-4 9 5-6 Magnet ±Z Silicon Tracker Z, P RICH Z, E ECAL E of e+, e-, γ Z, P are measured independently by the Tracker, RICH, TOF and ECAL 3 3 3

AMS Experiment The Alpha Magnetic Spectrometer (AMS) is a high energy physics experiment on board the ISS, featured: TOF: Four Layers of Scintillators (120 ps); Tracker: Nine Layers of Silicon Detectors (10 μ); Gaseous TRD Detector: h/e Rejection O(102); Pb/Sc EM Calorimeter: h/e Rejection O(104); RICH Detector: Velocity Measurement O(10-4); Geometrical Acceptance: 0.5 m2sr; Number of ReadOut Channels: ≈200K TRD TOF Tracker RICH ECAL 1 2 7-8 3-4 9 5-6

AMS Experiment The Alpha Magnetic Spectrometer (AMS) is a high energy physics experiment on board the ISS, featured: TOF: Four Layers of Scintillators (120 ps); Tracker: Nine Layers of Silicon Detectors (10 μ); Gaseous TRD Detector: h/e Rejection O(102); Pb/Sc EM Calorimeter: h/e Rejection O(104); RICH Detector: Velocity Measurement O(10-4); Geometrical Acceptance: 0.5 m2sr; Number of ReadOut Channels: ≈200K

AMS Data Model Average 500 cosmic ray events per second Data collection rate ~1.2MB/s Transferred via relay satellite, Marshall Space Flight Center, to CERN Recording original electron signals Organized to one RUN every ¼ orbit (~23 minutes) First Production Runs 24/7 on freshly arrived data Initial data validation and indexing Produces Data Summary Files and Event Tags for fast events selection Requires ~ 80 CPU cores to cope with data rate Usually be available within 2 hours after flight data arrived used to produce various calibrations for the second production as well as quick performance evaluation. Second Production To use all the available calibrations, alignments, ancillary data from the ISS, and slow control data (temperatures, pressures, voltages) to produce physics analysis ready set of reconstructed data.

Remote Computing Facilities AMS Computing Model DATABASE ORACLE(PDBR) Frames Web Server Preproduction Production Server Production platform RAW AMS Cluster (PBS) CERN lxbatch (LSF) Remote Computing Facilities (Producer) ROOT(DST) Remote Centers Validator Uploader EOS, CASTOR Data Meta-data / Control CERN Facilities

Highlights in AMS Offline Software Parallelization of Reconstruction Parallelization of Simulation Production Supporting Software

Parallelization of Reconstruction Motivations To shorten the (elapsed) time of data reconstruction To increase productivity by using multi-core/hyper-thread enabled processors To ease the production jobs management; Tool: openmp No explicit thread programming Nearly no code change except few pragmas Possible to change thread number during execution

Sequential Procession

Parallel Procession

Parallel Reconstruction Performance (Xeon?)

Parallel Reconstruction Performance (Intel MIC)

Parallel Reconstruction Performance (Intel X5482)

Parallel Reconstruction Performance (Intel Old Xeon HT)

Parallel Reconstruction Performance (Intel i7 HT)

Parallel Reconstruction Performance (Efficiency)

Parallel Reconstruction Performance (Xeon Nehalem HT)

Parallel Reconstruction Performance (Elapsed time change)

Parallelization of Simulation Software Motivations To decrease required memory amount per simulation job To allow faster Monte-Carlo parameter/software tuning Minimal software change: Geant 4.10.1 MT model + openmp (icc 15) Thread ID, thread number, and core number are taken from Geant 4.10.1, instead of from openmp barrier, master, parallel from openmp do not work, so barriers are implemented by emulation All other openmp pragmas work

Parallelization of Simulation Software Memory optimization -- G4AllocatorPool modification Garbage Collection with preserving of n chunks  (1 by default) with modes: Fully automatic At the end of event Fast unsafe by name of G4Allocator at the end of event Decreased the memory consumption by factor 2 and more for long jobs (W.R.T. original G4AllocatorPool) Decreased the memory consumption by factor ?? W.R.T. sequential simulation job

Parallel Simulation Performance (Memory consumption)

Parallel Simulation Performance (Xeon?)

Parallel Simulation Performance (Intel MIC)

Production Supporting Software Fully automated production cycle for: Production in local (CERN) and remote computing centers Data reconstruction and Monte-Carlo simulation Local production Production server/monitor Remote computing centers Light-weight production platform

Local production management Centralized job management by production server/monitor All production database reading is through server Load balance between different kinds of computing resources CORBA-based communication between producers and server Dynamic thread number adjusting Graphical production monitoring

Light-weight production platform Fully-automated production cycle Job acquiring, submission, monitoring, validation, transferring, and (optional) scratching Easy to deploy Based on perl/python/sqlite3 Installation-free Customizable Batch system, storage area, transferring method, etc. Running at: JUROPA, CNAF, IN2P3, NLAA

Latest Full Data Reconstruction Second half of Feb. 2015 Performed at CERN, JUROPA, CNAF, ACAD. SINICA 41 months’ flight data Reconstruction completed essentially in two weeks 100 times data collection speed 10-5 failing rate

Latest Full Data Reconstruction (I will add one plot here to show the completing status of the pass6 reconstruction)

Summary Parallelization of the AMS reconstruction software allows to increase the productivity of the reconstruction program up to 45% using the HT enable processors as well as to short per run reconstruction time and calibration jobs execution time by factor of 10. Parallelization of the AMS simulation software allows to decrease the total memory consumption up to ??, which makes it much easier to run long simulation jobs for light nuclei. With production supporting software, data reconstruction rate reached 100 times of data collection.