Ilija Vukotic Data size and IO performance report ATLAS Software & Computing Workshop.

Slides:

Advertisements

Similar presentations

How to Perform a SQL Server Health Check

Advertisements

Sander Klous on behalf of the ATLAS Collaboration Real-Time May /5/20101.

Memory Address Decoding

Resources for the ATLAS Offline Computing Basis for the Estimates ATLAS Distributed Computing Model Cost Estimates Present Status Sharing of Resources.

1 External Sorting Chapter Why Sort?  A classic problem in computer science!  Data requested in sorted order  e.g., find students in increasing.

1 Lecture 16B Memories. 2 Memories in General Computers have mostly RAM ROM (or equivalent) needed to boot ROM is in same class as Programmable Logic.

ATLAS Analysis Model. Introduction On Feb 11, 2008 the Analysis Model Forum published a report (D. Costanzo, I. Hinchliffe, S. Menke, ATL- GEN-INT )

CS 300 – Lecture 19 Intro to Computer Architecture / Assembly Language C Coding & The Simulator Caches.

In order to acquire the full physics potential of the LHC, the ATLAS electromagnetic calorimeter must be able to efficiently identify photons and electrons.

December Pre-GDB meeting1 CCRC08-1 ATLAS’ plans and intentions Kors Bos NIKHEF, Amsterdam.

Large scale data flow in local and GRID environment V.Kolosov, I.Korolko, S.Makarychev ITEP Moscow.

Wahid Bhimji University of Edinburgh J. Cranshaw, P. van Gemmeren, D. Malon, R. D. Schaffer, and I. Vukotic On behalf of the ATLAS collaboration CHEP 2012.

Compact TileCell for the ATLAS ESD (status) ( Continued from 29/09/04 TileCal Software meeting )  The compactification option implemented for TILE use:

Event Metadata Records as a Testbed for Scalable Data Mining David Malon, Peter van Gemmeren (Argonne National Laboratory) At a data rate of 200 hertz,

11 SYSTEM PERFORMANCE IN WINDOWS XP Chapter 12. Chapter 12: System Performance in Windows XP2 SYSTEM PERFORMANCE IN WINDOWS XP  Optimize Microsoft Windows.

Claudio Grandi INFN Bologna CMS Operations Update Ian Fisk, Claudio Grandi 1.

Digital Camera Overview IT 130 Web Graphics and Multimedia.

Analysis Plans for Jets + EtMiss Signatures Pierre Savard ATLAS Toronto Group Meeting January

 Optimization and usage of D3PD Ilija Vukotic CAF - PAF 19 April 2011 Lyon.

Plans for Trigger Software Validation During Running Trigger Data Quality Assurance Workshop May 6, 2008 Ricardo Gonçalo, David Strom.

IT253: Computer Organization

Analysis of the ROOT Persistence I/O Memory Footprint in LHCb Ivan Valenčík Supervisor Markus Frank 19 th September 2012.

Introduction Advantages/ disadvantages Code examples Speed Summary Running on the AOD Analysis Platforms 1/11/2007 Andrew Mehta.

DPDs and Trigger Plans for Derived Physics Data Follow up and trigger specific issues Ricardo Gonçalo and Fabrizio Salvatore RHUL.

PanDA Update Kaushik De Univ. of Texas at Arlington XRootD Workshop, UCSD January 27, 2015.

CS 3500 L Performance l Code Complete 2 – Chapters 25/26 and Chapter 7 of K&P l Compare today to 44 years ago – The Burroughs B1700 – circa 1974.

Efi.uchicago.edu ci.uchicago.edu Using FAX to test intra-US links Ilija Vukotic on behalf of the atlas-adc-federated-xrootd working group Computing Integration.

Navigation Timing Studies of the ATLAS High-Level Trigger Andrew Lowe Royal Holloway, University of London.

PWG3 Analysis: status, experience, requests Andrea Dainese on behalf of PWG3 ALICE Offline Week, CERN, Andrea Dainese 1.

Trigger Validation TAPM Open meeting 2 October 2007 Ricardo Goncalo.

Profiling Where does my application spend the time? Profiling1.

ALICE Operations short summary ALICE Offline week June 15, 2012.

Moritz Backes, Clemencia Mora-Herrera Département de Physique Nucléaire et Corpusculaire, Université de Genève ATLAS Reconstruction Meeting 8 June 2010.

Accessing PBeast and monitoring the L1 trigger Emily Thompson.

A New Tool For Measuring Detector Performance in ATLAS ● Arno Straessner – TU Dresden Matthias Schott – CERN on behalf of the ATLAS Collaboration Computing.

PESAsim – the e/  analysis framework Validation of the framework First look at a trigger menu combining several signatures Short-term plans Mark Sutton.

Integration of the ATLAS Tag Database with Data Management and Analysis Components Caitriana Nicholson University of Glasgow 3 rd September 2007 CHEP,

Excellence Publication Co. Ltd. Volume Volume 1.

Trigger Validation Olga Igonkina (U.Oregon), Ricardo Gonçalo (RHUL) on behalf of trigger community Physics Validation Meeting – Feb. 13, 2007.

PERFORMANCE AND ANALYSIS WORKFLOW ISSUES US ATLAS Distributed Facility Workshop November 2012, Santa Cruz.

Predrag Buncic Future IT challenges for ALICE Technical Workshop November 6, 2015.

What makes up a computer. By Miranda, Ally and Eloise.

Large scale data flow in local and GRID environment Viktor Kolosov (ITEP Moscow) Ivan Korolko (ITEP Moscow)

Virtual Memory Review Goal: give illusion of a large memory Allow many processes to share single memory Strategy Break physical memory up into blocks (pages)

AliRoot survey: Analysis P.Hristov 11/06/2013. Are you involved in analysis activities?(85.1% Yes, 14.9% No) 2 Involved since 4.5±2.4 years Dedicated.

Performance DPDs and trigger commissioning Preparing input to DPD task force.

ATLAS Metadata Interface Campaign Definition in AMI S.Albrand 23/02/2016ATLAS Metadata Interface1.

ATLAS Distributed Computing perspectives for Run-2 Simone Campana CERN-IT/SDC on behalf of ADC.

Victoria, Sept WLCG Collaboration Workshop1 ATLAS Dress Rehersals Kors Bos NIKHEF, Amsterdam.

What is it and why do we need it? Chris Ward CS147 10/16/2008.

Computer Performance. Hard Drive - HDD Stores your files, programs, and information. If it gets full, you can’t save any more. Measured in bytes (KB,

Analysis Performance and I/O Optimization Jack Cranshaw, Argonne National Lab October 11, 2011.

Main parameters of Russian Tier2 for ATLAS (RuTier-2 model) Russia-CERN JWGC meeting A.Minaenko IHEP (Protvino)

Points from DPD task force First meeting last Tuesday (29 th July) – Need to have concrete proposal in 1 month – Still some confusion and nothing very.

BEACH 04J. Piedra1 SiSA Tracking Silicon stand alone (SiSA) tracking optimization SiSA validation Matthew Herndon University of Wisconsin Joint Physics.

 IO performance of ATLAS data formats Ilija Vukotic for ATLAS collaboration CHEP October 2010 Taipei.

Mini-Workshop on multi-core joint project Peter van Gemmeren (ANL) I/O challenges for HEP applications on multi-core processors An ATLAS Perspective.

ATLAS Distributed Computing Tutorial Tags: What, Why, When, Where and How? Mike Kenyon University of Glasgow.

ANALYSIS TRAIN ON THE GRID Mihaela Gheata. AOD production train ◦ AOD production will be organized in a ‘train’ of tasks ◦ To maximize efficiency of full.

Peter van Gemmeren (ANL) Persistent Layout Studies Updates.

ROOT IO workshop What relates to ATLAS. General Both CMS and Art pushing for parallelization at all levels. Not clear why as they are anyhow CPU bound.

Atlas IO improvements and Future prospects

Some introduction Cosmics events can produce energetic jets and missing energy. They need to be discriminated from collision events with true MET and jets.

Developments of the PWG3 muon analysis code

FileStager test results

Offline data taking and processing

Associated Hardware and File Handling

An Empirical Analysis of Java Performance Quality

Year 10 Computer Science Hardware - CPU and RAM.

Query Processing.

Presentation transcript:

Ilija Vukotic Data size and IO performance report ATLAS Software & Computing Workshop

Data size –MC infoIOperformance/MCsizes.htmlhttp://athena-infoioperformance.web.cern.ch/athena- infoIOperformance/MCsizes.html –Streams infoioperformance.web.cern.ch/athena- infoIOperformance/StreamsSizes.htmlhttp://athena- infoioperformance.web.cern.ch/athena- infoIOperformance/StreamsSizes.html –Categories infoioperformance.web.cern.ch/athena- infoIOperformance/T0streams.htmlhttp://athena- infoioperformance.web.cern.ch/athena- infoIOperformance/T0streams.html IO performance Possibilities for improvement 05/03/2016Ilija Vukotic2

MC event size 05/03/2016Ilija Vukotic3

MC event size 05/03/2016Ilija Vukotic4 Note: trigger size is not realistic as used trigger menu was not realistic. Problem rectified with addition of new aliases: Physics_default -> Physics_pp_v1 MC_loose_default -> MC_pp_v1_loose_mc_prescale MC_tight_default -> MC_pp_v1_tight_Mc_prescale (see bug report #74712 ). Size AODSize ESD RecoTrf %* % stdSim % % fullSim % % physSim % % *Compared to 22 Aug. 2010

Streams sizes 05/03/2016Ilija Vukotic5 Only runs with 5k+ events and good LBs Shows: AODs,ESDs and all DESDs Events/total size/ev. size

Streams sizes 05/03/2016Ilija Vukotic6 StreamSize[GB] ESD AOD92319 DESDM_EGAMMA55288 DESD_SGLEL44191 DESD_SGLMU39375 DESDM_TRACK26548 DESD_MBIAS16841 DESD_CALJET12370 DESDM_MET5108 DESD_PHOJET4281 DESD_SGLMU DESD_SGLEL DESDM_EGAMMA AOD ESD

05/03/2016Ilija Vukotic7 Categories

05/03/2016Ilija Vukotic8 EgammaL1CaloMuonsJetTauETMiss Details and list of collections in categories can be found at:

05/03/2016Ilija Vukotic9 IO performance Slow Bad CF KEEP IN MIND In real data TrackCollections are even larger !

05/03/2016Ilija Vukotic10 IO performance Do we really need all of these? No way to say if object is ever used. What is so terrible about having a name attached to each object we store?

Possibilities for improvement TrackCollection/TrackParticle container –Maybe 50kb in size and factor 2 in time –Few MB in memory CaloShowerContainer –Probably can go down in size Calo(Topo)TowerContainer –Gets recreated at read time –Must be possible to optimize further TRT_Drift_Circle_Container –Must be faster and smaller 05/03/2016Ilija Vukotic11 My estimate: In total between 50 and 200kB/ev can be shaved in converters Should be possible >100ms/ev (currently at 480 ms/ev) Other options: Object removal. Cuts tuning