WP12 - General Development News Sandro Wenzel

Slides:



Advertisements
Similar presentations
System Integration and Performance
Advertisements

Chapter 4 : File Systems What is a file system?
Chapter 8 Operating System Support
Computer System Organization S H Srinivasan
Computer Organization and Architecture
1 Input/Output Chapter 3 TOPICS Principles of I/O hardware Principles of I/O software I/O software layers Disks Clocks Reference: Operating Systems Design.
CS364 CH08 Operating System Support TECH Computer Science Operating System Overview Scheduling Memory Management Pentium II and PowerPC Memory Management.
Chapter 5 Operating System Support. Outline Operating system - Objective and function - types of OS Scheduling - Long term scheduling - Medium term scheduling.
The Functions of Operating Systems Interrupts. Learning Objectives Explain how interrupts are used to obtain processor time. Explain how processing of.
A User-Lever Concurrency Manager Hongsheng Lu & Kai Xiao.
GLAST Gaudi Code Review, 10 Sept. 2002, H. Kelly, 2-1 GLAST Event Data Model and Persistency.
The ALICE short-term use case DataGrid WP6 Meeting Milano, 11 Dec 2000Piergiorgio Cerello 1 Physics Performance Report (PPR) production starting in Feb2001.
Analysis trains – Status & experience from operation Mihaela Gheata.
Precomputation- based Prefetching By James Schatz and Bashar Gharaibeh.
Memory management.
CIS250 OPERATING SYSTEMS Chapter One Introduction.
Different Microprocessors Tamanna Haque Nipa Lecturer Dept. of Computer Science Stamford University Bangladesh.
Requirements for the O2 reconstruction framework R.Shahoyan, 14/08/
STAR Simulation. Status and plans V. Perevoztchikov Brookhaven National Laboratory,USA.
Axel Naumann, DØ University of Nijmegen, The Netherlands 6/20/2001 Dutch Morning Meeting 1 From n-Tuples to b-Tags ?
21/5/03J.E Campagne Opera Framework: Status Report.
EE 345S Real-Time Digital Signal Processing Lab Fall 2008 Lab #3 Generating a Sine Wave Using the Hardware & Software Tools for the TI TMS320C6713 DSP.
Operating System Concepts with Java – 7 th Edition, Nov 15, 2006 Silberschatz, Galvin and Gagne ©2007 Chapter 0: Historical Overview.
1 Module 3: Processes Reading: Chapter Next Module: –Inter-process Communication –Process Scheduling –Reading: Chapter 4.5, 6.1 – 6.3.
HYDRA Framework. Setup of software environment Setup of software environment Using the documentation Using the documentation How to compile a program.
Computer Security: Chapter 5 Operating Systems Security.
Monthly video-conference, 18/12/2003 P.Hristov1 Preparation for physics data challenge'04 P.Hristov Alice monthly off-line video-conference December 18,
Welcome to Intro to Operating Systems Course Website:
Addressing modes, memory architecture, interrupt and exception handling, and external I/O. An ISA includes a specification of the set of opcodes (machine.
Process Management Deadlocks.
V4-18-Release P. Hristov 21/06/2010.
Chapter 8 I/O.
Input/Output Device Drivers
SNiPER在LHAASO实验中的应用: LodeStar
Non Contiguous Memory Allocation
Basic Computer Organization and Design
Applied Operating System Concepts
Processes and threads.
David Kauchak CS 52 – Spring 2017
Understanding Operating Systems Seventh Edition
Topic 3 (Textbook - Chapter 3) Processes
Cluster Optimisation using Cgroups
Advanced OS Concepts (For OCR)
COMBINED PAGING AND SEGMENTATION
Tree based validation tool for track reconstruction
Recent performance improvements in ALICE simulation/digitization
Chapter 2: System Structures
Lecture Topics: 11/1 Processes Process Management
William Stallings Computer Organization and Architecture
Chapter 3: Processes.
AliRoot status and PDC’04
Swapping Segmented paging allows us to have non-contiguous allocations
Chapter 8 I/O.
Computer Architecture
Subject Name: Digital Signal Processing Algorithms & Architecture
Chapter 8 I/O.
Computer System Overview
Machine Independent Features
Computer-System Architecture
Processor Fundamentals
CS399 New Beginnings Jonathan Walpole.
Chapter 8 I/O.
Processes Hank Levy 1.
Multithreaded Programming
Chapter 2: Operating-System Structures
Processes Hank Levy 1.
Chapter 2: Operating-System Structures
COMP755 Advanced Operating Systems
Virtual Memory 1 1.
Presentation transcript:

WP12 - General Development News Sandro Wenzel

Software quality / development workflow ROOT macros ROOT macros and compiled libraries go quickly out of sync and could cause defect code Did a campaign to systematically look if macros in O2 still execute fine with ROOT Fixed many macros which did not run (see https://github.com/AliceO2Group/AliceO2/issues/594) Few macros could not be fixed and were renamed to “.C_backup”; (please check if they are still needed) from now on the pull request checker will automatically check all our macros and flag error when there is a problem Makes keeping a good code state a lot easier

Arbitrary Types with FairRoot/IO Status quo So far, only classes of type TNamed or TCollections could be exchanged with the FairRootManager for persistent IO and exchange of data between tasks Registering outgoing data TClonesArray *mHits; mHits = new TClonesArray("tpc::Hit"); iomgr->Register("TPCHit", mHits, true); mHits = dynamic_cast<TClonesArray*>(iomgr->GetObject("TPCHit")); Asking for incoming data Memory overhead due to Data elements needs to be TObject (~16bytes overhead) Data elements are stored as pointers (8bytes overhead) Type unsafe code TClonesArray can’t do type checking at compile time

Arbitrary Types with FairRoot/IO New feature A pull request has been made to FairRoot which extends the FairRootManager to handle branches of any type for the purpose of IO and data exchange Store hits in stl containers Registering typed outgoing data std::vector<tpc::Hit> *mHits; iomgr->RegisterAs("TPCHit", mHits, true); mHits = iomgr->InitObjectAs<std::vector<tpc::Hit>*>("TPCHit"); Asking for typed incoming data No negative effect on split level of persistent branches Feedback/comments on PR welcome Once PR accepted, I would like to move completely away from TClonesArrays in favour of std::vectors

Lazy loading of branches with FairRoot analysis runs Simulation writes hits for all detectors into same tree (different branches) Can lead to very large trees/files quickly Example: 4 PbPb hijing events with ~15000 primaries each lead to ~1.7GB on disc When processing these hits (digitization) the standard FairRunAna class will load every single branch for a given event into memory Causes large memory buffers to be allocated (~3GB for above example) Might be substantial overhead in cases where we do not consume all these data, for example: Digitize only ITS hits Digitize only one TPC sector

Lazy loading of branches with FairRoot analysis runs Proposing an addition to FairRootManager (PR636) that allows to keep track of which processing task needs which data + made a prototype modification to FairRunAna (O2RunAna) Now able to load only branches actually needed for processing; May lead to large gain in memory and CPU time: tpc-run-sim -m digi -n 4 —-dSector 2 # digitize only sector 2 MaxMemory: 3742.56MB CPU time 21.08s MaxMemory: 789.52MB CPU time 7.66s Definitely useful for rapid prototypic, profiling algorithms A step in organizing data and processing in direction of efficient parallel device-based processing

MC labels Discussed some time ago the issue of MC-labels TPC has restructured their digitization to make use of external MC labels Allows to have arbitrarily many labels per data element Labels are stored in a separate branch and no more distinction between DigitMC and DigitRaw Will report in one of the next meetings… For now interested developers could take a look here: PR #578: Shows how the final label branch is filled PR #586: Generalization allowing to fill labels in random data order Feedback welcome!