Work in progress Philippe & Rene. Memory Pools when reading Currently we new/delete – zipped buffer (when no cache) – unzipped buffer – branches target.

Slides:



Advertisements
Similar presentations
External Sorting The slides for this text are organized into chapters. This lecture covers Chapter 11. Chapter 1: Introduction to Database Systems Chapter.
Advertisements

Intel® performance analyze tools Nikita Panov Idrisov Renat.
Database Management Systems, R. Ramakrishnan and J. Gehrke1 External Sorting Chapter 11.
1 External Sorting Chapter Why Sort?  A classic problem in computer science!  Data requested in sorted order  e.g., find students in increasing.
D. Düllmann - IT/DB LCG - POOL Project1 POOL Release Plan for 2003 Dirk Düllmann LCG Application Area Meeting, 5 th March 2003.
Database Management Systems, R. Ramakrishnan and J. Gehrke1 Query Evaluation Chapter 11 External Sorting.
File System Implementation
External Sorting R & G Chapter 13 One of the advantages of being
External Sorting R & G Chapter 11 One of the advantages of being disorderly is that one is constantly making exciting discoveries. A. A. Milne.
11/3/2005Comp 120 Fall November 10 classes to go! Cache.
Chapter 3.7 Memory and I/O Systems. 2 Memory Management Only applies to languages with explicit memory management (C or C++) Memory problems are one of.
1 External Sorting Chapter Why Sort?  A classic problem in computer science!  Data requested in sorted order  e.g., find students in increasing.
External Sorting 198:541. Why Sort?  A classic problem in computer science!  Data requested in sorted order e.g., find students in increasing gpa order.
1 External Sorting for Query Processing Yanlei Diao UMass Amherst Feb 27, 2007 Slides Courtesy of R. Ramakrishnan and J. Gehrke.
How can a mainframe use Linux to replace the equivalent of hundreds of Unix or Windows servers? Server consolidation – replacing dozens or even hundreds.
Module I Overview of Computer Architecture and Organization.
Filesytems and file access Wahid Bhimji University of Edinburgh, Sam Skipsey, Chris Walker …. Apr-101Wahid Bhimji – Files access.
Multimedia Operating Systems ●File System Paradigms ●File Replacement ●Caching ●Disk.
ROOT I/O recent improvements Bottlenecks and Solutions Rene Brun December
Swapping to Remote Memory over InfiniBand: An Approach using a High Performance Network Block Device Shuang LiangRanjit NoronhaDhabaleswar K. Panda IEEE.
Sorting.
Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke1 External Sorting Chapter 13.
Chep 2006, Mumbai, IndiaPhilippe Canal (FNAL)1 Recent Developments in the ROOT I/O and TTrees CANAL, Philippe (FERMILAB) BRUN, Rene (CERN) FRANK, Markus.
External Storage Primary Storage : Main Memory (RAM). Secondary Storage: Peripheral Devices –Disk Drives –Tape Drives Secondary storage is CHEAP. Secondary.
Database Management Systems, R. Ramakrishnan and J. Gehrke 1 External Sorting Chapter 13.
ROOT and Federated Data Stores What Features We Would Like Fons Rademakers CERN CC-IN2P3, Nov, 2011, Lyon, France.
Silberschatz, Galvin and Gagne  Operating System Concepts Chapter 12: File System Implementation File System Structure File System Implementation.
12.1 Silberschatz, Galvin and Gagne ©2003 Operating System Concepts with Java Chapter 12: File System Implementation Chapter 12: File System Implementation.
Silberschatz, Galvin and Gagne ©2009 Operating System Concepts – 8 th Edition File System Implementation.
The High Performance Simulation Project Status and short term plans 17 th April 2013 Federico Carminati.
OSG Abhishek Rana Frank Würthwein UCSD.
GeantV scheduler, concurrency Andrei Gheata GeantV FNAL meeting Fermilab, October 20, 2014.
Slide 1/29 Informed Prefetching in ROOT Leandro Franco 23 June 2006 ROOT Team Meeting CERN.
March, PROOF - Parallel ROOT Facility Maarten Ballintijn Bring the KB to the PB not the PB to the KB.
4P13 Week 9 Talking Points
ROOT I/O: The Fast and Furious CHEP 2010: Taipei, October 19. Philippe Canal/FNAL, Brian Bockelman/Nebraska, René Brun/CERN,
Update on G5 prototype Andrei Gheata Computing Upgrade Weekly Meeting 26 June 2012.
Introduction to Database Systems1 External Sorting Query Processing: Topic 0.
Database Management Systems 3ed, R. Ramakrishnan and J. Gehrke1 External Sorting Chapters 13: 13.1—13.5.
CPSC Why do we need Sorting? 2.Complexities of few sorting algorithms ? 3.2-Way Sort 1.2-way external merge sort 2.Cost associated with external.
ROOT Workshop M.Frank LHCb/CERN Improvements in the I/O Area (*)  General I/O related improvements  Tree related issues  Plans (*) I present.
S.Linev: Go4 - J.Adamczewski, H.G.Essel, S.Linev ROOT 2005 New development in Go4.
Database Management Systems, R. Ramakrishnan and J. Gehrke1 External Sorting Chapter 11.
External Sorting. Why Sort? A classic problem in computer science! Data requested in sorted order –e.g., find students in increasing gpa order Sorting.
CHEP09 after thoughts Rene Brun. CHEP09 and ROOT Conference was good for us ROOT is now in all HENP experiments SuperBelle and HSC moving to ROOT FAIR:
Analysis efficiency Andrei Gheata ALICE offline week 03 October 2012.
What’s In Store For ROOT I/O Philippe Canal March 22nd, 2012.
I/O aspects for parallel event processing frameworks Workshop on Concurrency in the many-Cores Era Peter van Gemmeren (Argonne/ATLAS)
MAUS Status A. Dobbs CM43 29 th October Contents MAUS Overview Infrastructure Geometry and CDB Detector Updates CKOV EMR KL TOF Tracker Global Tracking.
Mini-Workshop on multi-core joint project Peter van Gemmeren (ANL) I/O challenges for HEP applications on multi-core processors An ATLAS Perspective.
ROOT : Outlook and Developments WLCG Jamboree Amsterdam June 2010 René Brun/CERN.
1 Design and Implementation of a High-Performance Distributed Web Crawler Polytechnic University Vladislav Shkapenyuk, Torsten Suel 06/13/2006 석사 2 학기.
Ideas for ROOT in the future
Jean-Philippe Baud, IT-GD, CERN November 2007
Analysis framework - status
External Sorting The slides for this text are organized into chapters. This lecture covers Chapter 11. Chapter 1: Introduction to Database Systems Chapter.
Selected Topics: External Sorting, Join Algorithms, …
Memory Hierarchy Memory: hierarchy of components of various speeds and capacities Hierarchy driven by cost and performance In early days Primary memory.
Overview of Computer Architecture and Organization
Overview: File system implementation (cont)
PROOF - Parallel ROOT Facility
OO-Design in PHENIX PHENIX, a BIG Collaboration A Liberal Data Model
Lecture 28: Index 3 B+ Trees
Overview of Computer Architecture and Organization
External Sorting.
Planning next release of GAUDI
External Sorting Dina Said
New I/O navigation scheme
Presentation transcript:

work in progress Philippe & Rene

Memory Pools when reading Currently we new/delete – zipped buffer (when no cache) – unzipped buffer – branches target objects The idea is to create a permanent pool for – the unzipped buffer – the target object in case the branch has ownership main point is to reduce memory fragmentatio n

Parallel buffer merge The main bottleneck in parallel applications is the merging of the IO (buffers or files). For example a system with 100 workers can produce 1 Gbyte of data in one minute, then it takes more than 1hour to merge the output files!! The idea is to accumulate per worker large buffers (TreeCache around 20 Mbytes) and send these zipped buffers to one or more IO servers receiving these buffers asynchronously. Once this is implemented, it could also be used to implement a fast file parallel merge on the network.

TTree::GetRestOfEntry In skimming applications, typically one reads one or a few branches to select events, then one wants to read the rest of the event (or a collection of branches). The idea is to sort the buffers for the remaining branches (like the TreeCache is doing) with a MergeRequest. This requires a change in TTree::GetEntry to preload the baskets and a new function like TTree::GetEntry(Long64_t entry, TList *branches)

TStreamerInfo::ReadBuffer Two possible optimisations are under discussion: TVirtualCollectionProxy::At is an expensive non- inlined virtual function called for each member of a collection. Virtuality should be at the level of the collection, not at the member level. The function is currently implemented as a gigantic switch/case within a loop. Explore replacement by pointers to functions and at the same time minimize the number of “ifs” within loops.

TClass::AddRule To simplify the interface for the auto schema evolution, we propose TClass::AddRule(classname,rule) The rules could also be specified in a new file $ROOTSYS/etc/class.rules. We have an immediate candidate for class.rules with the HepMC class HepEvent/GenVertex when calling TFile::MakeProject on Atlas, CMS and LHCb files. – HepEvent m_vertex options=“notOwner”

TMemStat revisited Anar working to get the new collection algorithm working on Linux and MAC (32 & 64 bits versions). Once this is available (Anar at CERN next week) Rene will work on the visualisation/query system (a la TTreePerfStats) to understand memory leaks, fragmentation and abuse of new/delete (hence pools).

GDML interface Andrei has agreed to support the GDML interface (ROOT side).

Graphics & GL Timur will be at CERN for 3 weeks to work on the integration of his 3D algorithms in padgl and the GL viewer.