Use of variable size array to model the Atlas raw data

Slides:



Advertisements
Similar presentations
Naming Computer Engineering Department Distributed Systems Course Asst. Prof. Dr. Ahmet Sayar Kocaeli University - Fall 2014.
Advertisements

Objectivity Data Migration Marcin Nowak, CERN Database Group, CHEP 2003 March , La Jolla, California.
Multimedia Search and Retrieval: New Concepts, System Implementation, and Application Qian Huang, Atul Puri, Zhu Liu IEEE TRANSACTION ON CIRCUITS AND SYSTEMS.
Other File Systems: LFS and NFS. 2 Log-Structured File Systems The trend: CPUs are faster, RAM & caches are bigger –So, a lot of reads do not require.
1 Draft of a Matchmaking Service Chuang liu. 2 Matchmaking Service Matchmaking Service is a service to help service providers to advertising their service.
Introduction to Databases Transparencies 1. ©Pearson Education 2009 Objectives Common uses of database systems. Meaning of the term database. Meaning.
DHTML. What is DHTML?  DHTML is the combination of several built-in browser features in fourth generation browsers that enable a web page to be more.
2/10/2000 CHEP2000 Padova Italy The BaBar Online Databases George Zioulas SLAC For the BaBar Computing Group.
Cloud based storage. Cloud Storage Storage accessed by a web service API It is a block storage, it exposes its storage to clients as Raw storage that.
Information storage: Introduction of database 10/7/2004 Xiangming Mu.
Flashing Up the Storage Layer I. Koltsidas, S. D. Viglas (U of Edinburgh), VLDB 2008 Shimin Chen Big Data Reading Group.
The Pixel Detector ByteStream Converter M.Cobal (1), L. Santi (2) (1) University of Udine and INFN Trieste, Italy (2) University of Trieste and INFN Trieste,
1 Design and Performance of a Web Server Accelerator Eric Levy-Abegnoli, Arun Iyengar, Junehwa Song, and Daniel Dias INFOCOM ‘99.
July, 2001 High-dimensional indexing techniques Kesheng John Wu Ekow Otoo Arie Shoshani.
Test Of Distributed Data Quality Monitoring Of CMS Tracker Dataset H->ZZ->2e2mu with PileUp - 10,000 events ( ~ 50,000 hits for events) The monitoring.
Java SNMP Oplet Tal Lavian. Goals Portable across a range of devices Extensible Simple and convenient for client use Consistent with SNMP model Hide unnecessary.
Alignment Strategy for ATLAS: Detector Description and Database Issues
RELATIONAL FAULT TOLERANT INTERFACE TO HETEROGENEOUS DISTRIBUTED DATABASES Prof. Osama Abulnaja Afraa Khalifah
The Region of Interest Strategy for the ATLAS Second Level Trigger
Summary of 1 TB Milestone RD Schaffer Outline : u Goals and assumptions of the exercise u The hardware constraints u The basic model u What have we understood.
Multi-level Raid Multi-level Raid 2 Agenda Background -Definitions -What is it? -Why would anyone want it? Design Issues -Configuration and.
NOVA Networked Object-based EnVironment for Analysis P. Nevski, A. Vaniachine, T. Wenaus NOVA is a project to develop distributed object oriented physics.
Introduction to DFS. Distributed File Systems A file system whose clients, servers and storage devices are dispersed among the machines of a distributed.
The Vesta Parallel File System Peter F. Corbett Dror G. Feithlson.
ALICE Simulation Framework Ivana Hrivnacova 1 and Andreas Morsch 2 1 NPI ASCR, Rez, Czech Republic 2 CERN, Geneva, Switzerland For the ALICE Collaboration.
EXAMINATION OF CORRUPTED DATA IN THE TILE CALORIMETER Stephanie Hamilton Michigan State University The ATLAS Collaboration Supervisor: Irene Vichou (U.
Calorimeter Assisted Track Finder Tracking Infrastructure Dmitry Onoprienko Kansas State University Linear Collider Workshop 2007 May 30 – June 3, 2007.
Integration of the ATLAS Tag Database with Data Management and Analysis Components Caitriana Nicholson University of Glasgow 3 rd September 2007 CHEP,
Seminar on Service Oriented Architecture Distributed Systems Architectural Models From Coulouris, 5 th Ed. SOA Seminar Coulouris 5Ed.1.
Online Monitoring System at KLOE Alessandra Doria INFN - Napoli for the KLOE collaboration CHEP 2000 Padova, 7-11 February 2000 NAPOLI.
Virtual Memory Pranav Shah CS147 - Sin Min Lee. Concept of Virtual Memory Purpose of Virtual Memory - to use hard disk as an extension of RAM. Personal.
27 March 2003RD Schaffer & C. Arnault CHEP031 Use of a Generic Identification Scheme Connecting Events and Detector Description in Atlas  Authors: C.
January 20, 2000K. Sliwa/ Tufts University DOE/NSF ATLAS Review 1 SIMULATION OF DAILY ACTIVITITIES AT REGIONAL CENTERS MONARC Collaboration Alexander Nazarenko.
Object Oriented reconstruction of the CMS muon chambers CHEP February, Padova Annalina Vitelli - INFN Torino.
AMS02 Data Volume, Staging and Archiving Issues AMS Computing Meeting CERN April 8, 2002 Alexei Klimentov.
Mary Ganesan and Lora Strother Campus Tours Using a Mobile Device.
Data Grids, Digital Libraries and Persistent Archives: An Integrated Approach to Publishing, Sharing and Archiving Data. Written By: R. Moore, A. Rajasekar,
Databases and DBMSs Todd S. Bacastow January 2005.
DHTML.
KID - KLOE Integrated Dataflow
Virtual memory.
Random access memory Sequential circuits all depend upon the presence of memory. A flip-flop can store one bit of information. A register can store a single.
Scalability to Hundreds of Clients in HEP Object Databases
Random access memory Sequential circuits all depend upon the presence of memory. A flip-flop can store one bit of information. A register can store a single.
The COMPASS event store in 2002
MVC and other n-tier Architectures
Paging and Segmentation
Grid Canada Testbed using HEP applications
RAID RAID Mukesh N Tekwani
O.S Lecture 13 Virtual Memory.
Vincenzo Innocente CERN/EP/CMC
Data, Databases, and DBMSs
Cluster Resource Management: A Scalable Approach
Computer Architecture
Introduction to Databases Transparencies
TECHNICAL SEMINAR PRESENTATION
Distributed File Systems
Distributed File Systems
A Component-based Architecture for Mobile Information Access
Building ASP.NET Applications
Distributed File Systems
Chapter 15: File System Internals
RAID RAID Mukesh N Tekwani April 23, 2019
Distributed File Systems
Use of GEANT4 in CMS The OSCAR Project
Detector description News
Caching.
TensorFlow: A System for Large-Scale Machine Learning
Distributed File Systems
Disks Magnetic (hard) Disk arrays are used for reliable storage (RAID)
Presentation transcript:

Use of variable size array to model the Atlas raw data Simona Rolli - TUFTS University R.D. Schaffer - LAL/Orsay Martin Schaller - CERN/Innsbruck Andrea Salvadori - CERN CHEP, Padova, 02-10-00 Simona Rolli - TUFTS

N N 1 1 1 N Introduction PEvent PEventObjVector PEvtObj The current ATLAS event data model has been developed for raw data only. The classes providing access to the raw data (through an identifiers-based method) are the following: PEvent PEventObjVector PEvtObj PNode3 PNode2 PNode1 DetectorElement Digit implemented as Varray each channel response is organized in digits contained by an object representing the detector element which produced it. N N 1 1 1 N CHEP, Padova, 02-10-00 Simona Rolli - TUFTS

wafer_id = “barrel / layer 1 / ring 3 / phi sector 5” Logical Identifiers Hierarchical Identifiers are used to identify sensitive detector elements and read-out channels: the Identifiers are structured according to the logical structure of the detector: layers phi sectors barrel end-cap eta phi Pixel wafer wafer_id pixel_id rings wafer_id = “barrel / layer 1 / ring 3 / phi sector 5” pixel_id = wafer_id + “eta 19 / phi 33” CHEP, Padova, 02-10-00 Simona Rolli - TUFTS

Logical Identifiers (2) Identifiers provide a means of mapping detector-related information across domains or data stores: event raw data <-> detector description event raw data <-> calibration/alignment They also provide a means of selecting the information of interest: we have introduced a Range class for Identifier ranges: Pixel wafer eta phi id1 id2 Range(id1, id2) CHEP, Padova, 02-10-00 Simona Rolli - TUFTS

The Original Model A generalized model ( modeled after the original model) assumes a further node -PNode4 ( collection of events) -and the following fan out: 300 Pnode3, 10 PNode2, 100 Pnode1, 50 PDigit Server and Local/Remote client: 2*Sun UltraSparcII 399MHz, 512MB RAM - Hitachy RAID ( 25MB/sec) Remote Server : SUN UltraSparc II, 270MHz, 192 MB RAM, Elite23 drives - 9MB/sec. user time ~ 60-80% real time CHEP, Padova, 02-10-00 Simona Rolli - TUFTS

The Modified Model The Digit class has been moved to be attached to PNode2 (PEvtObj), through VArrays, while the association-based mechanism of navigation is implemented through arrays of references. user time ~ 60-80% real time CHEP, Padova, 02-10-00 Simona Rolli - TUFTS

The New Model The Pdigit class has been moved to be attached to PNode2 (PEvtObj), through a variable size arrays - SegVArray - The relationship between PNode3/PNode4 and PNode3/PNode2 is through Objy 1-to-many associations. CHEP, Padova, 02-10-00 Simona Rolli - TUFTS

SegVArray The class SegVArray is an extension of the ooVArray class ( same interface) : it contains a ooVArray of SVArraySegments, each containing an ooVArray of the objects to be stored. The advantages are that if only some objects are to be read in, only the segments are read that contains these objects ( ooVArray are always fetched as a whole). If a segment of a SegVArray fits onto a page a SegVArray can span several pages without the storage overhead of half page for large objects. Of course there might be poor performances if all the objects are read from the SegVArray ( further level of indirection) - partially overcome with the use of Iterators. CHEP, Padova, 02-10-00 Simona Rolli - TUFTS

Results user time ~ 15-30% real time CHEP, Padova, 02-10-00 Simona Rolli - TUFTS

Conclusions Performances improve when only a few digits are accessed by the application ( true for local access, for remote access -AMS- there is a further constraint from the internal page caching mechanism in Objectivity/DB) Well in the budget requesting at least 1 MB/sec per a standard reconstruction application. The model has been implemented in the real code and we are in the process of testing it with a large volume of data. Further studies needed in order to optimize the use of these type of arrays. CHEP, Padova, 02-10-00 Simona Rolli - TUFTS