Irina Makhlyueva ALICE DAQ group 28 February 2005 DATE multi-stream recorder.

Slides:



Advertisements
Similar presentations
More on Processes Chapter 3. Process image _the physical representation of a process in the OS _an address space consisting of code, data and stack segments.
Advertisements

CSCE430/830 Computer Architecture
Operating Systems ECE344 Ashvin Goel ECE University of Toronto Disks and RAID.
Operating-System Structures
Chapter 19: Network Management Business Data Communications, 4e.
Technical Architectures
Interrupts (contd..) Multiple I/O devices may be connected to the processor and the memory via a bus. Some or all of these devices may be capable of generating.
1 Fall 2005 Hardware Addressing and Frame Identification Qutaibah Malluhi CSE Department Qatar University.
Chapter 13 Embedded Systems
1/28/2004CSCI 315 Operating Systems Design1 Operating System Structures & Processes Notice: The slides for this lecture have been largely based on those.
Figure 1.1 Interaction between applications and the operating system.
CS533 - Concepts of Operating Systems
Chapter 3.7 Memory and I/O Systems. 2 Memory Management Only applies to languages with explicit memory management (C or C++) Memory problems are one of.
MSS, ALICE week, 21/9/041 A part of ALICE-DAQ for the Forward Detectors University of Athens Physics Department Annie BELOGIANNI, Paraskevi GANOTI, Filimon.
06/15/2009CALICE TB review RPC DHCAL 1m 3 test software: daq, event building, event display, analysis and simulation Lei Xia.
1 Lecture 4: Threads Operating System Fall Contents Overview: Processes & Threads Benefits of Threads Thread State and Operations User Thread.
Chapter 6 Operating System Support. This chapter describes how middleware is supported by the operating system facilities at the nodes of a distributed.
CH2 System models.
Module 10: Monitoring ISA Server Overview Monitoring Overview Configuring Alerts Configuring Session Monitoring Configuring Logging Configuring.
Chapter 2: Operating-System Structures. 2.2 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts Chapter 2: Operating-System Structures Operating.
Recall: Three I/O Methods Synchronous: Wait for I/O operation to complete. Asynchronous: Post I/O request and switch to other work. DMA (Direct Memory.
1 Alice DAQ Configuration DB
Guide to Linux Installation and Administration, 2e1 Chapter 2 Planning Your System.
Guide to Linux Installation and Administration, 2e1 Chapter 10 Managing System Resources.
Bookkeeping Tutorial. Bookkeeping & Monitoring Tutorial2 Bookkeeping content  Contains records of all “jobs” and all “files” that are created by production.
The ALICE Data-Acquisition Software Framework DATE V5 F. Carena, W. Carena, S. Chapeland, R. Divià, I. Makhlyueva, J-C. Marin, K. Schossmaier, C. Soós,
Operating Systems David Goldschmidt, Ph.D. Computer Science The College of Saint Rose CIS 432.
ALICE Computing Model The ALICE raw data flow P. VANDE VYVRE – CERN/PH Computing Model WS – 09 Dec CERN.
Roberto Divià, CERN/ALICE 1 CHEP 2009, Prague, March 2009 The ALICE Online Data Storage System Roberto Divià (CERN), Ulrich Fuchs (CERN), Irina Makhlyueva.
Vertical Optimization Of Data Transmission For Mobile Wireless Terminals MICHAEL METHFESSEL, KAI F. DOMBROWSKI, PETER LANGENDORFER, HORST FRANKENFELDT,
Institute of Technology Sligo - Dept of Computing Sem 2 Chapter 12 Routing Protocols.
CE Operating Systems Lecture 13 Linux/Unix interprocess communication.
Overview of DAQ at CERN experiments E.Radicioni, INFN MICE Daq and Controls Workshop.
SKYPIAX, how to add Skype capabilities to FreeSWITCH (and Asterisk) CHICAGO, USA, September 2009.
EPICS Release 3.15 Bob Dalesio May 19, Features for 3.15 Support for large arrays - done for rsrv in 3.14 Channel access priorities - planned to.
XA R7.8 Link Manager How to Manage an R7.8 Environment Ruth Anne Pharr Sr. IT Consultant, CISTECH Inc.
VApp Product Support Engineering Rev E VMware Confidential.
Silberschatz, Galvin and Gagne  Operating System Concepts UNIT II Operating System Services.
GLOBAL EDGE SOFTWERE LTD1 R EMOTE F ILE S HARING - Ardhanareesh Aradhyamath.
A simple Desktop DAQ for U2F readout Ulf jörnmark Physics Dept. Lund Status and plans.
NTOF DAQ status D. Macina (EN-STI-EET) Acknowledgements: EN-STI-ECE section: A. Masi, A. Almeida Paiva, M. Donze, M. Fantuzzi, A. Giraud, F. Marazita,
Status & development of the software for CALICE-DAQ Tao Wu On behalf of UK Collaboration.
© 2004, D. J. Foreman 1 Device Mgmt. © 2004, D. J. Foreman 2 Device Management Organization  Multiple layers ■ Application ■ Operating System ■ Driver.
© 2004, D. J. Foreman 1 Device Mgmt. © 2004, D. J. Foreman 2 Device Management Organization  Multiple layers ■ Application ■ Operating System ■ Driver.
1 Transport Layer: Basics Outline Intro to transport UDP Congestion control basics.
System calls for Process management Process creation, termination, waiting.
Introduction Contain two or more CPU share common memory and peripherals. Provide greater system throughput. Multiple processor executing simultaneous.
Multimedia Retrieval Architecture Electrical Communication Engineering, Indian Institute of Science, Bangalore – , India Multimedia Retrieval Architecture.
Copyright 2007, Information Builders. Slide 1 iWay Web Services and WebFOCUS Consumption Michael Florkowski Information Builders.
1 Sheer volume and dynamic nature of video stresses network resources PIE: A lightweight latency control to address the buffer problem issue Rong Pan,
1 Device Controller I/O units typically consist of A mechanical component: the device itself An electronic component: the device controller or adapter.
Event Management. EMU Graham Heyes April Overview Background Requirements Solution Status.
R.Divià, CERN/ALICE 1 ALICE off-line week, CERN, 9 September 2002 DAQ-HLT software interface.
Interrupts and Exception Handling. Execution We are quite aware of the Fetch, Execute process of the control unit of the CPU –Fetch and instruction as.
CSCI/CMPE 4334 Operating Systems Review: Exam 1 1.
Cofax Scalability Document Version Scaling Cofax in General The scalability of Cofax is directly related to the system software, hardware and network.
Sem 2 v2 Chapter 12: Routing. Routers can be configured to use one or more IP routing protocols. Two of these IP routing protocols are RIP and IGRP. After.
Scalable Readout System Data Acquisition using LabVIEW Riccardo de Asmundis INFN Napoli [Certified LabVIEW Developer]
Fermilab Scientific Computing Division Fermi National Accelerator Laboratory, Batavia, Illinois, USA. Off-the-Shelf Hardware and Software DAQ Performance.
DAQ thoughts about upgrade 11/07/2012
Maintaining Windows Server 2008 File Services
LHC experiments Requirements and Concepts ALICE
Chapter 2: System Structures
Chapter 2: System Structures
Lecture Topics: 11/1 General Operating System Concepts Processes
Chapter 13: I/O Systems I/O Hardware Application I/O Interface
Device Mgmt © 2004, D. J. Foreman.
Device Mgmt © 2004, D. J. Foreman.
Chapter 13: I/O Systems.
Presentation transcript:

Irina Makhlyueva ALICE DAQ group 28 February 2005 DATE multi-stream recorder

Irina Makhlyueva – ALICE DAQ2 28 February 2005 Contents Purpose of DATE recorders  Throughput balancing Multiple stream recording DATE mStreamRecorder  Structure  Features  Configuration files Performance Plans

Irina Makhlyueva – ALICE DAQ3 28 February 2005 Purpose of DATE online recorders To provide an interface between EBs and CASTOR:  Event Builder by itself has only limited recording functions (local disk and “online recording” to FIFOs/buffers). Anything beyond that is the task of independent processes running on the GDC hosts – online recorders Data conversion option (raw event format→ ROOT) EB/CASTOR throughput balancing  To ensure a continuous data extraction from the EB buffers and make the best use of the available CASTOR throughput

Irina Makhlyueva – ALICE DAQ4 28 February 2005 EB/CASTOR throughput balancing EB’s main task is event building. The EB load balancing is performed by EDM. The total number of GDCs is tuned to optimize the DAQ performance. In theory, the CASTOR segment dedicated to ALICE should have a sufficient (or better) aggregate throughput to match the total data rate delivered by all GDCs. Recorders must not be a bottleneck! In particular, EBs should never get blocked due to the recorder latencies. Therefore, the recorder layer needs a sufficient bandwidth reserve to provide the best possible balance between GDCs and CASTOR throughputs. Currently, CASTOR is lagging behind DATE. In this situation, the important task of the recording layer is to make the best use of the CASTOR capacity. (*) Photo non contractuelle GDC farm (*)CASTOR (*) Total GDC throughput Aggregate CASTOR throughput recorder layer loss recorder bandwidth reserve

Irina Makhlyueva – ALICE DAQ5 28 February 2005 From mono- to multi-stream recording The past DC experience showed that GDCs with single recording streams could not fully load CASTOR. However, increasing the number of GDCs is expensive. The solution: a multiple stream recording, with >1 recording channel (stream) per GDC. This is also beneficial for the GDC efficiency: should a latency in one stream occur, the other stream(s) can take over. Recording layer capacity (BW) ≈ number of ACTIVE streams = total – blocked In practice, the optimal number of streams per GDC is dictated by the requirement of CASTOR saturation → its tuning is one of the DC goals castor Bad: CASTOR is underloaded ~1 stream/GDC DAQ Not good : no BW reserve ≈2 streams/GDC OK : adequate recorder BW, losses are due limited CASTOR BW. ≥3 streams/GDC

Irina Makhlyueva – ALICE DAQ6 28 February 2005 The new DATE multi-stream recorder Was introduced in August, 2004 and has passed extensive testing since then. in DATE since release 5.0 very useful for debugging/testing of new CASTOR facilities is used in the ongoing DC (3 streams/GDC, raw data mode) Event descriptors via SimpleFifo Events via EB Consumer API Fatal condition signals (interrupts) (optional) raw/ROOT transformation Dispatching algorithm n File destination assigned to stream n Reporting to infoLogger/Stat EB buffer GDC disp ….. 1 ……….. 2N Recorder shared memory (internal logging etc) stream (1) Config file mStreamRecorder Event Builder 1 2 N GDC host stream (2) stream (N)

Irina Makhlyueva – ALICE DAQ7 28 February 2005 mStreamRecorder: features The recorder consists of disp + N × stream processes, running on a GDC host EB is configured for online recording (recording device “:” in run parameters) Dispatching process disp:  started by the DATE run control  reads/interpretes the configuration file mStreamRecorder.config;  forks and configures n identical stream processes, distributes event descriptors from the EB between the streams according to a specified dispatching algorithm: “equal-load”, “first- available” or “special” (trigger-mask etc – not yet implemented).;  handles interrupts from the streams. Recording stream process:  totally independent of other streams, communicates only with disp via FIFO (ev. descriptors), signals (fatal conditions), shared memory (status reporting) and a pipe (configuration);  retrieves the events from the EB buffer, using eventBuilder API ;  (optionally, transforms the events into ROOT objects);  Opens, closes, writes files to a specified destination: local disk, CASTOR (RFIO or ROOTd);  Reports its status to disp via shared memory. disp and streams send log messages via DATE infoLogger Both disp and streams use polling with timeout/sleep (10 ms): very low CPU load, good response

Irina Makhlyueva – ALICE DAQ8 28 February 2005 mStreamRecorder: configuration Flexible, easily scalable configuration, currently based on a traditional ASCII config file Simple case the recorders for all GDCs have the same configuration:  3 identical streams  “first-available” dispatching method  all streams have the same properties (destinations, file sizes, polling regimes, buffering). A bit more sophisticated configuration:  Separate configurations for different GDCs (e.g., different number of streams/GDC)  A variety of streams with different destinations >COMMON method=2 Nstreams=3 loglevel=1 >RECORDERS default_rec stream=default_str >OSTREAMS default_str sleep=1 fsize=1024 mxrecl=0 \ pool=alimdc6 stager=lxs5007 \ path=/castor/cern.ch/alice/mdc6/tapes DC6 >COMMON loglevel=1 method=1 dump=1 >RECORDERS default_rec stream=default_str !! 1 stream tbed0084gdc Nstreams=2 !! 2 (default) streams tbed0021gdc stream=default_str stream=public \ stream=local !! 3 streams >OSTREAMS default_str sleep=1 fsize=1024 mxrecl=0 \ pool=alimdc6 stager=lxs5007 \ path=/castor/cern.ch/alice/mdc6/tapes public sleep=2 \ path=/castor/cern.ch/user/m/makhlyui \ pool=public stager=stagepublic fsize=128 local path=/tmp =public !! write to local /tmp Demo

Irina Makhlyueva – ALICE DAQ9 28 February 2005 Recorder performance Rough start with a transition of CASTOR to the new platform → the DC delays. E.g. (end of January): very long file open latency (from seconds to ~1 hour!). Multi-stream feature of the DATE recorder helped to push the CASTOR load to the extreme. Numerous modifications done to mStreamRecorder (protections, error recovery, internal communication, reporting, bugs) → a more robust and better performing code. Ongoing DC configuration: 15 LDCs, 40 GDCs and 3 streams/GDC. Very fresh results: an average data rate of ~300 MB/s, with large momentary fluctuations (CASTOR is being debugged in flight !) Initial debugging and timing was done using local disks and the old (“production version”) CASTOR. Smooth operation with up to 100 streams/GDC and 1-2 GDCs.

Irina Makhlyueva – ALICE DAQ10 28 February 2005 Plans for near-term future ROOT recording commissioning ……….. Transition to DB-based configuration (retaining ASCII option) ………………….. Full integration in the DATE Run Control (SOR.command → SMI object) ………….. Reporting to DATE status display ……… A detailed study of the overall recording performance, as function of # streams / GDC ROOT recording commissioning ……….. Transition to DB-based configuration (retaining ASCII option) ………………….. Full integration in the DATE Run Control (SOR.command → SMI object) ………….. Reporting to DATE status display ……… A detailed study of the overall recording performance, as function of # streams / GDC in progress to be done in progress during the DC in progress to be done in progress during the DC

Irina Makhlyueva – ALICE DAQ11 28 February 2005 Iguatzu Falls (Argentina). © Long-term

Irina Makhlyueva – ALICE DAQ12 28 February 2005 Thanks and Credits R. Divia for his help with the recorder debugging, the event builder API and other aspects of DATE P. Vande Vyvre for numerous discussions B. Panzer for the idea of multiple stream recording J-D Durand, O.Barring and other members of IT/ADC – for help in struggling with CASTOR K. Schossmaier and U.Fuchs, for help with DATE and Linux installation T. Kuhr and C. Cheshkov of ALICE offline group – for the ROOT recording codes and related stuff Photographs from WWW sites: (stream) (Waterfalls) The end

Irina Makhlyueva – ALICE DAQ13 28 February 2005 LDCs GDC pool CASTOR multiple recording streams multiple recording streams Spare slide