11-Nov-2008 1 Distr. DB Operations workshop - November 2008 The PVSS Oracle DB Archive in ATLAS ( life cycle of the data ) Gancho Dimitrov (LBNL)

Slides:



Advertisements
Similar presentations
Rollback Segments Nilendu Misra (MAR99)
Advertisements

CIT 613: Relational Database Development using SQL Revision of Tables and Data Types.
Chris Ruel * Image used without permission.
BY LECTURER/ AISHA DAWOOD DW Lab # 3 Overview of Extraction, Transformation, and Loading.
Filegroup “Stage A” Filegroup “Stage A” Filegroup “A” Partition 1,2 Filegroup “B” Partition 3,4 Filegroup “C” Partition 5,6 Filegroup “D” Partition.
CERN IT Department CH-1211 Geneva 23 Switzerland t Marcin Blaszczyk, IT-DB Atlas standby database tests February.
1 Databases in ALICE L.Betev LCG Database Deployment and Persistency Workshop Geneva, October 17, 2005.
Distributed Databases Logical next step in geographically dispersed organisations goal is to provide location transparency starting point = a set of decentralised.
IS 4420 Database Fundamentals Chapter 6: Physical Database Design and Performance Leon Chen.
Harvard University Oracle Database Administration Session 2 System Level.
Chapter 9 Overview  Reasons to monitor SQL Server  Performance Monitoring and Tuning  Tools for Monitoring SQL Server  Common Monitoring and Tuning.
DAT702.  Standard Query Language  Ability to access and manipulate databases ◦ Retrieve data ◦ Insert, delete, update records ◦ Create and set permissions.
1 RAL Status and Plans Carmine Cioffi Database Administrator and Developer 3D Workshop, CERN, November 2009.
DB Audit Expert v1.1 for Oracle Copyright © SoftTree Technologies, Inc. This presentation is for DB Audit Expert for Oracle version 1.1 which.
Phil Brewster  One of the first steps – identify the proper data types  Decide how data (in columns) should be stored and used.
07-June Database futures workshop, June 2011 CERN Enhance the ATLAS database applications by using the new Oracle 11g features Gancho Dimitrov.
From Olivier to commissioning team plans for the start-up of regular operations of LHCb 30/06 to 4/07 : Global commissioning week, all detectors, full.
Chapter 6 Physical Database Design. Introduction The purpose of physical database design is to translate the logical description of data into the technical.
05-June CERN Oracle tutorials 2013 Database solutions to ATLAS specific application requirements (real-life examples) Gancho Dimitrov (CERN) G.
CERN - IT Department CH-1211 Genève 23 Switzerland t The High Performance Archiver for the LHC Experiments Manuel Gonzalez Berges CERN, Geneva.
IT The Relational DBMS Section 06. Relational Database Theory Physical Database Design.
Chapter Oracle Server An Oracle Server consists of an Oracle database (stored data, control and log files.) The Server will support SQL to define.
ATONR Database Tests – 17th Feb. Schedule and Goals & Results Florbela Viegas, CERN ADP 18/09/20151.
Physical Database Design & Performance. Optimizing for Query Performance For DBs with high retrieval traffic as compared to maintenance traffic, optimizing.
TM 7-1 Copyright © 1999 Addison Wesley Longman, Inc. Physical Database Design.
Extents, segments and blocks in detail. Database structure Database Table spaces Segment Extent Oracle block O/S block Data file logical physical.
Oracle Index study for Event TAG DB M. Boschini S. Della Torre
Architecture Rajesh. Components of Database Engine.
Chapter 6 1 © Prentice Hall, 2002 The Physical Design Stage of SDLC (figures 2.4, 2.5 revisited) Project Identification and Selection Project Initiation.
ATLAS Detector Description Database Vakho Tsulaia University of Pittsburgh 3D workshop, CERN 14-Dec-2004.
André Augustinus 21 June 2004 DCS Workshop Detector DCS overview Status and Progress.
CERN Physics Database Services and Plans Maria Girone, CERN-IT
Database Management COP4540, SCS, FIU Physical Database Design (ch. 16 & ch. 3)
Database structure and space Management. Database Structure An ORACLE database has both a physical and logical structure. By separating physical and logical.
Week 7 : Chapter 7 Agenda SQL 710 Maintenance Plan:
Database structure and space Management. Segments The level of logical database storage above an extent is called a segment. A segment is a set of extents.
08-Nov Database TEG workshop, Nov 2011 ATLAS Oracle database applications and plans for use of the Oracle 11g enhancements Gancho Dimitrov.
Peter Chochula ALICE Offline Week, October 04,2005 External access to the ALICE DCS archives.
WLCG Service Report ~~~ WLCG Management Board, 16 th December 2008.
Physical Database Design Purpose- translate the logical description of data into the technical specifications for storing and retrieving data Goal - create.
17-Oct CHEP conference, Amsterdam Oct 2013 Next generation database relational solutions for ATLAS Distributed Computing Gancho Dimitrov (CERN)
Roy Ernest Database Administrator Pinnacle Sports Worldwide
16-May ADC technical interchange meeting Tokyo, May 2013 Database aspects of ATLAS distributed computing Gancho Dimitrov (CERN) G. Dimitrov.
Distributed Logging Facility Castor External Operation Workshop, CERN, November 14th 2006 Dennis Waldron CERN / IT.
CERN IT Department CH-1211 Genève 23 Switzerland t COOL performance optimization using Oracle hints Andrea Valassi and Romain Basset (IT-DM)
Maria Girone CERN - IT Tier0 plans and security and backup policy proposals Maria Girone, CERN IT-PSS.
November 1, 2004 ElizabethGallas -- D0 Luminosity Db 1 D0 Luminosity Database: Checklist for Production Elizabeth Gallas Fermilab Computing Division /
Copyright © 2009 Rolta International, Inc., All Rights Reserved Michael R. Messina, Management Consultant Rolta-TUSC, Oracle Open World 2009 (60 min) ID#:
11th November Richard Hawkings Richard Hawkings (CERN) ATLAS reconstruction jobs & conditions DB access  Conditions database basic concepts  Types.
CERN IT Department CH-1211 Genève 23 Switzerland t COOL Performance Tests ATLAS Conditions Database example Romain Basset, IT-DM October.
The DCS Databases Peter Chochula. 31/05/2005Peter Chochula 2 Outline PVSS basics (boring topic but useful if one wants to understand the DCS data flow)
BNL dCache Status and Plan CHEP07: September 2-7, 2007 Zhenping (Jane) Liu for the BNL RACF Storage Group.
Database Issues Peter Chochula 7 th DCS Workshop, June 16, 2003.
Maria del Carmen Barandela Pazos CERN CHEP 2-7 Sep 2007 Victoria LHCb Online Interface to the Conditions Database.
uses of DB systems DB environment DB structure Codd’s rules current common RDBMs implementations.
The Database Project a starting work by Arnauld Albert, Cristiano Bozza.
In quest of the operational database for real-time environmental monitoring and early warning systems Bartosz Baliś, Marian Bubak, Daniel Harezlak, Piotr.
Oracle structures on database applications development
Module 11: File Structure
Database structure and space Management
PVSS Evolution in Relation to Databases
Database Services at CERN Status Update
Elizabeth Gallas - Oxford ADC Weekly September 13, 2011
Translation of ER-diagram into Relational Schema
CHAPTER 5: PHYSICAL DATABASE DESIGN AND PERFORMANCE
Case studies – Atlas and PVSS Oracle archiver
Data Lifecycle Review and Outlook
Physical Database Design
Lecture 20: Representing Data Elements
Presentation transcript:

11-Nov Distr. DB Operations workshop - November 2008 The PVSS Oracle DB Archive in ATLAS ( life cycle of the data ) Gancho Dimitrov (LBNL)

11-Nov Outline  Introduction to the PVSS system  Nature of the data and organization  The ATLAS PVSS schemas and EVENTHISTORY table description  The need of having PVSS data replication from ATONR to ATLR  Some optimizations  Data volumes  Conclusions

11-Nov Introduction to the PVSS system and its use in ATLAS PVSS (Prozessvisualisierungs- und Steuerungssystem) is a Control and Data Acquisition system, developed by the Austrian company ETM (now 100% owned by Siemens AG). In year 2000, it has been chosen as a control system for the LHC experiments The ATLAS detector The ATLAS ‘online’ Oracle DB ATONR Thousands of data point elements PVSS Oracle archive - keeps history of the detector status, e.g. high voltages, temperatures

11-Nov Nature of the data and organization The data representation in the PVSS Oracle archive is simple: - data point element ID - time stamp - value VALUE_NUMBER ( BINARY_DOUBLE ) or VALUE_STRING (VARHAR2(4000)) - plus few more additional columns The primary key is based on the ELEMENT ID and the TIMESTAMP

11-Nov The ATLAS PVSS DB accounts and table desc. A database schema per subdetector (as total 14) Table is ‘switched’ when it reaches a certain size and a view is updated to keep them together for the application to access the data ( the EVENTHISTORY view) The row length is in the range bytes Data point elements, in the LAR case are about 4500 Not used from ATLAS, get NULL values, thus do not take occupy space

11-Nov The need of having PVSS data replication from ATONR to ATLR (‘online’ => ‘offline’) In order to have the PVSS data accessible for the sub-detector expert analysis from the CERN public network and even from outside CERN a need for its replication showed up. After series of testing a production replication of the PVSS data was introduced in the beginning of July Firewall to be put in place in Feb The PVSS replication The COOL replication

11-Nov Replicating the PVSS Archive  Load on the replication : avg. 4 GB per day of data  Replication activity can be measured in LCR/sec (Logical Change Record)  The limit on the APPLY on the destination DB is about 3000 LCR/sec  Activity above rows/sec is considered over the normal and we work with the sub-detectors to keep the rate within that range Avg inserted rows per second for Sept. 2008Avg inserted rows per second for Oct. 2008

11-Nov Snapshot from the PVSS_APPLY in a moment of compensating the backlog

11-Nov Optimizations on the ATLAS PVSS Archive  Compression on the PVSS data – done on a regular basis on the old ‘switched history’ tables - (move/compress to a different TBS)  factor of 2 decrease in storage of the archived tables.  35 % saved space from the index compression Thus the overall PVSS space saving is about 45%  Intention to delete yearly window of data from the online server. Procedure still to be tested, as replication must block this operation on the offline.  Usage offline: PL/SQL libraries are being developed to control and optimize the access to the offline archive, so that free-form SQL will not overuse resources.

11-Nov PVSS data volumes  Every day PVSS ‘eats’ about 8 GB of the disk space (table + index segments)  Non-compressed PVSS data (estimated on ) ATONR: 606 GB ATLR: 593 GB The difference comes from the fact that on the ATONR the indexes are not compressed on the ‘current’ tables (default for the PVSS)  Compressed PVSS data - about 1/2 TB on each of the DBs  The policy is to keep the data for most recent 12 months on the online database ATONR (the older one will be deleted) and keep the replicated data on the ATLR forever.

11-Nov Current status  Currently several sub-detectors run in debug mode, thus sending more data to Oracle than is expected in stable running mode.  On the other side in running more we may expect upto 30% increase because of the instability into the detector conditions  With this in hand the maximum space we need would be 4TB. After applied compression, this requirements would 2,2 TB  For 2009 we request 3 TB and we believe it would be sufficient

11-Nov Conclusions  We got almost a factor of 2 decrease in the disk space usage by compressing the PVSS data  The PVSS replication worked quite well since it was put in production mode. However, we have to keep the insert rate in the reasonable range of LCR/sec that we have now. Work with the sub- detector experts is ongoing.  PVSS data retention on the ATONR agreed to be 12 months (sliding window). Procedure to be tested on the INTR database