Stanford Linear Accelerator Center R. D. Hall1 EPICS Collaboration Mtg Oct. 13-14, 2007 Oracle Archiver Past Experience Lessons Learned for Future EPICS.

Slides:



Advertisements
Similar presentations
Managed by UT-Battelle for the Department of Energy Best Ever Archive Utility, Yet (BEAUtY) Kay Kasemir April 2013.
Advertisements

The Architecture of Oracle
BY LECTURER/ AISHA DAWOOD DW Lab # 3 Overview of Extraction, Transformation, and Loading.
18 Copyright © Oracle Corporation, All rights reserved. Transporting Data Between Databases.
CIT 613: Relational Database Development using SQL Introduction to SQL.
Let’s try Oracle. Accessing Oracle The Oracle system, like the SQL Server system, is client / server. For SQL Server, –the client is the Query Analyser.
PHP (2) – Functions, Arrays, Databases, and sessions.
1 - Oracle Server Architecture Overview
NLC - The Next Linear Collider Project Lee Ann Yasukawa 05/25/99 NLC Archiving Requirements (Preliminary)
Database Systems: Design, Implementation, and Management Eighth Edition Chapter 11 Database Performance Tuning and Query Optimization.
Chapter 7 Managing Data Sources. ASP.NET 2.0, Third Edition2.
Working with SQL and PL/SQL/ Session 1 / 1 of 27 SQL Server Architecture.
IS 4510 – Database Administration Module – 2 Database Backup 10/24/20141Compiled by: Zafar Iqbal Khan.
Introduction To Databases IDIA 618 Fall 2014 Bridget M. Blodgett.
Advance Computer Programming Java Database Connectivity (JDBC) – In order to connect a Java application to a database, you need to use a JDBC driver. –
7 Copyright © 2006, Oracle. All rights reserved. Dealing with Database Corruption.
Lecture 3 – Data Storage with XML+AJAX and MySQL+socket.io
Channel Archiver Stats & Problems Kay Kasemir, Greg Lawson, Jeff Patton Presented by Xiaosong Geng (ORNL/SNS) March 2008.
Database Systems: Design, Implementation, and Management Tenth Edition Chapter 11 Database Performance Tuning and Query Optimization.
Database Systems: Design, Implementation, and Management Eighth Edition Chapter 10 Database Performance Tuning and Query Optimization.
Chapter Oracle Server An Oracle Server consists of an Oracle database (stored data, control and log files.) The Server will support SQL to define.
5 Copyright © 2004, Oracle. All rights reserved. Using Recovery Manager.
Database Technical Session By: Prof. Adarsh Patel.
Stanford Linear Accelerator Center November 15, 2000Lee Ann Yasukawa1 Archive Data to ORACLE The Prototype PEPII model.
Data File Access API : Under the Hood Simon Horwith CTO Etrilogy Ltd.
By Lecturer / Aisha Dawood 1.  You can control the number of dispatcher processes in the instance. Unlike the number of shared servers, the number of.
9 Copyright © 2004, Oracle. All rights reserved. Flashback Database.
The protection of the DB against intentional or unintentional threats using computer-based or non- computer-based controls. Database Security – Part 2.
NMED 3850 A Advanced Online Design January 12, 2010 V. Mahadevan.
Stanford Linear Accelerator Center R. Hall/L. Yasukawa1 EPICS Collaboration Mtg May 21, 2002 Oracle Storage for the Channel Archiver Managing Channel Archiver.
Introduction to the new mainframe © Copyright IBM Corp., All rights reserved. Chapter 12 Understanding database managers on z/OS.
Module 13 Implementing Business Continuity. Module Overview Protecting and Recovering Content Working with Backup and Restore for Disaster Recovery Implementing.
Database Design and Management CPTG /23/2015Chapter 12 of 38 Functions of a Database Store data Store data School: student records, class schedules,
The Client/Server Database Environment Ployphan Sornsuwit KPRU Ref.
Lecture2: Database Environment Prepared by L. Nouf Almujally 1 Ref. Chapter2 Lecture2.
Module 4 Designing and Implementing Views. Module Overview Introduction to Views Creating and Managing Views Performance Considerations for Views.
8 1 Chapter 8 Advanced SQL Database Systems: Design, Implementation, and Management, Seventh Edition, Rob and Coronel.
1 Biometric Databases. 2 Overview Problems associated with Biometric databases Some practical solutions Some existing DBMS.
DDM Monitoring David Cameron Pedro Salgado Ricardo Rocha.
11 Copyright © 2004, Oracle. All rights reserved. Dealing with Database Corruption.
3 Copyright © 2004, Oracle. All rights reserved. Working in the Forms Developer Environment.
SQL/Lesson 7/Slide 1 of 32 Implementing Indexes Objectives In this lesson, you will learn to: * Create a clustered index * Create a nonclustered index.
3 Copyright © 2006, Oracle. All rights reserved. Using Recovery Manager.
Visual Basic for Application - Microsoft Access 2003 Finishing the application.
CIT 613: Relational Database Development using SQL Introduction to SQL DeSiaMorePowered by DeSiaMore 1.
Student Centered ODS ETL Processing. Insert Search for rows not previously in the database within a snapshot type for a specific subject and year Search.
A Guide to SQL, Eighth Edition Chapter Six Updating Data.
Distributed Logging Facility Castor External Operation Workshop, CERN, November 14th 2006 Dennis Waldron CERN / IT.
Oct HPS Collaboration Meeting Jeremy McCormick (SLAC) HPS Web 2.0 OR Web Apps and Databases (Oh My!) Jeremy McCormick (SLAC)
November 1, 2004 ElizabethGallas -- D0 Luminosity Db 1 D0 Luminosity Database: Checklist for Production Elizabeth Gallas Fermilab Computing Division /
Bigtable: A Distributed Storage System for Structured Data
The DCS Databases Peter Chochula. 31/05/2005Peter Chochula 2 Outline PVSS basics (boring topic but useful if one wants to understand the DCS data flow)
Channel Archiver Overview Jan Channel Archiver Channel Access client Stores samples in disk files Design target: handle values/sec Documentation,
Unit-8 Introduction Of MySql. Types of table in PHP MySQL supports various of table types or storage engines to allow you to optimize your database. The.
19 Copyright © 2004, Oracle. All rights reserved. Database Backups.
ADO .NET from. ADO .NET from “ADO .Net” Evolution/History of ADO.NET MICROSOFT .NET “ADO .Net” Evolution/History of ADO.NET History: Most applications.
ISC321 Database Systems I Chapter 2: Overview of Database Languages and Architectures Fall 2015 Dr. Abdullah Almutairi.
23 Copyright © 2004, Oracle. All rights reserved. Preferring the Past: Flashback.
Oracle Database Architectural Components
SQL IMPLEMENTATION & ADMINISTRATION Indexing & Views.
Introduction to Partitioning in SQL Server
Managing Multi-User Databases
Client/Server Databases and the Oracle 10g Relational Database
Database System Concepts and Architecture
Database Performance Tuning &
Creating an Oracle Database
The Client/Server Database Environment
Database Performance Tuning and Query Optimization
Database System Concepts and Architecture
Chapter 11 Database Performance Tuning and Query Optimization
Presentation transcript:

Stanford Linear Accelerator Center R. D. Hall1 EPICS Collaboration Mtg Oct , 2007 Oracle Archiver Past Experience Lessons Learned for Future EPICS Channel Archiver RDB Development

R. D. HallEPICS Collaboration Mtg Oct , Topics Background System Elements Key Technologies Retrieval Considerations Table Structure Summary

R. D. HallEPICS Collaboration Mtg Oct , Background Designed and developed from late 2001 through 2002 by Lee Ann Yasukawa and Bob Hall utilizing version 1 of the EPICS Channel Archiver Thomas Birke of BESSY developed the Oracle LibIO C++ classes used to support retrieval of Oracle data Was in production use at SLAC from February 2003 until the end of 2004

R. D. HallEPICS Collaboration Mtg Oct , Factors in Discontinuing Use The new version 2 of the Channel Archiver became available with many improvements. We could also use the Archiver Viewer with this version, which gave our users an improved browser. The retrieval speed of the file-based version 2 channel archiver was better than the Oracle archiver. Discovered in late 2004 that the Stanford license for Oracle did not cover our use of the Oracle partitioning feature at SLAC, which would cost our group approximately $ 10 K.

R. D. HallEPICS Collaboration Mtg Oct , System Elements Oracle Channel Archiver Engine based on the version 1 EPICS Channel Archiver Channel Archiver Monitor program running every 5 minutes to check the health of the Archiver Engine by reading the Engine log. Archive Browser written in Python and using SWIG to be able to call LibIO C++ routines. Capable of browsing scalar data in a plot window and storing data of interest into Matlab files for further analysis. Data may also be viewed using the version 1 EPICS Channel Archiver CGIExport facility. Matlab Archiver Retrieval facility written in C using Matlab MEX to retrieve Oracle Archiver data via function calls from within Matlab.

R. D. HallEPICS Collaboration Mtg Oct , Oracle Archiver PEPII Archiver ( slcs2) TARF Archiver ( opi00gtw04) 8 Pack Archiver ( opi00gtw05) 2-T3 Disks RAID Controller SCS AFS File Server Archive Browser Web Server Web Browsers (for CGI Export) Oracle 9i Instance Apache Listener SQL Listener Retrieve Oracle 9i Client MCC Computer Room

R. D. HallEPICS Collaboration Mtg Oct , Key Technologies Oracle Application Direct Path Storage Oracle Partitioning

R. D. HallEPICS Collaboration Mtg Oct , Direct Path Storage The Oracle direct path mechanism bypasses the normal, conventional-path Oracle buffering mechanism used with conventional SQL INSERT statements to write directly to Oracle data files. Results in dramatic increases in data storage rates when loading large amounts of data into Oracle. During scalar EPICS data performance testing over 16,000 samples per second could be stored.

R. D. HallEPICS Collaboration Mtg Oct , Direct Path Storage Oracle Archive Engine calls C routines that call Oracle Call Interface (OCI) routines to store data into Oracle tables using the direct path mechanism. OCI routines are low-level, C callable routines optimized for performance and provide the only API available for using direct path storage. OCI routines are quite low-level and require considerable programmer time and effort to call correctly.

R. D. HallEPICS Collaboration Mtg Oct , Direct Path Storage Blocks of data (e.g., 2 K rows) are stored together, not individual rows. This means that data is not stored until a block is full or logic is added to flush partially filled blocks periodically. Direct path loading locks the storage table for any other Oracle operations other than non-indexed SELECT statements. Therefore, active data storage tables cannot be indexed and other methods must be used to improve data retrieval performance for these tables (e.g. partitioning). A restricted number of column datatypes are supported for direct path load operations.

R. D. HallEPICS Collaboration Mtg Oct , Partitioning Partitioning allows active data storage and archive data storage tables to be subdivided by time ranges. This feature can dramatically speed data retrieval requests due to partition elimination. That is, requests for data for a specified time span from a partitioned table subdivided by time ranges allows Oracle to search only those partitions that may contain data for the specified time range rather than the whole table. This is critical to retrieval performance when data tables become very large.

R. D. HallEPICS Collaboration Mtg Oct , Partitioning Partitioning can allow simplified data maintenance. Maintenance operations that need to be performed on only a subset of partitions do not need to be applied to an entire table. Data from old partitions may be conveniently stored to table and dropped from online storage.

R. D. HallEPICS Collaboration Mtg Oct , Partitioning At SLAC relatively small 10 minute partitions were used for the active storage data tables to allow fast retrieval of the most recent data, which tends to be the data most requested by users. After midnight each night a job ran which effectively moved the data in the 10 minute partitions of the previous day for each data storage table into a day long archive partition, which was then indexed to improve data retrieval performance. This strategy allowed reasonably fast data retrieval while keeping the number of partitions manageable.

R. D. HallEPICS Collaboration Mtg Oct , Daily Processing of Partitions This was accomplished by copying all partitions for the previous day into a temporary table, validating the data in the table and creating indexes, and then exchanging the temporary table (along with the indexes) with a partition in the archived data table. This method allowed the use of the Oracle PL/SQL “alter table exchange partition” command, which is very fast and does not involve the physical copying of data.

R. D. HallEPICS Collaboration Mtg Oct , Daily Processing of Partitions (Scalar Data) All partitions for yesterday are copied to a temporary table The data is validated and indexes are created arch_data_f arch_data_i temp_data_f temp_data_i The temporary table, along with indexes, is exchanged with a partition in the archived data tables archive_data_f archive_data_i ~10 min partitions daily partitions

R. D. HallEPICS Collaboration Mtg Oct , Retrieval Considerations Data retrieval is accomplished by SQL SELECT statements made with OCI routine calls. The SELECT statement contains the channel identification and the time range of interest. There are two tables for each supported data type: an active data table and an archive data table. Direct path loading is done in an active data table, where indexing is not allowed.

R. D. HallEPICS Collaboration Mtg Oct , Retrieval Considerations Since there are two tables for each supported data type, an Oracle view is created to retrieve data for each data type. An Oracle view is a logical table comprised of one or more physical tables. The retrieval SQL queries the view instead of querying the active and archive data tables directly. The views are created as read-only. Views allow flexibility as to what data the user has access to. That is, the definition of the view may be changed to retrieve data in a different way without changes to retrieval code.

R. D. HallEPICS Collaboration Mtg Oct , Retrieval Considerations In addition to the use of timestamp range partitioning, the use of indexes for archive data is crucial to retrieval performance. The archive data tables are indexed by a combination of timestamp and channel identification for best retrieval performance.

R. D. HallEPICS Collaboration Mtg Oct , Table Structure Overview arch_pvname arch_table_data pv_id arch_data_i (scalar integer) arch_data_f (scalar float) archive_wave_i (waveform integer) pv_name pv_starttime pv_id pv_endtime pv_active pv_id data_start data_table data_end pv_id stat value sevr timestamp ostat nanosecs archive_wave_f (waveform float) arch_meta pv_id

R. D. HallEPICS Collaboration Mtg Oct , Summary There were some limitations to the Oracle Archiver that we did not have time to address (e.g., the ability to store strings). Also there were many ideas for new features that were not implemented. A database related example is the creation of an Oracle management web interface to specify the partition scheme and manage user connections. The SLAC Oracle Archiver project was successful. However, the retrieval performance was only “acceptable” and not nearly as fast as version 2 of the EPICS file-based Channel Archiver. This was the area of greatest concern. At the time the project was abandoned, we did not run out of ideas to experiment with to improve retrieval performance, such as creating better indexes and changing the partition characteristics.