From the ChannelArchiver to the Best Ever Archive Utility, Yet July 2009.

Slides:



Advertisements
Similar presentations
Dicembre 2009Spes Control System Group CSS and Archiver: installation and setup Giovannini Loris 1.
Advertisements

Epics Configuration Management Steve Hunt v1.0. Goals Maximize control system availability Minimize development cycle time Reduce risk.
Managed by UT-Battelle for the Department of Energy Best Ever Archive Utility, Yet (BEAUtY) Kay Kasemir April 2013.
Control System Studio (CSS)
SNS Integrated Control System SNS RDB Requirements, wish list, status A little history l Oracle RDB used exclusively by accelerator physics group l JERI.
1 Trieste, April 17 th 2008Lucio Zambon Electronic Graphic Interface for Global Archiving Technology: PHP, MySQL, JavaScript, JPGraph, etc Development.
Best Ever Alarm System Toolkit Kay Kasemir, Xihui Chen, Katia Danilova SNS/ORNL April, 2013.
Update on CSS at SNS Xihui Chen, Kay Kasemir June 2010.
Archive Systems What you always wanted to know but were afraid to ask: What’s available? Who’s doing what? PAL EPICS Meeting Oct
Managed by UT-Battelle for the Department of Energy Kay Kasemir ORNL/SNS Jan Control System Studio Training - Alarm System Use.
Managed by UT-Battelle for the Department of Energy Kay Kasemir ORNL/SNS April 2013 Control System Studio Training - Alarm System Use.
NovaBACKUP 10 xSP Technical Training By: Nathan Fouarge
DØ Channel Archiver Tutorial V.Sirotenko, 4/4/2001.
RAID Redundancy is the factor for development of RAID in server environments. This allows for backup of the data in the storage in the event of failure.
CVSQL 2 The Design. System Overview System Components CVSQL Server –Three network interfaces –Modular data source provider framework –Decoupled SQL parsing.
Experimental Physics and Industrial Control System (EPICS) Alarm Handler (ALH) Bob Dalesio, August 14, 1999.
Managed by UT-Battelle for the Department of Energy Kay Kasemir ORNL/SNS April 2013 Control System Studio Training - Workspaces, Shared.
PPOUG, 05-OCT-01 Agenda RMAN Architecture Why Use RMAN? Implementation Decisions RMAN Oracle9i New Features.
Hands-On Microsoft Windows Server 2008 Chapter 1 Introduction to Windows Server 2008.
Fundamentals of Networking Discovery 1, Chapter 2 Operating Systems.
Managed by UT-Battelle for the Department of Energy Best Ever Archive Utility, Yet Creating a BEAST was easy. BEAUtY seems harder. April.
Loris Giovannini, Mauro Giacchini Epics Collaboration Meeting
Scan System Kay Kasemir, Xihui Chen Jan Managed by UT-Battelle for the U.S. Department of Energy Automated Experiment Control “Scan” should be.
Scan System: Experiment Automation Kay Kasemir, Xihui Chen RAL EPICS Meeting, May 2013.
Channel Archiver Stats & Problems Kay Kasemir, Greg Lawson, Jeff Patton Presented by Xiaosong Geng (ORNL/SNS) March 2008.
Channel Archiver Introduction 2006
AQS Web Quick Reference Guide Changing Raw Data Values Using Maintenance 1. From Main Menu, click Maintenance, Sample Values, Raw Data 2. Enter monitor.
Managed by UT-Battelle for the Department of Energy Kay Kasemir, Ph.D. ORNL/SNS July 2011 at Control System Studio - CSS - Overview.
Managed by UT-Battelle for the Department of Energy Kay Kasemir ORNL/SNS 2012, April at SLAC Control System Studio - Introduction.
Managed by UT-Battelle for the Department of Energy Kay Kasemir, Ph.D. ORNL/SNS July 2011 at Control System Studio - CSS - Overview.
Managed by UT-Battelle for the Department of Energy Kay Kasemir ORNL/SNS Jan Control System Studio Training - Web OPI.
Improving Efficiency of I/O Bound Systems More Memory, Better Caching Newer and Faster Disk Drives Set Object Access (SETOBJACC) Reorganize (RGZPFM) w/
Update on a New EPICS Archiver Kay Kasemir and Leo R. Dalesio 09/27/99.
Stanford Linear Accelerator Center R. Hall/L. Yasukawa1 EPICS Collaboration Mtg May 21, 2002 Oracle Storage for the Channel Archiver Managing Channel Archiver.
Managed by UT-Battelle for the Department of Energy Kay Kasemir ORNL/SNS Oct EPICS Meeting, PAL, Korea Control System Studio Training.
Control System Studio (CSS) Overview Kay Kasemir, July 2009.
The CSS Scan System Kay-Uwe Kasemir SNS/ORNL Dec
Stanford Linear Accelerator Center R. D. Hall1 EPICS Collaboration Mtg Oct , 2007 Oracle Archiver Past Experience Lessons Learned for Future EPICS.
Managed by UT-Battelle for the Department of Energy Kay Kasemir ORNL/SNS Jan Control System Studio Training - Archive System Setup.
Managed by UT-Battelle for the Department of Energy Kay Kasemir ORNL/SNS Oct EPICS Meeting, PAL, Korea Control System Studio Training.
Managed by UT-Battelle for the Department of Energy Kay Kasemir ORNL/SNS Jan Control System Studio, CSS Overview.
Dudok de Wit David.  Documents management in a deskless company  SharePoint Online as a solution  Redesigning the documentary organization  Interoperability.
Managed by UT-Battelle for the Department of Energy Kay Kasemir ORNL/SNS Oct EPICS Meeting, PAL, Korea Control System Studio Training.
Managed by UT-Battelle for the Department of Energy Kay Kasemir ORNL/SNS With slides from Xihui Chen May 2014 Control System Studio Training.
Managed by UT-Battelle for the Department of Energy CSS Update Matthias Clausen, Helge Rickens, Jan Hatje and DESY Delphy Armstrong, Xihui Chen,
Managed by UT-Battelle for the Department of Energy Kay Kasemir ORNL/SNS 2012, January 9-12 at NSRRC, Taiwan Control System Studio Training.
MySQL Gubbins Richard Sinclair HepSysMan – 11/05/2012.
Troubleshooting SQL Server Performance: Tips &Tools Amit Khandelwal.
1 KEKB Archiving Dec Tatsuro KEK.
Managed by UT-Battelle for the Department of Energy Kay Kasemir ORNL/SNS April 2013 Control System Studio, CSS Overview.
BROOKHAVEN SCIENCE ASSOCIATES Enhanced Directory Service Ralph Lange EPICS Collaboration Meeting October 11, 2009.
LHCb File-Metadata: Bookkeeping Carmine Cioffi Department of Physics, Oxford University UK Metadata Workshop Oxford, 04 July 2006.
Managed by UT-Battelle for the Department of Energy Kay Kasemir ORNL/SNS 2012, January 9-12 at NSRRC, Taiwan Control System Studio Training.
RDB Issues at SLAC Archiver Store General EPICS Support.
Channel Archiver Overview Jan Channel Archiver Channel Access client Stores samples in disk files Design target: handle values/sec Documentation,
Data Aging - Euthanasia Bob Dalesio. Data Management - Channel Data Allow recovery of unused data space Allow selective delete of channels Allow selective.
CSS Synoptic Display Comments Kay Kasemir, July 2009.
Managed by UT-Battelle for the Department of Energy Kay Kasemir ORNL/SNS May 2014 Control System Studio Training - Hierarchical Preferences.
Managed by UT-Battelle for the Department of Energy Kay Kasemir ORNL/SNS 2012, April at SLAC Control System Studio Training - Alarm System.
Implementation and Testing of RDB Channel Archiver with MySQL Richard Ma, DePauw University Supervisor: Richard Farnsworth, Argonne National Laboratory.
Your current Moodle 1.9 Minimum Requirements Ability to do a TEST RUN! Upgrading Moodle to Version 2 By Ramzan Jabbar Doncaster College for the Deaf By.
Managed by UT-Battelle for the Department of Energy Quest for the Best Ever Alarm System Tool Kay Kasemir Oct
Best Ever Alarm System Toolkit Kay Kasemir, Xihui Chen, Katia Danilova, SNS/ORNL ICALEPCS 2009, Kobe, Japan, Oct 2009.
DB Questions and Answers open session (comments during session) WLCG Collaboration Workshop, CERN Geneva, 24 of April 2008.
SNS Integrated Control System ORACLE –JERI DB Generation April 27, 2004 Coles Sibley Jeff Patton.
Managed by UT-Battelle for the Department of Energy Channel Archiver Update Oct Kay Kasemir
Flash Storage 101 Revolutionizing Databases
Control System Studio (CSS)
Channel Archiver Kay-Uwe Kasemir, LANL
Current State - and Replacement
Presentation transcript:

From the ChannelArchiver to the Best Ever Archive Utility, Yet July 2009

2Managed by UT-Battelle for the U.S. Department of Energy CSS-based OPI Archive Archive Engine config.xml IOC Binary Data Files Data Server “ASCII” Config. Channel Access XML-RPC Channel Archiver History ~2000: Started by Bob Dalesio ~2003: Index Tools, Data Server ~2007: CSS Client

3Managed by UT-Battelle for the U.S. Department of Energy Problems Data file format optimized to write many samples quickly – More then 40000/second –.. but we only used maybe 1000/sec –.. and many ill-configured or duplicate channels Headaches with data maintenance: – Scripts to restart engines, copy data, update indices. – Index time grows with data – Stuck when index files reach 2GB – SNS Users faced with ~80 sub-archives – No clue what needs fixing after network/power problems – No idea who contributes how many samples – No way to remove selected channels or time ranges – Improving on this means implementing an RDB

4Managed by UT-Battelle for the U.S. Department of Energy CSS-based OPI ArchiveEngine config.xml IOC RDB (Oracle/MySQL) Channel Access New Setup Samples Config. EngineConfig -Import Other tools for config & samples

5Managed by UT-Battelle for the U.S. Department of Energy CSS Data Browser Handles Both New URL Just one ‘RDB’ sub-archive Old and new data can be combined in one plot

6Managed by UT-Battelle for the U.S. Department of Energy Web Interface to Engine Config Tomcat/JSP/Servlets to view and edit

7Managed by UT-Battelle for the U.S. Department of Energy Web Config View: Channel Stats

8Managed by UT-Battelle for the U.S. Department of Energy Web Config View: Sample Stats OK ??

9Managed by UT-Battelle for the U.S. Department of Energy Stats 38 sample engines, channels Host that runs sampling engine: – CPU load 45%, zero disk I/O wait, very responsive Oracle – Cluster – Sample tables partitioned by day – 8000 samples/sec peak in write tests – Operationally maybe ¼ of that Better configuration would likely have fewer samples/sec

10Managed by UT-Battelle for the U.S. Department of Energy Summary, Status At SNS, BEAUtY replaced Channel Archiver – Parallel operation for ~2 month – Turned old sample engine off this month About a year of testing, many Oracle setup issues – Oracle cluster setup – Updated partitioning Next – Data reduction: Replace Oracle partitions of old data with reduced channel/sample count

11Managed by UT-Battelle for the U.S. Department of Energy Stuff

12Managed by UT-Battelle for the U.S. Department of Energy Hurdles Months: Get new Oracle server configured – Interface cards for storage array, fiber switches – Firewall holes for office access, backup, admin Changes in 10g – SELECT MIN(stamp), MAX(stamp) -> NULL, NULL – No “range” partitioning on Index-Organized-Tables Configuration issues – ORA-01654: unable to extend … – ORA-00257: archiver error What used to be impossible is now “easy”, but still expensive – DELETE FROM SAMPLES WHERE …

13Managed by UT-Battelle for the U.S. Department of Energy Configure Sample Engines Configuration is in RDB – Directly use SQL – EngineConfigImport for legacy config files – View/Edit via web Hierarchical (as before) – Sampling engine (name, where to run, …) Groups – Channels No more duplicate channels!

14Managed by UT-Battelle for the U.S. Department of Energy Other Ideas Jlab’s MyA – Operational, but Wrapper code around MySQL to create ‘cluster’ Handles less meta info (units, limits, …) Viewer is one-of C++/TCL/Tk Gabriele Carcassi mentioned RDDTool – Toolset for logging data with data-aging – Command-line RPN tools, web viewer – May be faster than general-purpose RDB but lacks advantage of gpp RDB