From the ChannelArchiver to the Best Ever Archive Utility, Yet July 2009

2Managed by UT-Battelle for the U.S. Department of Energy CSS-based OPI Archive Archive Engine config.xml IOC Binary Data Files Data Server “ASCII” Config. Channel Access XML-RPC Channel Archiver History ~2000: Started by Bob Dalesio ~2003: Index Tools, Data Server ~2007: CSS Client

3Managed by UT-Battelle for the U.S. Department of Energy Problems Data file format optimized to write many samples quickly – More then 40000/second –.. but we only used maybe 1000/sec –.. and many ill-configured or duplicate channels Headaches with data maintenance: – Scripts to restart engines, copy data, update indices. – Index time grows with data – Stuck when index files reach 2GB – SNS Users faced with ~80 sub-archives – No clue what needs fixing after network/power problems – No idea who contributes how many samples – No way to remove selected channels or time ranges – Improving on this means implementing an RDB

4Managed by UT-Battelle for the U.S. Department of Energy CSS-based OPI ArchiveEngine config.xml IOC RDB (Oracle/MySQL) Channel Access New Setup Samples Config. EngineConfig -Import Other tools for config & samples

5Managed by UT-Battelle for the U.S. Department of Energy CSS Data Browser Handles Both New URL Just one ‘RDB’ sub-archive Old and new data can be combined in one plot

6Managed by UT-Battelle for the U.S. Department of Energy Web Interface to Engine Config Tomcat/JSP/Servlets to view and edit

7Managed by UT-Battelle for the U.S. Department of Energy Web Config View: Channel Stats

8Managed by UT-Battelle for the U.S. Department of Energy Web Config View: Sample Stats OK ??

9Managed by UT-Battelle for the U.S. Department of Energy Stats 38 sample engines, channels Host that runs sampling engine: – CPU load 45%, zero disk I/O wait, very responsive Oracle – Cluster – Sample tables partitioned by day – 8000 samples/sec peak in write tests – Operationally maybe ¼ of that Better configuration would likely have fewer samples/sec

10Managed by UT-Battelle for the U.S. Department of Energy Summary, Status At SNS, BEAUtY replaced Channel Archiver – Parallel operation for ~2 month – Turned old sample engine off this month About a year of testing, many Oracle setup issues – Oracle cluster setup – Updated partitioning Next – Data reduction: Replace Oracle partitions of old data with reduced channel/sample count

11Managed by UT-Battelle for the U.S. Department of Energy Stuff

12Managed by UT-Battelle for the U.S. Department of Energy Hurdles Months: Get new Oracle server configured – Interface cards for storage array, fiber switches – Firewall holes for office access, backup, admin Changes in 10g – SELECT MIN(stamp), MAX(stamp) -> NULL, NULL – No “range” partitioning on Index-Organized-Tables Configuration issues – ORA-01654: unable to extend … – ORA-00257: archiver error What used to be impossible is now “easy”, but still expensive – DELETE FROM SAMPLES WHERE …

13Managed by UT-Battelle for the U.S. Department of Energy Configure Sample Engines Configuration is in RDB – Directly use SQL – EngineConfigImport for legacy config files – View/Edit via web Hierarchical (as before) – Sampling engine (name, where to run, …) Groups – Channels No more duplicate channels!

14Managed by UT-Battelle for the U.S. Department of Energy Other Ideas Jlab’s MyA – Operational, but Wrapper code around MySQL to create ‘cluster’ Handles less meta info (units, limits, …) Viewer is one-of C++/TCL/Tk Gabriele Carcassi mentioned RDDTool – Toolset for logging data with data-aging – Command-line RPN tools, web viewer – May be faster than general-purpose RDB but lacks advantage of gpp RDB