Archive Systems What you always wanted to know but were afraid to ask: What’s available? Who’s doing what? PAL EPICS Meeting Oct. 2012

1. Performance? How many samples/second write & read? 2

2. How to get at the data? CSS archive plugin? 3 Easy web reports?

3. All the Rest? How to configure? How to – start/stop sample engines? – Monitor if they’re running? connected to all channels? write without errors? Impact of disk failure, engine crash? Limits when data grows? How to spread data across disks, servers? – How to delete certain channels or time ranges? – Data backup? Info: – Which channel has the most samples in last 24 hours? – Which subsystem generates the most samples? 4

System Language, Platform: Java, C++, ?? Data Store Advantage Great Stuff -Disadvantage -Problem User, user, user 6

Channel Archiver C++ Custom Data/Index Files Fast: Write 60k, 70k samples/sec -Need library to read data -Trouble when engine crashed -File size limits -Maintenance nightmare SNS until 2009, KEK, J-PARC, SLAC, KSTAR,.. 7

Channel Archiver, 64 bit.. As before, but Extended file size limit -Even Bigger Trouble when engine crashed? -Even Bigger Data maintenance nightmare? Diamond 8

RDB-Based Channel Archiver (BEAUTY) Java (Eclipse, CSS) RDB: MySQL, PostgreSQL, Oracle, … Any data maintenance possible No size limits; partitions, cluster, … Any type of reporting on data Anybody can reach data from any programming language Engines don’t crash, and if they do: Just no new samples for that engine -Slow: 8000 samples/sec write -~15k for special MySQL setup SNS since 2009, KEK, ITER? 9

DESY MySQL Archiver Java (started with BEAUTY code) RDB: MySQL with special setup – Writing original sample plus a few averages Faster writing because of some MySQL optimization Faster retrieval of long time ranges Most of the usual RDB benefits -Only MySQL DESY? 10

Fermilab (not EPICS) Java MySQL ‘BLOBs’ used as ring buffer Faster Automatically replace oldest data, not running out of space -Need library to write and read, encode and decode content of BLOBs -None of the usual RDB advantages Fermilab 11

MyA (Jlab) C++? Homebrew MySQL ‘cluster’ – Several MySQL Servers. One table per channel. Faster Most of the usual RDB benefits -Need library to read from correct MySQL server and table JLab 12

Archive Appliance Java Google Protocol Buffers Faster “Appliance” Idea Submit channel name. Rest Auto-configured. -Need library to access data? SLAC, BNL, IHEP, PSI, ITER? 13

CERN Language, Platform: Java, C++, ?? Oracle Fast because they tuned it well -not EPICS CERN 14

SDDS C SDDS Files Integrate4d w/ SDDS tools -Needs library to access -Data management APS, … 15

Fast Acqu. Archive C Raw disk Fast, for specific purpose -Need lib. -Data management Diamond 16

SLC History Buffers Fortran Binary files Worked for decades -Slow SLAC 17