1. Staging Archived Data at DESY Albert.Kagarmanov@desy.de December 2008

2. Channel Archiver Currently we permanently archive ~ 4000 PVs We are using ChannelArchiver for initial archive They devided to 10 sub-archive correspondent to our main facilities – MKS, MKK, CTA, CMTB, etc ChannelArchiver working in our SUN-FIRE Solaris FileServer ChannelArchiver produced 2 Gb raw binary data pro week

3. Archive Daemon web interface

4. Channel Archive Limitations “Archive is perfect but how to extract data from it?” raw-data saved in model “one directory/day/sub-archive” –i.e. /CAR/mkk/2008/11_24 Channel Archive working fast and relaible for store data, but extracting from binary file is not so fast. Channel Archive produced one unique master master_index file for real-time indexing data using balanced R-Tree theory Really slow if master_index keeps all tracks for more than 3 month history

5. Our Long term Archiving Schema Data from ‘now‘ till 2 month before is collected under normal ChannelArchiving Schema BinaryData 2 month old till 12 month old is moved under another directory CAR_Save In parallel, data which 1 month old convert to SDDS- format. –SDDS- well-known „Self Describing Data Set“-format is APS- Argone special format for physical data store and handeling : www.aps.anl.gov/asd/oag/software.shtml SDDS-archive is our “final” archive format. All data from 1996-2008 permanently avaiable under this format

6. ChannelArchiver2SDDS converting details: 1.This is periodical (1 time pro week) script 2.Pipe of 2 tasks: Standart ArchiveExport command which provide ASCII output from ChannelArchiver ASCII2SDDS DESY C++ program which from ascii input create SDDS file 3.Source for ASCII2SDDS : -www- mks2.desy.de/content/e4/e40/e41/e12077/e12085/e21187 4.Few years ago we also reduce old data using special DESY developed algorithms, now we’re not use it since we’re don’t problem with disk space: Source for DataReduction avaiable under www-mks2.desy.de/content/e4/e40/e41/e12077/e12085/e21382

7. ChannelArchiver2SDDS converting details: IOC Now Disk current Archive Disk longTerm Archive CA mv Disk ASCII2SDDS SDDS Disk Reduction/ZIP compress

8. SDDS-format and AAPI client/server SDDS-data store in model –I file pro PV pro month: i.e. /data/sdds/2008/08/wienerVME:memUsed_ai.gzip SDDS-lib support auto-unzip (ZIP-ratio ~ 50%) Extracting SDDS-data tool is AAPI-server AAPI-server is C/C++ TCP/IP daemon program which waits request for archive-data from AAPI-clients AAPI-server can handle: –ChannelArchiver Binary Data (last few months) –SDDS-data (1-12 year old archive data which converted to SDDS-format) –DESY-specific TINE and DOOCS archive data (another DESY Control Systems Main AAPI-clients is CSS-DataBrowser and old StripTool

9. SDDS-format and AAPI client/server Source for AAPI-server available under www-mks2.desy.de/content/e4/e40/e41/e12077/e12082 Curent distribution is AAPI-2.7.tar.gz Java-client lib can be found in CSS CVS distribution

10. Gap between live and ChannelArchiver data ChannelArchiver collect data in memory and flushed data to a disk periodicaly (DESY-default=5 min) => Initial ChannelAtchiver browsers have small gap (~5 min) between live- and archive-data.

11. Small gap in data How to avoid this gap: Our idea is to store last ~200 values in IOC, in some sort of waveform record ( IOC-cache) -- archiveRecord.

12. IOC-archive cache -- ArchiveRecord Here, at DESY archiveRecord was developed. For each normal PV we have PV_h-’waveform- like’ record where we store last N (default=200) PV value,status and timestamp from normal PV. CSS-dataBrowser and old StripTool can handle archiveRecord value Gzip distribution is http://www-kryo.desy.de/ documents/EPICS/DESY/General/Archiver/DesyStripTool/archRec.tar.gz

13. IOC-archive cache -- ArchiveRecord

14. CSS Databrowser Most robust EPICS archive data browser Support different archive data extension points – independent archive data types: –ChannelArchiver raw binary data –AAPI protocol (ChannelArchiver + SDDS) – archiveRecord data –Direct DESY TINE archive data –…

15. CSS Databrowser xnds://www-kryo.desy.de/applets/LACA/cgi/ArchiveDataServer.cgi - is standard ChannelArchiver ArchiveDataType aapi://epicsh.desy.de:4053 - is DESY AAPI-server ArchiveDataType archiveRecord:// - is DESY AAPI-server ArchiveDataType …

16. CSS Databrowser Preference Page of DataType extension point configuration The end