NLC - The Next Linear Collider Project Lee Ann Yasukawa 05/25/99 NLC Archiving Requirements (Preliminary)

Lee AnnYasukawa 05/25/99 2 NLC - The Next Linear Collider Project Why does the NLC need a new Archiver? Since the NLC will be such a complex machine, there needs to be greater flexibility in the data archived than is currently available in order to troubleshoot problems The archived data for some devices need to be on the scale of milliseconds for a short period of time (seconds) while data for other devices may be on the scale of minutes over a longer period of time (10’s of minutes to hours) Data viewing from many different control system applications should have a consistent interface: such as archived data, correlation plots data, buffered data acquisitions, configuration files, etc.

Lee AnnYasukawa 05/25/99 3 NLC - The Next Linear Collider Project Why does the NLC need a new archiver (cont)? Collaborators will be internationally based so access to data must be accessible from around the world The amount of data is huge (not including fast-timed based archives): 600,000 pvs x 80 bytes/pv x 20 sets of data/hour ~1.0 Gbyte/hour x 24 hours Gbytes/day

Lee AnnYasukawa 05/25/99 4 NLC - The Next Linear Collider Project Requester and Archiver Requester Interface between user and archiver Supports time-based entries, monitor- based entries, triggered entries, time- based + monitor-based entries and fast- timed entries (ring buffers) Validates requests Bookkeeper of archived channels/rate/requesters Limits requests when necessary Generates diagnostic reports Archiver Retrieves pv data at specified rates or monitor intervals and stores the data in files quickly and efficiently Limited ring buffer capability Ability to create “other” data files for configuration data, correlated data, etc. Must be able to handle up to ~600,000 pvs Much of the data will come in bursts instead of regular intervals

Lee AnnYasukawa 05/25/99 5 NLC - The Next Linear Collider Project Data Converter and Data Retriever Data Converter Runs periodically to convert data from fast file save format to fast file access format Not part of the archiver Ages and compresses data when necessary Calculates statistics of compressed data Keeps master index of data locations Data Retriever Accepts requests to retrieve data Validates requests Utilizes master index to locate data Can process data in fast file save format and also fast file access format Can retrieve data from archiver, correlation plots, buffered data acquisitions and configuration files Sends data back to requester

Lee AnnYasukawa 05/25/99 6 NLC - The Next Linear Collider Project Browser User interface for data retrieval Web based component Capability to display graphs and histograms Capability to perform simple calculations (mean, std dev, etc.) and simple fits Ability to zoom, pan and lasso graphics Ability to export data to various output formats (MATLAB, spreadsheets, etc) Ability to display time-based data, correlated data, correlated history data, configuration data, etc

Lee AnnYasukawa 05/25/99 7 NLC - The Next Linear Collider Project Requester/Archiver Questions Requester Questions: Can anyone make archiver requests? Should there be special privileges for different types of requests (based on rate? Subsystem?) Do people based internationally need this capability? Should it be web-based? Archiver Questions? How closely sync’d in time do the IOCs need to be? What are the requirements for fast-time based entries (ring buffers)? How many data points should be available? What is the maximum frequency which needs to be sampled (based on hardware requirements)?

Lee AnnYasukawa 05/25/99 8 NLC - The Next Linear Collider Project Data Converter/Data Retriever Questions Data Converter Questions: How is data aged? Should statistics be kept on all aged data? If so, what type? (May depend on hardware requirements). Do users have input as to how much data is kept? How long should data be kept available online? How often should data be converted? Once daily? More often? Performance requirements? Data Retriever Questions: What are the performance requirements? How much data, how fast so that it is acceptable to users?

Lee AnnYasukawa 05/25/99 9 NLC - The Next Linear Collider Project Browser Questions Does the browser need security imposed or can anyone request data? What are the display performance requirements? How many pvs? How many graphs? Timing? How sophisticated should the calculation capability be? Can the browser do simple calculations and simple fits and rely on exporting data for sophisticated calculations and fits? Or, do we want the browser to encompass everything the user wants to do with the data? Should MATLAB (or some other calculation engine) and some spreadsheet program be integrated into the browser?

Lee AnnYasukawa 05/25/99 10 NLC - The Next Linear Collider Project Technical Challenges Optimizing algorithms which stores data and accesses data fast and efficiently Finding storage mediums to handle the volume of data which will be archived and that is capable of meeting performance requirements Creating a robust system with built-in redundancies which can handle failovers seamlessly Designing a system which meets requirements now yet is flexible since the requirements will inevitably change over time