1. 1 BROOKHAVEN SCIENCE ASSOCIATES NSLSII Control System The Control Group – presented by Bob Dalesio EPICS Meeting April 12, 2015

2. 2 BROOKHAVEN SCIENCE ASSOCIATES Outline Project Planning Integration shortcomings (Lessons learned) Setting Standards Control System Studio (CSS) Tools Fast Orbit Feedback Components Development Middle Layer Services for High Level Application (Relational) Database Developments Experiment Control, DAQ, Data Analysis, Data Management. Conclusions

3. 3 BROOKHAVEN SCIENCE ASSOCIATES Project Planning Identify critical requirements Define scope and interfaces Recruit, Recruit, Recruit Project Execution

4. 4 BROOKHAVEN SCIENCE ASSOCIATES Control System High Level Requirements – 1 of 3 2.642 usec ring revolution Injection system runs up to 1 Hz Top off every 1 minute Top off bunch train ~20 psec Top off damping time 10-50 msecs (inform beam lines) 80% Fill – 1056 of 1320 buckets filled different fill patterns high charge single bunch at mid-gap Orbit stability to.3 microns at injection 5 Hz updates to operators of up to 1000 chosen parameters Archive up to 20,000 parameters at a rate of 1 Hz continually Must scale to support 150,000 physical I/O connections and 400,000 computed variables 99.7% availability during operation

5. 5 BROOKHAVEN SCIENCE ASSOCIATES Control System Requirements – 2 of 3 80 psecs pulse to pulse timing jitter. During top off, some beam lines will need 1.1 - 1.8 psecs of timing jitter Timing resolution 2 nsec for bunch to bunch timing 200 psec for single bunch injection Provide electron detector as event for beam line 10 kHz power supply read backs triggered from timing sys All data available to system with revolution identifier for turn by turn data correlation. 20 msec equipment protection mitigation Sub 1 msec at other labs when undulator gap is closed – it’s the xrays Also – slow detection/shut down on RTDs on beam pipe for beam studies when beam emittance is wide Plc provides voltage to comparator to bpm signal – forward/reflected to give current – with fiber optic transmission of this signal to the RF – one from each cabinet. 10’s of micro seconds 200 turns to loose energy after RF is off Photon shutter – can take full energy? Transient Recording Take coherent turn by turn orbit data for up to 800 channels for 1024 turns (each turn is 2.36 usec) Latch the last 10 seconds of data from all parameters in the storage ring Beam line needs 1 msec archiving over 1 minute for temperatures and positions Provide data for all control aspects (no hidden parameters) Conventional facilities Tunnel and experimental floor Vibration < 25nm from 4-50Hz Thermal stability of storage ring tunnel enc +-.1 dgc Experimental floor +-.5 dgc

6. 6 BROOKHAVEN SCIENCE ASSOCIATES Control System Requirements – 3 of 3 5 KHz RF Feedback on beam phase 10 kHz orbit feedback, (100 usec loop time) 360 BPMs (12 per cell) 90 Corrector PS (3 per cell) Slow orbit correction 180 Slow PS (6 per cell) synchronized with fast orbit feedback 1 Hz model based control 10 ’ s of Hz Data Collection for RF loop correction. 200 MeV to 3 GevV in 400 msec ramp in booster for gap changes Transverse feed back Bunch to Bunch 1 H and 1V stripline RF cavity control BPM Requirements Turn by turn on demand Fast 10 kHz output Slow 10 Hz data ADC Raw data Requirements at an accelerator evolve!! Standard engineering flow is iterative – Requirements-specification-design-implement-test-integrate, so flexible designs and attention are required.

7. 7 BROOKHAVEN SCIENCE ASSOCIATES EVGEVG EPICS Time – events / data / triggers CPUCPU RF Osc 500 MHz MHz w/ 60Hz Fid Timing Channel Access Protocol Timing Digitizer EVREVR CPUCPU RF Equipment PLC LLRF RF EVREVR TCMTCM CPUCPU TCMTCM TCMTCM TCMTCM TCMTCM Non-BPM Diagnostics EVREVR CPUCPU Libera Modules Diagnostics Machine Protection Bus Ethernet – Instrumentation Bus Control System Scope and Interfaces (1/3)

8. 8 BROOKHAVEN SCIENCE ASSOCIATES EVREVR Timing – events / data / triggers CPUCPU GPIBGPIB PLCPLC BIBI BOBO AIAI AOAO AIAI GP307 IG HP937 CCG Power Supplies Valve Controllers CPUCPU Power Supply Control Ethernet - Channel Access Protocol Power Supply Controllers Ethernet – Instrumentation Bus Vacuum Machine Protection Bus EVREVR CPUCPU PPS I/O GPIBGPIB PLCPLC BIBI BOBO AIAI AOAO AIAI PSIPSI Temperature & High Prec. DMM CPUCPU Facility I/O Facility Personnel Safety Sys. Control System Scope and Interfaces (2/3)

9. 9 BROOKHAVEN SCIENCE ASSOCIATES Timing I/OI/O EVREVR CPUCPU Beam-line Control Timing I/OI/O EVREVR CPUCPU Undulator Control Timing I/OI/O EVREVR CPUCPU Booster Ring Control Timing I/OI/O EVREVR CPUCPU Injection Linac Control Help write specifications, monitor progress, Particpate in integration testing, take over maintenance Design, Prototype solutions, Produce beamline controls Design, Prototype solutions, Produce undulator control Control System Scope and Interfaces (3/3)

10. 10 BROOKHAVEN SCIENCE ASSOCIATES Drivers EPICS Device Support I/O EPICS Database Configuration Tools Client Tools Engineering Screens Engineering Applications XAL/RDB Tools Physics Applications Populate RDB Device test / subsystem integration Includes: LINAC, Booster, and Storage Ring, Facility, Undulator, Beamline Commissioning Support Develop Control Algorithms Control System Software Scope

11. 11 BROOKHAVEN SCIENCE ASSOCIATES Project Execution (1/2) Budget: 20.59M Controls of 462M: 349M accelerator 113M beam lines One PCR added approximately 2M for network and computer infrastructure Actuals: 21.36M -770K (CPI =.96) Project missed early completion date by several months. Staffing: Now 40 - 30 New Hires into Light Source Division, 10 transferred from LS1. Recruiting and building a team of experts is a major job. Engineers were given responsibility and authority over complete subsystems (BPM, PS, RF, Physics..). They also were given projects that pushed their technical skills. The control engineers are co-located to help them keep up with the work of their control colleagues. One person reported EVMS. EVMS execution has certain knobs for execution: accurate, precise, effective,

12. 12 BROOKHAVEN SCIENCE ASSOCIATES Project Completion (2/2) Scope includes: FPGA code for power supplies (1000), BPMs (300), Fast Orbit Feedback (30), Position Capture / Detector Triggering, Single Channel Correlator, Fast Machine Protection, Timing, and Active Interlock. PLC code for low speed IO: utilities on the accelerator, all PLC IO on beam lines including EPS EPICS Driver for all hardware (most provided by community) EPICS Databases for all subsystem and facility IO CSS Synoptic Displays Middle layer services for physics applications including a model server Network and computer layout Choose EPICS tools where they are adequate New EPICS tools and services: channel finder, save/restore, metaDataStore, Olog, CSS build, extended Boy widgets, unit conversion, areaDetector PVAccess Plugin, No. of EPICS records985K Accelerator + 180K for 6 beam line delivery systems No. of IOCs 163: 45 VME, 7 cPCI, 111 IBM Servers (Instr Bus – Ethernet) No. of 19" racks 485 No. of devices 61 unique device types No. of drivers 26 No. of signals 200K Ines of code for EPICS tools: 1.8M lines of in-house application code 185K

13. 13 BROOKHAVEN SCIENCE ASSOCIATES NSLS II Integration Lessons Learned I (1/4) Linac LINAC Beam Dump Faraday Cup Bending Magnet Second Dump Faraday Cup Radiation Monitor Booster Tunnel Restricted Area

14. 14 BROOKHAVEN SCIENCE ASSOCIATES NSLS II Integration Lessons Learned I (2/4)

15. 15 BROOKHAVEN SCIENCE ASSOCIATES NSLS II Integration Lessons Learned I (3/4)

16. 16 BROOKHAVEN SCIENCE ASSOCIATES NSLS II Integration Lessons Learned I (4/4) The primary issue here is incomplete consideration of machine modes and insufficient training of operators. Good procedures for commissioning avoided any safety issue. Thorough follow up uncovered and resolved this and other shielding issues. Defining the modes of operation with soft limits could have prevented this by limiting power supply range when operating with only one RF system. There was no control design document for PPS from controls as another team was implementing it and controls was integrating it. This could have lead to a deeper questioning of modes of operation. Instruments were integrated, but not tested for operation. Linearization errors resulted in oversaturation of the instrument and no alarm condition flagged in the control room. Archiving all critical parameters prior to commissioning is a critical component in commissioning..

17. 17 BROOKHAVEN SCIENCE ASSOCIATES NSLS II Integration Lessons Learned II (1/4) During low power tests, the Active Interlock System did not respond under testing when the beam was moved. Upon study of the PVPutLog it was discovered that all BPMs were put into test mode. This was done by someone unwittingly pressing an ambiguous button.

18. 18 BROOKHAVEN SCIENCE ASSOCIATES NSLS II Integration Lessons Learned II (2/4) Modifications: Remove the ability to set the BPMs into simulation mode from the screen Change the FPGA code to disable simulation greater than 2 mA would need to transmit to BPMs. This requires: Run a signal from the cell controller nearest the EVG into a signal in the EVG that indicates that we are over 2 mA. This signal is wired from the safety PLC into a cell controller near the ICT. Send the machine mode from the EVG to the EVR in each of the BPMs and use it to disable simulation mode in the FPGA. Modify the AI cell controller to compute BPMs that are not responsive and take appropriate action based on the loss of one or more BPMs. Other parameters were identified that could render compromise the BPM readings by changing the calibration/conversion constants. Further tests were listed as a result of an engineering meeting to discuss potential failures through disconnecting signals and removing power from critical devices.

19. 19 BROOKHAVEN SCIENCE ASSOCIATES NSLS II Integration Lessons Learned II (3/4) 1‐ Remove each fault signal from a cell controller and note the response from the AIS a. Disconnected the 2 mA current signal from the cell controller at cell 3. i. No interlock. b. Removed the DCCT heartbeat signal from the Cell controller at cell 3. i. Indication of heartbeat stops on the control screen. ii. No interlock. c. Disconnected the request to trip signals from the front end EPS system. i. Fault indicated on the CSS screen. ii. Interlock. d. Removed the EPS heartbeat signal from the cell controller at cell 3. i. Indication of heartbeat stops on the control screen. ii. No interlock. e. AIS disengaged: removed the ID gap status input from the cell controller at cell 3. i. AIS engages. f. AIS engaged: removed the BM photon shutter input. i. Interlock. g. AIS engaged: removed the ID photon shutter cable. i. No interlock. 2‐ DCCT fault condition test. a. Powered off the DCCT from the control room this caused an audible alarm in the control room as expected. b. Crashed the DCCT IOC and the result is the same audible alarm in the control room as expected. 3‐ Remove timing at cell controller (additional testing added). a. Removed timing fiber from the cell controller at cell 3. i. BPM data stopped updating, ii. No interlock. iii. Restored the timing cable and the system came back on line as expected. iv. Enabled AI and repeated the test, this resulted in an RF trip. b. Removed timing fiber from ID BPM. i. No interlock. c. Removed timing fiber from RF BPM. i. No interlock. d. Removed RF rev tick from ID BPM and AI tripped the RF. 4‐ Remove the SDI cables from the BPM. a. Interlock. b. Repeated the test no interlock. 5‐ Power off any cell controller. Powered off the cell controller at cell 3. a. No interlock. 6‐ Power off ID BPM. a. Interlock. 7‐ Remove the timing fiber from cell 31. a. Interlock after delay. b. Repeated the test same result. 8‐ Remove timing fiber from the EVG. a. Interlock. b. Repeated the test interlock. c. System recovered smoothly after reconnect and beam was quickly restor

20. 20 BROOKHAVEN SCIENCE ASSOCIATES Fault discussion: task lead: “We had these issues as no one was in charge”, response: “You were”. Be clear about subsystem/task ownership across organizational lines. Good commissioning plans turn up surprises and avoid disaster. Carefully develop operations screens and hide engineering screens. Control system engineers should spend 10% of their time understanding what must be done, and 90% of their time considering what is missing and how it will fail. TEST ALL FAILURE MODES!! NSLS II Integration Lessons Learned II (4/4)

21. 21 BROOKHAVEN SCIENCE ASSOCIATES Standards Save Time (1 of 2) All Hardware Integration is achieved through EPICS Hardware standards are followed wherever possible through NSLS II https://wiki.bnl.gov/nsls2controls/index.php/Beamline_Controls_Equip ment#Standards Control System Studio is used for operator tools Archive Appliance or Channel Archiver to archive time series data BEAST – alarm server Channel Finder Service – Provides properties/tag searches of the Process Variables MASAR – save set manager and user interface Olog – Electronic log book MetaDataStore – store sweep information to search data sets

22. 22 BROOKHAVEN SCIENCE ASSOCIATES Standards Create Clarity (2 of 2) Naming convention https://ps.bnl.gov/acc/controls/Shared%20Documents/NamingConvention/LT-ENG-RSI-STD-002_Rev4.docx https://ps.bnl.gov/acc/controls/Shared%20Documents/NamingConvention/LT-PRJ-STD-CO-NAM-001.xlsx https://ps.bnl.gov/acc/controls/Shared%20Documents/NamingConvention/LT-ENG-RSI-STD-002_Rev4.docx https://ps.bnl.gov/acc/controls/Shared%20Documents/NamingConvention/LT-PRJ-STD-CO-NAM-001.xlsx Coordinate system https://ps.bnl.gov/phot/ProjectBeamlines/Controls/References%20and%20Standards/Coordinate%20System/ LT-C-XFD-STD-BL-COORD-001_Ver3.docx https://ps.bnl.gov/phot/ProjectBeamlines/Controls/References%20and%20Standards/Coordinate%20System/ LT-C-XFD-STD-BL-COORD-001_Ver3.docx Interfacing standards https://ps.bnl.gov/phot/ProjectBeamlines/Controls/References%20and%20Standards/Wiring%20Standards/LT -C-XFD-SPC-CO-IIS-001_Ver5.docx https://ps.bnl.gov/phot/ProjectBeamlines/Controls/References%20and%20Standards/Wiring%20Standards/LT -C-XFD-SPC-CO-IIS-001_Ver5.docx Controls RSI https://ps.bnl.gov/phot/DocumentReferenceLibrary/RSIDocuments/1.04.02%20%20Standard%20Local%20C ontrols%20and%20Data%20Acquisition%20Systems/Stnd%20Local%20Controls%20and%20DAQ%20RSI% 20Development/LT-C-XFD-RSI-CO-001_Ver3.docx https://ps.bnl.gov/phot/DocumentReferenceLibrary/RSIDocuments/1.04.02%20%20Standard%20Local%20C ontrols%20and%20Data%20Acquisition%20Systems/Stnd%20Local%20Controls%20and%20DAQ%20RSI% 20Development/LT-C-XFD-RSI-CO-001_Ver3.docx EPS standards (draft) https://ps.bnl.gov/acc/controls/Shared%20Documents/BeamlineControls/Industrial%20Controls/LT-C-XFD- STD-BL-EPS-001_Ver1_draft.docx https://ps.bnl.gov/acc/controls/Shared%20Documents/BeamlineControls/Industrial%20Controls/LT-C-XFD- STD-BL-EPS-001_Ver1_draft.docx PMAC programming standards (based on Diamond standards): https://ps.bnl.gov/acc/controls/Shared%20Documents/MotionSystems/Standards/LT-R-XFD-STD-BL-MC- 002_RevA.docx https://ps.bnl.gov/acc/controls/Shared%20Documents/MotionSystems/Standards/LT-R-XFD-STD-BL-MC- 002_RevA.docx

23. 23 BROOKHAVEN SCIENCE ASSOCIATES Control System Studio Adopted CSS for our user interface after evaluation Developed ChannelFinderService to return process variables by function, location, device, etc… Developed a channel access package to provide all client functions and time synchronous vectors from an arrays of process variables (PVManager) Developed multichannel visualization tools Developed an Integrated Log Book Developed a multi-controller application (ChannelOrchestrator) Developed a save/restore application (MASAR) Developed standard extension points in CSS for services such as log book and directory service. Developed standard extension points in CSS for plots. This first deployment of CSStudio for a facility is exposing many new requirements and some shortcomings that need to be handled.

24. 24 BROOKHAVEN SCIENCE ASSOCIATES Creating a log entry from the alarm tree Create log entry command available is most applications. Control System Studio

25. 25 BROOKHAVEN SCIENCE ASSOCIATES Pre initialized log entries The applications define how to effectively capture the state e.g. when creating a log entry from the alarm viewer the alarming PV, the alarm state the time are automatically added to the log entry. Each application provides the most useful way to capture its information into a log entry. Control System Studio

26. 26 BROOKHAVEN SCIENCE ASSOCIATES Log viewer Once the entry is made users can later search the log book for a particular alarms e.g. search for : alarms *ACMI* Control System Studio

27. 27 BROOKHAVEN SCIENCE ASSOCIATES PV aware entry If log entries contain pv names defined with a specified format, cs-studio is able to extract that pv information from the log entry. CS-Studio will add the “Process Variable” sub menu to the context menu when it can extract a PV. The default syntax is PV: /n, however each site can define their own regular expression for recognizing pv names. Control System Studio

28. 28 BROOKHAVEN SCIENCE ASSOCIATES Open Databrowser to appropriate time The extracted PV can be piped to any other cs-studio applications. In this case we opened the alarming pv in databrowser, with the time axis automatically set to +/- 30 mins from the time the log entry was made. Control System Studio Good tools improve efficiency.

29. 29 BROOKHAVEN SCIENCE ASSOCIATES Fast orbit feedback system in one cell Orbit feedback system architecture

30. 30 BROOKHAVEN SCIENCE ASSOCIATES Orbit feedback system is in commissioning Cell 1 Cell 2 Cell 3 Cell 16 Cell 30 Cell 29 Storage Ring Cell 15 Cell 17 SDI link PI control demonstrated suppressing noise from DC to 200 Hz

31. 31 BROOKHAVEN SCIENCE ASSOCIATES Fast Orbit Feedback – Time Budget 17.7 µs for BNL BPM Electronics delay to pipeline 3.7 µs for analog value averaged from raw data 14 µs estimated for BNL BPM Electronics (FYI 57 µs measured on Libera BPM Electronics with minimum filter) 7 µs measured 300 BPM data distribution by Redundant Data Link 14 µs measured when one of the redundant links not operational 4 µs measured for FOFB calculation with separate mode compensation 5 µs measured for set power supply by PS Shared Memory link 8 KHz measured for fast corrector PS bandwidth w 10 deg phase shift. 1 KHz measured for corrector magnet/chamber bandwidth. Measured on with Inconel beam pipe. This value is at 10 degree phase shift. New magnet/chamber set up will be tested when they are available. Fast machine protection data is provided to a dedicated cell controller that uses ancillary IOC and BPM data to determine mitigation at the RF.

32. 32 BROOKHAVEN SCIENCE ASSOCIATES Production DFE and BPM AFE BPM: 117 MHz Raw Data BPM: 2.36 usec TBT Data BPM: Xmit 10 kHz fast acquisition data for FOFB Cell Controller: 10 kHz fast orbit Cell controller: 10 Hz slow orbit

33. 33 BROOKHAVEN SCIENCE ASSOCIATES Active Interlock / Cell Controller Virtex-6 FPGA Embedded MicroBlaze soft core processor running TCP/IP lwIP stack in conjunction with EMAC Ethernet core 2 Gbyte DDR3 SO-DIMM 1 Gbps Ethernet  Hardware TEMAC  Memory throughput 6GBytes/sec (6) 6.5 Gbps SFP modules 1Gbit FLASH memory 4 Lemo differential output Embedded Event Receiver  MRF’s EVR functions embeded to FPGA  Very flexible and precise timestamp and time synchronization  16 Digital Inputs  12 Digital Outputs

34. 34 BROOKHAVEN SCIENCE ASSOCIATES Beamline IO Position capture and trigger board

35. 35 BROOKHAVEN SCIENCE ASSOCIATES Correlator Combo The input data to correlator is random digital pulse with ~5ns width. Maximum rate is 100MHz. Correlator Combo, i.e. multi tau correlator plus linear correlator, where multi tau correlator is to get the decay curve of G2 in a large correlation lag range and linear correlator is to capture the high frequency oscillation, deploy one correlator combo on Virtex 6 in Cell controller for prototype first, more on Zynq later. Cell controller Digital Front End (DFE) board where Correlator Combo deployed 5ns digital pulse input fr om ADP detector fr om ADP detector Correlator combo FPGA Ethernet port for G2 curve output RS232 port for debug

36. 36 BROOKHAVEN SCIENCE ASSOCIATES Modular Digital Front End Firmware Priovides: EVR decoder Clock at 117 MHz SDI redundant RT data Ethernet to EPICS 4GB RAM over PLB Filing all paperwork to make it OPEN SOURCE Not license it so we can buy it back w/ markup. Successfully developed a number of instruments Given to LBL where improvements were made for their use. Being evaluated for APS & PLS.

37. 37 BROOKHAVEN SCIENCE ASSOCIATES Middle Layer Services Simplify Applications High level application architecture is client/server based on the EPICS version 4 network protocol, PVAccess One server provides results to all clients Tables are used to serve database results These are being implemented in PVAccess Servers are completed that provide: Channel Finder - List of process variables based on function and attributes MASAR – save set service Model/lattice services Developer document for thin clients development Standard clients in CSS can be used to display HLA data

38. 38 BROOKHAVEN SCIENCE ASSOCIATES V4 HLA Architecture Distributed Front-Ends MMLT Client Middle Layer Servers Physical Device Ethernet PVAccess Model Server PVAccess/CAC Control System Studio PVAccess / CAC Diagnostics CAS PVA Elegant/Tracy PVAccess Channel Finder Svr SQL RDB PVAccess/CAC Multi-Channel Arrays Svr Physical Device Power Supplies CAS PVA Physical Device RF CAS PVA Physical Device Vacuum CAS PVA Physical Device Utilities etc.., CAS PVA PVAccess/CAC Save/Compare Restore Svr SQL Scripting HLA Client PVAccess/CAC RDB PVAccess Lattice Server SQL RDB PVAccess/CAC Serves orbit, magnets, any array of channels LS2 Simulation Diag & PS CAS PVA Save/Retrieve Server RDB SQL Magnet, Resp Mtx, etc..

39. 39 BROOKHAVEN SCIENCE ASSOCIATES Channel Finder Overlays Different Views Control Engineers talk about PV/IOC, physicists talk about element/cell/girder. PVs are associated with properties and tags, which are used to construct a lattice familiar to physicist. “the BPM readings in cell 10 girder 2” is easier than PV name

40. 40 BROOKHAVEN SCIENCE ASSOCIATES HLA uses several thousand PVs. Each have properties like “elementname”, “elementtype”, “cell”, “girder”, “s_position”, … Some PVs are tagged with “ HLA.X ”, “ HLA.EGET ”, “ HLA.EPUT ” which means that PV is a “horizontal reading/setting”, “default reading” and “default setting” The lattice object is a list of elements, each owns several PVs. All of the data is initialized from channel finder service: pv, properties and tags. No need to remember PV when writing a normal script. Channel Finder Makes Scale Managable

41. 41 BROOKHAVEN SCIENCE ASSOCIATES Channel Finder Overlays Different Views

42. 42 BROOKHAVEN SCIENCE ASSOCIATES Dynamically creates hierarchical views based on channel property values Channel Finder w/ CSS For those who cannot remember 1M PV names or do not have the patience to type them into screen configurations.

43. 43 BROOKHAVEN SCIENCE ASSOCIATES ChannelFinder Viewer  CSS  Display  ChannelFinderViewer  Searching Name, Property value, Tags Regular Expressions using “*”, “?” prop=val1, val2 is equivalent to prop=val1 OR prop=val2 prop=val1 tag=myTag is equivalent to prop=val1 AND tag=myTag

44. 44 BROOKHAVEN SCIENCE ASSOCIATES HLA – Thin Clients built on services

45. 45 BROOKHAVEN SCIENCE ASSOCIATES HLA – MASAR Use 45 Events (by 2014-07-14 05:12:08) total taken: 1226, approved: 862

46. 46 BROOKHAVEN SCIENCE ASSOCIATES Middle Layer Services for High Level Apps Middle layer services reduce the time and complexity of high level applications. Developing Application Programming Interfaces (APIs) to support thin applications requires time and collaboration that is not needed in monolithic codes. Developing narrow APIs that can be used by applications and tools, such as CSS, requires enough action to get it done and enough collaboration to get it right. (We had mixed execution here). Most physics applications are now layered: IOC, Middle Layer Services, python and/or CSS

47. 47 BROOKHAVEN SCIENCE ASSOCIATES (Relational) Database Experience Goal: All relational data information integrated in IRMIS IRMIS table evolution and application development conflicted IRMIS table redesign broke all JDBC production code ……………..so domains were separated out except New IRMIS tools were being developed for Component type editor Component editor Component Browser Wiring Editor BUT…… never completed in time to deploy ………… other domains developed into deployed applications

48. 48 BROOKHAVEN SCIENCE ASSOCIATES (Relational) Database Experience Channel Finder application uses MySQL (moving to MongoDB) (middle layer service in development) Lattice development with MySQL (middle layer service deployed) Electronic log uses MySQL (moving to MongoDB) (Python API allows scripts to make log entries) Save Retrieve uses MySQL (middle layer service deployed) Unit conversion uses mySQL for Magnet Measurements (deployed into an IOC application) New metaDataStore uses MongoDB (middle layer service in development)

49. 49 BROOKHAVEN SCIENCE ASSOCIATES Relational Database Experience One large database with all of the facility data was not successful. There are around thirty identified database domains. Collaboration led by MSU with ITER, ESS, NSLS II and others to develop applications We have serious questions to answer on cross domain queries and technology evaluations that are ongoing. Relational databases are not the correct solution for all data storage / retrieval problems. Flexible dictionaries used for searching are not support in RDBs.

50. 50 BROOKHAVEN SCIENCE ASSOCIATES Experiment Cntrl / DAQ / Data Management This was not in the scope of the project. However, we did analyze the problem, and do a survey of what was going on in the community. When we scaled some data collection to the frame size (8 MB) and data sets (1kfps/set) that are expected at NSLS II and ran previous SPEC files we found that the process went from 1 minute to 30 minutes spread evenly over three areas: Time to find the data sets that were needed for the analysis. Movement of the data into memory. Time to do computations Beamlines have approximately 30K signals each In our survey, no Xray facility has demonstrated an architecture that supports the range of requirements posed by an entire facility. Everyone has their favorite analysis codes and we will never have the staff to support this for all facility needs. (Make good interfaces and release the masses) We designed a modular architecture, brought forward middle layer services from the accelerator, and found funding to support the development in whatever areas we could start.

51. 51 BROOKHAVEN SCIENCE ASSOCIATES metadata Store The metadata store contains all configuration and environment parameters that are used to characterize a data set. Searches for specific data sets must be optimized. Experiment Log Book The Experiment Log Book contains information about the data collection It may be used to log steps in the data analysis such as export file formats used. Log entries can be made by manually or programmatically Scientific Data The Scientific Data is the repository for large data sets. These data sets include matrices and image sets with the metadata required to plot or view the data. These are stored by sample and time stamp. Data can be written from experiment control or submitted from analysis. Other data sets can also be sent here such as reference or model data. Machine Data Machine data is all time series data that is taken from the accelerator that is available for use in analysis. Beamline Data Beamline data is all time series data that is taken from the end station, beam line or accelerator that is available for use in analysis. Data Is Stored in Appropriate Repositories

52. 52 BROOKHAVEN SCIENCE ASSOCIATES Experiment Support By Construction End  A MetaDataStore service is being provided to support the tracking of data – Pre Alpha Version that reduced search time from 13 minutes to 40 msec when there were 1M frames with 120 searchable properties each.  A ChannelArchiver is provided to store time series data from the beam lines. – Prototype service to provide network access to archive data.  A Science Data Store is being provided to store large frame rate data sets. – Nothing  Movement of large arrays – Zero copy implementation for EPICS, PVAccess plug-in prototype to send ND arrays over PVAccess.  Experiment Control – Nothing.  Standardization on Python.  New facilities are limited by computing infrastructure and software investments

53. 53 BROOKHAVEN SCIENCE ASSOCIATES Samples are either viewed by their order in the sample changer dewar (DewarView), or by the order the user would like to collect (PriorityView). Priority The color coding for now is cyan=done, green=running, checked means requested to run when the "Collect Queue" button is pressed below the tree. Recently – Sample management/automation

54. 54 BROOKHAVEN SCIENCE ASSOCIATES Exp Control, DAQ, Visualization, Analysis

55. 55 BROOKHAVEN SCIENCE ASSOCIATES Standard DAQ Library – Alpha release

56. 56 BROOKHAVEN SCIENCE ASSOCIATES Data Analysis on Vistrails – Alpha Release

57. 57 BROOKHAVEN SCIENCE ASSOCIATES Exp Control integration with Middle Layer

58. 58 BROOKHAVEN SCIENCE ASSOCIATES Visualization integrated with metaDataStore

59. 59 BROOKHAVEN SCIENCE ASSOCIATES Detector Ethernet CAC Control System Studio with Correlation Plots CAC Process Database. CAS Channel Archiver View PVManager PVAS Channel Finder Server SQL* RDB PVAC PVAS PVAC PVAS Experiment Information. SQL* MetaData Store PVAC PVAS Archive Retrieval XML/RPC Beamline Data N-lanes PVAS Archive Store/Retrieve Science Data XML/RPC PVAS Archive Retrieval XML/RPC Machine Data REST Experiment Information. SQL PASS Science Data Device Support Area Detector Driver Process Database. CAS PVAS Device Support Driver Web Clients HTTP Instrumentation NFS File Formatter CAC PVAC NFS Experimental Data Architecture DataBroker Control & Analysis Clients DataBroker PVAS Experiment Information. MongoDB OLOG

60. 60 BROOKHAVEN SCIENCE ASSOCIATES Conclusions (1/2) NSLS II has leveraged off of years of development from the community : We have invested in several areas of development (3.15 extensions, V4, CSStudio, middle layer services, fast orbit feedback hardware) and most of them greatly improved functionality and productivity. We are starting to deploy these services for data acquisition and have been able to reuse several of the services. Setbacks/oversights were uncovered painlessly with cautious commissioning plans. They were used to productively resolve the issue and uncover any related issues.

61. 61 BROOKHAVEN SCIENCE ASSOCIATES Conclusions (1/2) There are several key areas that need development in the future: PVAccess as the primary protocol for EPICS, CSStudio data layer maturation, and an open-source hardware platform. Scientists are unable to do science yet as serious scope that was required was not in the scope of the project nor in the plan for commissioning and operations. Operations: Feb, through the end of Mar) 842 hrs scheduled, 106.4 hrs down, 1.92 hrs (in 2 faults) charged to controls. (network problem accounted for most)