1. Gunther Haller haller@slac.stanford.edu LUSI DOE Review July 23, 2007 Controls (WBS 1.6) 1 Control and Data Systems (WBS 1.6) Gunther Haller Scope Control and Data Systems Architecture Control System Data System Team/Organization WBS Cost and Schedule Summary Scope Control and Data Systems Architecture Control System Data System Team/Organization WBS Cost and Schedule Summary

2. Gunther Haller haller@slac.stanford.edu LUSI DOE Review July 23, 2007 Controls (WBS 1.6) 2 Scope – WBS 1.6 Control & Data Systems NEH Hutch 3: X-ray Pump-probe Instrument (WBS 1.2) FEH: Hutch 1: X-Ray Photon Correlation Spectroscopy Instrument (WBS 1.4) FEH Hutch 2: Coherent X-Ray Imaging Instrument (WBS 1.3) X-Ray Beam Direction Near Experimental Hall (NEH) Far Experimental Hall (FEH) Tunnel Separate WBS 1.6 to combine all LUSI control & data needs due to commonality in requirements, design, implementation, and integration

3. Gunther Haller haller@slac.stanford.edu LUSI DOE Review July 23, 2007 Controls (WBS 1.6) 3 Scope LUSI Scope is Control and Data Systems for XPP instrument CXI instrument XCS instrument Common Controls and Data Systems for LCLS and LUSI G. Haller is manager for LUSI as well as for LCLS Benefit from common hardware and software solution LUSI Scope is Control and Data Systems for XPP instrument CXI instrument XCS instrument Common Controls and Data Systems for LCLS and LUSI G. Haller is manager for LUSI as well as for LCLS Benefit from common hardware and software solution

4. Gunther Haller haller@slac.stanford.edu LUSI DOE Review July 23, 2007 Controls (WBS 1.6) 4 System Block-Diagram LUSI Control & Data Systems Control Subsystem for Operation & Controls Data Subsystem for Acquisition & Management, Online Processing LCLS Control System SLAC Sci. Computing & Computing Srvs. Data Farm (PB Tape Drive) Computer Cluster (2000 processor node) Data Archiving/Retrieval Offline Analysis/Rendering Timing & Triggering High peak rate/ large volume Pulse-by-pulse info exchange

5. Gunther Haller haller@slac.stanford.edu LUSI DOE Review July 23, 2007 Controls (WBS 1.6) 5 Control & Data Systems Intensity Monitor Stages, Motors Software EPICS Firmware programming Hardware Cabling, racks, trays Controllers for Pumps Gauges, sensors Motors Signal processing Electrical Image Data acquisition and storage Workstations Data servers, storage arrays Network hardware Software EPICS Firmware programming Hardware Cabling, racks, trays Controllers for Pumps Gauges, sensors Motors Signal processing Electrical Image Data acquisition and storage Workstations Data servers, storage arrays Network hardware 2-D Detector EndstationControls & Data Systems Pumps Controllers Computers Local Storage

6. Gunther Haller haller@slac.stanford.edu LUSI DOE Review July 23, 2007 Controls (WBS 1.6) 6 Control Sub-System VME-based IO Controllers VME PPC CPU’s Network Diffractometer Spectrometer Pump/GaugeMotors Sensors 1-Gb/s Network (LUSI Subnet) Remote Access Configuration Database Remote Access Server Application Framework Display Manager Device Program Layer App GUIEng GUI Based on EPICS, LCLS design Terminal Servers

7. Gunther Haller haller@slac.stanford.edu LUSI DOE Review July 23, 2007 Controls (WBS 1.6) 7 Main Controls Devices Valves Ion pumps Turbo pumps Vacuum gauges Motors Encoders Limit switches Pneumatic actuators RF phase shifters Spectrum analyzers Cameras Waveform sampling devices Pulse-height sampling devices Power supplies Laser diagnostics devices Temperature control

8. Gunther Haller haller@slac.stanford.edu LUSI DOE Review July 23, 2007 Controls (WBS 1.6) 8 Data Acquisition and Processing Acquire 120-Hz Images Without beam (dedicated calibration runs or in-between ~8-ms spaced beams) Images to be used to calculate calibration constants With beam Images which need to be corrected Event-building with 120-Hz accelerator timing & beam-quality data information Filtering of images (e.g. vetoing) Realtime pixel correction using calibration constants Processing of corrected images Example: Coherent imaging of single molecules Combine 10 5 to 10 7 images into 3-D data-set Learn/pattern recognition/classify/sort images, e.g. 1,000 bins Average bins Alignment Reconstruction ~ 5-Hz real-time display of images

9. Gunther Haller haller@slac.stanford.edu LUSI DOE Review July 23, 2007 Controls (WBS 1.6) 9 Data Management and Scientific Computing Data Management Make LUSI data accessible for science Appropriate format(s) Appropriate storage/retrieval hardware and software Scientific Computing for LUSI Science Opportunities and needs being evaluated Very dependent on the detailed nature of the science Unprecedented size (for photon science) of data sets to be analyzed Unprecedented computational needs (for photon science) Comparable in scale to a major high-energy physics experiment Greater need for flexibility than for a major high-energy physics experiment

10. Gunther Haller haller@slac.stanford.edu LUSI DOE Review July 23, 2007 Controls (WBS 1.6) 10 Data Sub-System Data Rate/Volume of CXI Experiment (comparable to other LUSI experiments) LCLS Pulse Rep Rate (Hz)120 Detector Size (Megapixel)1.2 Intensity Depth (bit)14 Success Rate (%)30% Ave. Data Rate (Gigabit/s)0.6 Peak Data Rate (Gigabit/s)1.9 Daily Duty Cycle (%)50% Accu. for 1 station (TB/day)3.1 Challenge is to perform data-correction and image processing while keeping up with continuous incoming data-stream Tradeoff between tasks implemented online versus offline Important to produce science output without piling up more and more raw images Difference to conventional experiments: High peak rate & large volume comparable to high-energy physics experiments such as BaBar @ SLAC

11. Gunther Haller haller@slac.stanford.edu LUSI DOE Review July 23, 2007 Controls (WBS 1.6) 11 Data Acquisition/Mgmt Architecture Detector Control Node Quick View Rendering Node Disk Arrays/ Controller Tape Drives/ Robots Volume Rendering Node Volume Rendering Cluster ADC FPGA Online Data Server SCCS LUSI 4 x 2.5 Gbit/s fiber Offline Data Servers 2D Detector SLAC LCLS DAQ Box 10–G Ethernet Accelerator 120-Hz Data Exchange & Timing Interface Data system must be able to handle peak rate as well as sustain 120-Hz throughput (corrections, filtering) SLAC LCLS DAQ Box being developed by Particle Physics and Astrophysics Electronics Group (G. Haller) Highly scalable Very high CPU-memory bandwidth (8 Gbytes/sec) Data system must be able to handle peak rate as well as sustain 120-Hz throughput (corrections, filtering) SLAC LCLS DAQ Box being developed by Particle Physics and Astrophysics Electronics Group (G. Haller) Highly scalable Very high CPU-memory bandwidth (8 Gbytes/sec) Online Processors Detector-Specific Front-End Electronics

12. Gunther Haller haller@slac.stanford.edu LUSI DOE Review July 23, 2007 Controls (WBS 1.6) 12 SLAC LCLS DAQ Box DAQ box contains custom SLAC Cluster-Element (CE) board Virtex-4 FPGA 2 PowerPC processors IP cores 512 Mbyte RLDRAM 8 Gbytes/sec cpu-data memory interface 10-G Ethernet science data interface 1-G Ethernet control interface EVG machine timing fiber interface 120-Hz machine beam-data interface RTEMS operating system EPICS Option: up to 512 Gbyte of FLASH memory Easily scalable If several CE boards are used Packaged in ATCA crate with network card Network card based on 24-port 10-G Ethernet Fulcrum switch ASIC Crate holds up to 14 CE boards Option to have up to 7 Tbyte integrated with high- speed memory access Network card interconnects multiple CE boards Network card interconnects multiple crates CE development board 10-G switch development board

13. Gunther Haller haller@slac.stanford.edu LUSI DOE Review July 23, 2007 Controls (WBS 1.6) 13 1.6 WBS to Level 3 1.6 Control & Data Systems 1.6.1 Phys. Support Engineering Integration 1.6.2 Cabling 1.6.3 Computer HW Computers & Clusters Data Storage 1.6.4 EPICS Prog, 1.6.5 DAQ HW & FW Acquisition HW FPGA Prog. 1.6.6 Timing & Triggering 1.6.7 Laser Ctrl & Safety 1.6.8 Vacuum Ctrl 1.6.9 Optics, Diag., & Sample env. 1.6.10 MPS/PLC

14. Gunther Haller haller@slac.stanford.edu LUSI DOE Review July 23, 2007 Controls (WBS 1.6) 14 Core Control & Data Systems Team 1.6. CAM: G. Haller Main personnel A. Perazzo, Online (PPA) D. Nelson, Control (PPA ) S. Peng, Control Software (Controls Division) R. Rodriquez, Infrastructure (PPA) C. O’Grady, DAQ (PPA) Steffen Luitz, Data Management (SCCS) Richard Mount (Director, Sci. Computing & Computing Services) Niels van Bakel (Detector Physicist, LUSI) 1.6. CAM: G. Haller Main personnel A. Perazzo, Online (PPA) D. Nelson, Control (PPA ) S. Peng, Control Software (Controls Division) R. Rodriquez, Infrastructure (PPA) C. O’Grady, DAQ (PPA) Steffen Luitz, Data Management (SCCS) Richard Mount (Director, Sci. Computing & Computing Services) Niels van Bakel (Detector Physicist, LUSI) CAM: Control Account Manager PPA : Particle Physics and Astro Physics Division CAM: Control Account Manager PPA : Particle Physics and Astro Physics Division

15. Gunther Haller haller@slac.stanford.edu LUSI DOE Review July 23, 2007 Controls (WBS 1.6) 15 Control & Data Systems - Cost Estimate

16. Gunther Haller haller@slac.stanford.edu LUSI DOE Review July 23, 2007 Controls (WBS 1.6) 16 WBS 1.6 – Control & Data Systems Cost estimate at level 3 by fiscal year –

17. Gunther Haller haller@slac.stanford.edu LUSI DOE Review July 23, 2007 Controls (WBS 1.6) 17 Control & Data Systems Schedule in Primavera 3.1

18. Gunther Haller haller@slac.stanford.edu LUSI DOE Review July 23, 2007 Controls (WBS 1.6) 18 Control & Data Systems Milestones CD-1Aug 01, 07 Controls & Data PRD Aug 27, 07 Phase 1 Conceptual Design CompleteSep 27, 07 CD-2aDec 03, 07 Controls & Data Final Design Review Apr 25, 08 CD-3aJul 21, 08 Complete Controls & Data Apr 16, 09 Ready for InstallationMay 18, 09 Phase I Installation CompleteAug 07, 09 CD-4aFeb 08, 10 CD-1Aug 01, 07 Controls & Data PRD Aug 27, 07 Phase 1 Conceptual Design CompleteSep 27, 07 CD-2aDec 03, 07 Controls & Data Final Design Review Apr 25, 08 CD-3aJul 21, 08 Complete Controls & Data Apr 16, 09 Ready for InstallationMay 18, 09 Phase I Installation CompleteAug 07, 09 CD-4aFeb 08, 10

19. Gunther Haller haller@slac.stanford.edu LUSI DOE Review July 23, 2007 Controls (WBS 1.6) 19 Summary Control subsystem is based on EPICS Controllers will be common to LCLS systems Motion, vacuum, cameras, power supplies, etc Peak data rate/volume requirements are comparable to HEP experiments, requiring dedicated data acquisition and management systems Data subsystem concept & architecture is being developed Use standard interface to all detectors Use data systems that have high bandwidth and are scalable Is leveraging significant expertise at SLAC in data acquisition and management Ready to proceed with baseline cost and schedule development Control subsystem is based on EPICS Controllers will be common to LCLS systems Motion, vacuum, cameras, power supplies, etc Peak data rate/volume requirements are comparable to HEP experiments, requiring dedicated data acquisition and management systems Data subsystem concept & architecture is being developed Use standard interface to all detectors Use data systems that have high bandwidth and are scalable Is leveraging significant expertise at SLAC in data acquisition and management Ready to proceed with baseline cost and schedule development