1. Robert C. Sass Facility Advisory Committee April 16-17 2007 XES Controls/DAQ 1 AMOS/LUSI Controls and Data Acquisition LCLS eXperimental End Stations (XES) Robert C. Sass

2. Facility Advisory Committee April 16-17 2007 XES Controls/DAQ 2 XES Slow Controls No major problems forseen AMOS has ~ 100 movers Ultra high vacuum, power supplies etc. Some requiring special consideration are: Beam inhibit next pulse from the XES via MPS Use chopper to rate limit Synch gas jet & Laser to beam pulse Synch mechanical chopper to the beam

3. Robert C. Sass Facility Advisory Committee April 16-17 2007 XES Controls/DAQ 3 Experimental Data Generation, storage, retrieval and analysis of experimental data is the “product” of the LCLS. LSST, if funded, will produce ~30 TB of data per night. The AMOS experiment may eventually take data @120Hz from: 6 spectrometers @~15 KB. 5 CCDs @1 MB each That’s ~700 MB/second or 2.4 TB/hour or ~58 TB/24 hour running period. Larger CCD densities are already planned. LCLS experiments are relatively short term and so will follow the technology more quickly than LSST. These numbers have nowhere to go but up. With the possible exception of the LHC or until the ILC comes along, LCLS will be the world driver for Data Acquisition, Storage, Retrieval and Analysis.

4. Robert C. Sass Facility Advisory Committee April 16-17 2007 XES Controls/DAQ 4 How Will We Handle This? Build on previous SLAC experience. Initially use commercial digitizer and COTS CPU for AMOS spectrometer data. Only one vendor found that makes an 8GHz digitizer. Use scalable technology developed for LSST for CCD data. Capitalize on years of BaBar experience with hierarchical storage and management of HEP data. Many aspects of traditional HEP computing and data management are applicable to XES/LUSI Large parts of the BaBar data management system have been developed at SLAC Significant expertise for this type of system exists at SLAC. BaBar Stores: ~1 TB/day raw data. >>1 TB/day derived data products. ~1.5 PB total Babar data. Old data purged. Extend/integrate several SLAC-developed and EPICS Java-based technologies for data retrieval and analysis. Java Analysis Studio (JAS) Accelerator Integrated Data Access (AIDA) EPICS Archive Viewer

5. Robert C. Sass Facility Advisory Committee April 16-17 2007 XES Controls/DAQ 5 SCCS XES Hutch Detector Instrument(s) Hutch IOCs & Instruments X-Ray Transport to Diagnostics & Next Hutch Ethernet Fast DAQ Data Detector Control Anywhere LCLS Data Retrieval eXperimental End Station System Overview EPICS Archive MPG/LCLS Event Generator Linac & LCLS Control GUI MCC Main Control Room Linac/LCLS Injector Linac Injector Undulator X-Ray Beam EVR Hardware Triggers XES private subnet Data Acquisition Subsystem(s) XES Controller Experimenter LCLS Analysis / Viewing SLAC WAN Event Receiver Channel Access Gateway Visual Data Monitor 120Hz IOC 120 Hz Archive Engine 120 Hz Beam Line Data 120 Hz Beam Line & Timestamp Data (dedicated Ethernet) XTOD Machine Protection (MPS) EELOG Beam Line Processor Other Databases DAQ Archive Slow Instrument Archive XES Data Cache Fast Instrument Archive Experiment Specific IOCs & Instruments MPS Digitized Data Channel Access DAQ Config SLAC WAN Config/Status

6. Robert C. Sass Facility Advisory Committee April 16-17 2007 XES Controls/DAQ 6 XES Detail – Data Acquisition XES Hutch 6. DAQ Data to SCCS D. ADC Control & Digitized Data B. Distributed EVR Hardware Triggers XES private subnet XES Controller Channel Access Gateway Visual Data Monitor C. Beam Line & Timestamp Data (dedicated Enet) EELOG Beam Line Processor XES Data Cache 4. MPS (Reflective Memory Fiber) E. Detector Control & Digitized Data Non-RTOS SBCRTOS PC Non-RTOSPC EVR VME Beam Line Data PMC 1 Gb E 2 Detector Types Spectrometer CCD Pixel Detector 2 DAQ Types SBC DAQ CE Module DAQ PPC 10 Gb Enet Experiment Chamber cPCI ADC RTOS Front End Board TOF/Momentum Instrument EPICS Config PVs Fast Serial 1 Gb E 2. Beam Line 120 Hz Data 3. EVR (Fiber) 5. SLAC WAN (Ethernet) 1. Channel Access (Ethernet) A. EPICS & Local Control (Hutch Subnet) F. DAQ Data to Cache 1 2 3 5 6 4 A F B C MPS PMC FPGA 1 Gb E EPICS Config PVs Spectrometer Monitor CCD Monitor 1 Gb E D E G. Visual Monitor Data G G 1 Gb E

7. Robert C. Sass Facility Advisory Committee April 16-17 2007 XES Controls/DAQ 7 CE Module block diagram Left side MFD FPGA 200 DSPs Lots of gates Xilinx XC4VFX60 Right side Configuration memory 128 Mbytes) Samsung K9F5608 Right side Memory (512 Mbytes) Micron RLDRAM II Right side Multi-Gigabit Transceivers (MGT) 8 lanes Left side 100-baseT Reset Reset options JTAG Right side PPC-405 (450 MHZ) Fabric clock Right side MGT clock

8. Robert C. Sass Facility Advisory Committee April 16-17 2007 XES Controls/DAQ 8 Dual CE Module reset options JTAGreset 100B-T10 GE 100B-T PHYs (0-16) CE May mix and match lanes to each CE footprint: ~ 50 cm 2 power: ~ 7 watts total + ~ 3/4 Watt/port Cost ~1K

9. Robert C. Sass Facility Advisory Committee April 16-17 2007 XES Controls/DAQ 9 CE Module Characteristics Memory Subsystem 512 Mbytes of RAM Sustains 8 Gbytes/sec “Plug-In” DMA interface (PIC) Designed as a set of IP cores Designed to work in conjunction with MGT and protocol cores Bootstrap loader (with up to 16 boot options and images) Interface to configuration memory Open Source R/T kernel (RTEMS) 10 GE Ethernet interface 100 base-T Ethernet interface Full network stack Utility software to manage I/O

10. Robert C. Sass Facility Advisory Committee April 16-17 2007 XES Controls/DAQ 10 LCLS Operations SCCS Experiment Operations Slow Archive Configs Archive Data Management System Overview 1. EPICS Channel Access Archive Configuration Interface Anywhere on Channel Access Networks Slow Archive Engines XES Hutch Archive Configurations Interface Data Acquisition Module(s) Modeling & Other Databases Global Archive Dictionary XES DAQ Data Cache Global Archive Configuration Management 6. Slow Archive Configuration 5. DAQ Archive Configuration 9. Global Archive Dictionary Management 3. Slow Archive Data 2. DAQ Archive Data 8. Offsite/Cache Archive Transfer Slow Archives Fast Archives DAQ Archives Fast Archive Config DAQ Archive Configs Anywhere on SCRAMNet Network Fast Archive Engine 4. Fast Archive Data 7. Fast Archive Configuration 1 2 3 4 5 6 7 8 9

11. Robert C. Sass Facility Advisory Committee April 16-17 2007 XES Controls/DAQ 11 Data Volume Makes DAQ Archive More Complicated SBC 10 GigE C E 1 GigE Ethernet Network Switch Local Cache Farm 10 GigE Links to Computer Center

12. Robert C. Sass Facility Advisory Committee April 16-17 2007 XES Controls/DAQ 12 Anywhere SCCS Slow Archives Server Archive Retrieval/Analysis System Overview LCLS Analysis / Viewing DAQ Archives Server Modeling & Other Databases Global Archive Dictionaries LCLS Data Retrieval Relational Database Server Fast Archive Server 2. Data Retrieval Request 1. Global Archive Dictionary Read 6. Slow Archive Data 7. Fast Archive Data 8. DAQ Archive Data 5. Database Data Tape Data 2 Slow Archives Fast Archives DAQ Archives Data Retrieval Temporary Cache 1 678 5 4. Tape Data Request 4 3. Locate server & storage 3 9. Results 9

13. Robert C. Sass Facility Advisory Committee April 16-17 2007 XES Controls/DAQ 13 Summary Minor slow controls issues. Data Acquisition and Archive are the major LCLS challenges. With CE for CCD and a commercial digitizer & SBC for spectrometers we can scale the DAQ needs of LCLS. With BaBar experience we can scale the data archive needs. In the near future we must: Specify and design an integrated data retrieval and analysis package that can be used anywhere and is user extensible. Specify and design a framework and tools for experimenters to use in designing and building their instruments. The CE provides a standard CCD interface Existing EPICS tools could be extended to construct the experiment- specific EPICS database and screens.