Readout & Controls Update DAQ: Baseline Architecture DCS: Architecture (first round) August 23, 2001 Klaus Honscheid, OSU

Data Rates Input rate from Detector:1.5 TBytes/s 3x the rate from the PTDR Simulation group works on new event size estimate Noise? Includes 50% extra capacity Expansion options L1 Acceptance:1% ~20 GBytes/s to L2/L3 farm Event rate ~100 kHz L2/L3 Acceptance:5% ~ 200 MBytes/s (event size reduction in L3) ~ 4000 Hz

DAQ Architecture I Data Flow Front-end – DCB – L1buffer Non-trigger systems DCB can distribute data from one crossing to many L1Bs OR DCB can send all data from one crossing to a single L1B Trigger systems DCB send all data from one crossing to a single L1B Conclusion Before L1B the question of highways is only relevant for the implementation.

DAQ Implementation: DCB -> L1B Implementation based on highways offers: - Significant advantage/simplifications for L1B processing - Lower event rate and larger packets => easier implementation - (Pseudo) random distribution of crossings possible - No net cost difference DCB +$200K L1B - $200K Proposal: The DCB-L1B connections will be structured as 8 (*) highways. (*) strong preference for a fixed number of highways. 4, 6, 8 or 12 are Considered. For baseline cost estimate 8 highways are assumed.

DAQ Architecture II How much flexibility is needed in sending specific events to specific L2/L3 processors? Provide support to split DAQ into multiple logical partitions that can be operated independently. There is interest in getting the same event delivered to more than one L2/L3 node. Deliver data from consecutive crossings? Support partial event readout (for L2) and complete event readout for L2/L3 A partition (description) includes a list of L2/L3 nodes, trigger condition(s), output stream (?), required rate or kind of service (e.g. sampling) Clearly there would be rate/bandwidth concerns that need to be supervised. Highways need to be connected. Mostly a Global L1 and Event Manager issue. (At rates < 50 MBytes/s no difference between single or multiple highways) Yes

DAQ Implementation: L1B -> L2/L3 Farm Implementation based on highways offers: - Some cost savings ($ K) from smaller switches - Reduced control traffic (per highway) to signal L1 accepts etc. - Not difficult to change: Multiple highways can be combined by adding a second stage of identical switches. Single highway can be split by removing a stage of switches or reprogramming a larger switch. Proposal: The L1B – L2/L3 connections will be structured as 8 (*) highways. (*) strong preference for a fixed number of highways. 4, 6, 8 or 12 are Considered. For baseline cost estimate 8 highways are assumed.

System Overview

Example 1.5 TBytes/s System DCB - 48 serial input 1 Gbps (average 300 Mbps per link ) - 12 serial output 2 Gbps - option to double output links if input rate doubles - 6, 8 (*), or 12 highways DCBs (includes Pixel and Cal) - up to 24,192 FEBs or FEB equivalents Optical Links cables (1 per DCB, expandable to 2) - 12 fibers/cable (6048 total) - bandwidth = 1.5 TBytes/sec (expandable to 3.0) L1B - 24 serial input 2 Gbps - fiber to copper conversion external to L1B (for compatibility with trigger) - 1 serial output 1 Gbps (Gigabit Ethernet) L1Bs Event Builder - 12 switches X 48 ports/switch L2/3 Farm serial input 1 Gbps - up to 12 processors per link (with standard Gigabit to Fast Ethernet switch), 3024 total serial return 1 Gbps - same switch used for L2/3 input and output Storage -12 serial 1Gbps from Event Builder

Example 1.5 TBytes/s System DCB - 48 serial input 1 Gbps (average 300 Mbps per link ) - 12 serial output 2 Gbps - option to double output links if input rate doubles - 6, 8 (*), or 12 highways DCBs (includes Pixel and Cal) - up to 24,192 FEBs or FEB equivalents Optical Links cables (1 per DCB, expandable to 2) - 12 fibers/cable (6048 total) - bandwidth = 1.5 TBytes/sec (expandable to 3.0) L1B - 24 serial input 2 Gbps - fiber to copper conversion external to L1B (for compatibility with trigger) - 1 serial output 1 Gbps (Gigabit Ethernet) L1Bs Event Builder - 12 switches X 48 ports/switch L2/3 Farm serial input 1 Gbps - up to 12 processors per link (with standard Gigabit to Fast Ethernet switch), 3024 total serial return 1 Gbps - same switch used for L2/3 input and output Storage -12 serial 1Gbps from Event Builder

BTeV Controls? A lot of things are hiding behind this topic: 1.Configuration Run state transitions 2.Data Quality Monitor 3.Fast Interlock, Fire Alarm 4.“Classical” Slow Control 5.Calibration Run Control DAQ group provides skeleton software, hardware (Detector Manager) “Consumer” DAQ group provides skeleton software, hardware (?) Detector Control (DCS) DAQ group provides skeleton software, hardware (Detector Manager)

Typical Control System (CMS)

“Classical” Control System (MINOS) SCADA Analog/Digital channels, PLC, Fieldbuses LAN-ETHERNET OPI terminal OPI terminal OPI laptop Experiment Sub-Detectors & Equipment DB server Oracle DB FNAL Safety Server Beam Server PLC CAN fieldbus Beam-line Swics,BPM Magnets, Scalers RS232 LeCroy 1440 RS232 GPS IOS OPI WAN MIL/STD-1553B fieldbus Remote Workstations Local Workstations I/O Servers distributed in Experimental area OPC PLC fieldbuses GPIB IOS GPIB-ENET bridge OPI terminal

Supervisory Control and Data Acquisition Commercial systems, typically used in industrial production plants. Examples include LabView/BridgeView from National Instruments iFIX from Intellution (CDF, MINOS) EPICS (Babar, D0) PVSS II (CERN) OLE for Process Control Defines a standard to interface programs (SCADA) to hardware devices in a control system. Based on Microsoft’s COM/DCOM object model Provides multi-vendor inter-operability

SCADA Architecture User Process SCADA Utilities C,C++,VBA Wizards Engine Process Alarm handling Event/Alarm logging Historical trending Networking Device Servers HMI Logging & Archiving Handles distributed systems Reports Access Control Alarms Trending …

Control Example DAQ Group Detector Group

BTeV Control System (DCS) Solicit feedback from detector groups Treat infrastructure in similar fashion Rack monitoring Magnet Detector hall Evaluate SCADA software Develop/set up DCS test lab Develop sample solutions (HV?) Define DAQ – DCS connection relevant for HV, Pixel “motor” Calibration