June, 2000NIU Workshop Online DAQ System: From Detector to Tape T. Yasuda Fermilab

June, 2000NIU Workshop Overview Hardware Control System Primary/Secondary Data Path DAQ Applications DAQ in action Conclusions

June, 2000NIU Workshop Overview The DØ DAQ system is divided into two components: Trigger system Level 1 hardware trigger components Level 2 specialized processors Level 3 crate readout and software trigger components Online or Host system Detector controls Data Logging Monitoring Control room applications

June, 2000NIU Workshop Overview DAQ Architecture –Event data rate and operational redundancy achieved by a high degree of parallelism Level 3 Host –Capability for multi-user, multi-stream operation With central resource configuration manager –Network-centric Host design

June, 2000NIU Workshop DAQ Components Detector FCC UNIX Servers NT Level 3 Linux PCs Control Room PCs Controls Trigger and Readout

June, 2000NIU Workshop Hardware Description 3 Compaq/Digital Alpha Servers –d0ola: Alpha Server 4000, 1 processor, 466MHz, 500 MB memory –d0olb: Alpha Server 4000, 2 processors, 600 MHz, 500 MB memory –d0olc: Spec out by Aug 3. Probably Alpha Server ES40, 4 processors, 667 MHz –Clustered / redundant –500 GB shared RAID disks for online apps and database (Mirrored) –500+ GB local ‘data buffer’ disks, fiber channel based (40M/sec)

June, 2000NIU Workshop Hardware Description Linux/NT nodes –Buying 6 nodes with dual PIII, 600 MHz, 500MB memory, 2 graphics cards –3 Linux nodes and 3 NT nodes exist –Will run Vmware on Linux nodes Control system embedded 68Ks and PowerPCs (VxWorks) Network –Cisco 6509 GB Ethernet switch for all FCH nodes –Satellite 100 MB switches in MCH –GB fiber to FCC Security –Access control filter to online machines –Kerberos authenticated ssh session only

June, 2000NIU Workshop Control & Monitoring Detector Readout Crate Controls Crate Trigger and Readout UNIX ServersControl Room PCs EPICS Clients: Low Voltage High Voltage Rack Monitor 1553 Devices SMT Monitor FT Monitor etc EPICS Clients: Low Voltage High Voltage Rack Monitor 1553 Devices SMT Monitor FT Monitor etc Controls Ethernet 1553 ORACLE Hardware Database EPICS DB Generator Vertical Interconnect

June, 2000NIU Workshop Control System Built upon EPICS control system –A ‘standard’ toolkit upon which we’ve built DØ extensions –Lots of user-community supplied tools ORACLE Hardware Database –Extract EPICS db from ORACLE –Web-based and batch interfaces Hardware Control –Low Voltage, High Voltage, etc dedicated GUI applications Downloading –Registers, Pedestals, etc Significant Event (alarm) System Interface to Accelerator and Cryogenics controls systems

June, 2000NIU Workshop Status of Control System Calorimeter –Preamp PS, BLS PS, ADC PS control exist –Pulser control (in progress) SMT –EIPCS records for Sequencer, Sequencer control, VRB, VRBC, Emulator (in progress) exist and used in the test stands Muon –Tested communication using 1553 for PDT, MDT, SRC cards FPD –Used RM support to control motors

June, 2000NIU Workshop Status of Control System Luminosity –Scalers and FE processing results communicated to Accelerator via ACNET Cryo –Communicated with DMAX system Common Tools –generic 1553 support –generic VME support –HV used in SMT, Muon, Lum V1 running for months, V2 work starting –Diagnostic support for busses –Standard operator interface (GUI)

June, 2000NIU Workshop Hardware Database Describes Control aspects of electronics Based on ORACLE –2 instances of the database ( dev, user testing) Web based interface for entering, modifying, deleting records Python script for batch entries exists Calorimeter records in the database

June, 2000NIU Workshop Hardware Database

June, 2000NIU Workshop Significant Event (Alarm) System System to detect alarm conditions and state changes in the DAQ system –Server with DAQ components as clients –COOR sends alarm and run control messages –CR, DL, DD send alarm messages –Version 1 Display exists –Working on version 2 Display (Summer student) –Need to integrate EPICS alarms into the system (Fall 2000) on IOC EPICS alarms -> ITC client ITC client sends alarms to Server on host EPICS Alarm Handler can be used for now

June, 2000NIU Workshop Significant Event System Significant Event Server Fault Watcher Archiver HV Control Heartbeat Display Front-End Run Control (COOR) Process Periodic Heartbeat Run Suspend F F F F F F Filter SE Message Filtered Message

June, 2000NIU Workshop Configuration & Run Control Detector L1, L2, TCC L3 Supervisor L3 VRC Readout Crate L3 Filter FCC Data Cable Controls Crate UNIX Servers NT Level 3 Ethernet Linux PCs Control Room PCs ControlsTrigger and Readout Run Control Client Run Control Client Comics DSM COOR Collector / Router Data Logger Disk Data Distributor EXAMINE RIP 1553 Vertical Interconnect

June, 2000NIU Workshop Software Description Configuration Management and Run Control –Coordination (COOR) –User interface (TAKER) –Download manager (COMICS) Primary event path –DAQ State Manager (DSM) –Collector / Router –Data Logger –Event metadata manager (SAM) –Event data manager (enstore) Secondary event path –Secondary DAQ Supervisor –Data Merger

June, 2000NIU Workshop Software Description Event monitoring –Data Distributor –Analysis applications (EXAMINE) DAQ Monitoring –Client/Server access to DAQ flow statistics, trigger rates, etc Detector Monitoring –Front End active & parasitic monitors Calibration –Client/Server interface to database Infrastructure –Databases (ORACLE) –Task-to-task communication (ITC)

June, 2000NIU Workshop Secondary DAQ Data Flow Controls/Readout Crates 1553 Bus VME Bus FCC Linux PCs UNIX Server EXAMINE UNIX Server Collector / Router Data Logger RIP Disk Collector / Router Data Logger Data Logger Disk Data Distributor Data Distributor RIP Data Merge Control Room PCs EPICS CA Client EPICS CA Client Monitor GUI Examine GUI Examine GUI ITC Detector CA Link EPICS CA Server EPICS CA Server Detector DAQ Shared Data

June, 2000NIU Workshop Secondary DAQ System Alternative data path Mainly used for monitoring and calibration Takes advantage of powerful front- end processors Uses the same data path as the primary path after Data Merger

June, 2000NIU Workshop DAQ Monitor Monitors the status of DAQ subsystems (L1/2, CR, DL, DD) Collects statistics information from the subsystems C++ itc Server with python Display clients

June, 2000NIU Workshop DAQ Monitor

June, 2000NIU Workshop Event Monitoring: EXAMINE Samples and reconstructs event based on Stream IDs and trigger IDs Clients of Distributor –Network and file event transfer modes work –Calorimeter EXAMINE used for preamp testing –CFT EXAMINE Getting ready for raw data unpacking MC packed data? –SMT EXAMINE used for SiDet data –Muon EXAMINE used for commissioning

June, 2000NIU Workshop Event Monitoring: EXAMINE Detector L1, L2, TCC L3 Supervisor L3 VRC Readout Crate L3 Filter Collector / Router Data Logger Disk Data Cable UNIX Servers NT Level 3 Ethernet Linux PCsControl Room PCs Data Distributor EXAMINE Express Line ROOT Client Trigger and Readout

June, 2000NIU Workshop Event Monitoring: EXAMINE –Need: L3 EXAMINE Vertex EXAMINE Preshower EXAMINE –Planned improvements Histoscope -> Root after NIU workshop on-the-fly histogram e-browser Framework improvement –name server for accessing only the histograms

June, 2000NIU Workshop Online Event Display

June, 2000NIU Workshop Online Calibration Perform electronics calibration of sub-detectors and insert results to ORACLE database COOR controlled via Taker Common server and database interface for all sub-detectors Calibration results transmitted as special event messages through DAQ paths Current status –Successfully ran SMT calibration at 1% test stand and NW test stand

June, 2000NIU Workshop Calibration Manager Taker COOR COMICS Crates Calibration Manager Calibration Database Calib. Data Processor Database Interface Configure Request start run Request download Download Start run End run Data Pedestals Gains Pedestals Gains Comparison Results Database Access End calib. Start run, End run Validator Calib Manager Display

June, 2000NIU Workshop Online DAQ in Action Electronics/DAQ Commissioning –2 VRB crates with 10 cards each, synchronized with SCL from TFW to L3 –L1 muon crate –1 Muon Scint crate with 2 MRCs –1 Muon PDT crate with 1 MRC –1 Calorimeter crate –Combinations of 2 systems done but not with MCH2+MCH3 –2 simultaneous runs done 3 simultaneous runs require one more L3 node or script runner

June, 2000NIU Workshop Online DAQ in Action SMT Test Stands –1% and NW test stands 1 HDI, 1 Sequencer, 1 VRB, 1 VRBC, 1 VBD Download done with COMICS and database –10% test stand 3 HDIs, a few Interface Boards, a few Sequencers, a few VRBs, 1 VRBC, controler, 1 VBD + L3 Download done by spread sheet for now –Databases (Electronics) exist for all three stands –Calibration run performed at 1% and NW test stands

June, 2000NIU Workshop Online DAQ in Action Commissioning Run –Two detectors installed for the upcoming Commissioning Run Run I Luminosity scintillation counters Forward Proton Detector –Both detectors will be read out using the Run II Online system. –Data will be transferred to and from the Accelerator Controls System via the EPICS/ACNET Gateway.

June, 2000NIU Workshop Conclusions All of the DAQ components exist and function. Improvements are implemented daily following user suggestions. We have been intimately involved in daily commissioning activities for the past few months. Bring in your sub-detectors!!