Towards a Homogeneous Software Environment for DAQ Applications Luciano Orsini Johannes Gutleber CERN EP/CMD

2 Topics The CMS DAQ model –RU, BU, EVM, FU –Event building process Interoperability (DAQ Column, demonstrator) Configuration Management Use of the model in testbeams Summary

3 Data Acquisition Baseline Detector Frontend Computing Services Readout Systems Filter Systems Event Manager Builder Networks Level 1 Trigger Run Control

4 DAQ Main Subsystems FS Interconnect CSN EVM RCN BCN BDN Interconnect FU BUOBUS BUMBUI RUORUS RUMRUI RU link FMU DDU FED RU BU LV1

5 Readout Unit RU FEDGTP EVM BCN RCN DSNInterconnect RUM RUO RUS RUI RDL BDN RUReadout Unit RUIReadout Unit Input RUMReadout Unit Memory RUOReadout Unit Output RUSReadout Unit Supervisor GTPGlobal Trigger Processor FEDFrontEnd Driver RDLReadout Data Link EVMEvent Manager RCNReadout Control Network BCNBuilder Control Network BDNBuilder Data Network DSNDAQ Service Network

6 Filter Column DSN FS BDN CSN BCN EVM BU RURCN BUBuilder Unit FSFilter Subfarm RUReadout Unit EVMEvent Manager RCNReadout Control Network BCNBuilder Control Network BDNBuilder Data Network DSNDAQ Service Network CSNComputing Service Network

7 Builder Unit BU FS EVMBCN RCN DSN BUM BUOBUS BUI RU BDN Interconnect BUBuilder Unit BUIBuilder Unit Input BUMBuilder Unit Memory BUOBuilder Unit Output BUSBuilder Unit Supervisor FSFilter Subfarm EVMEvent Manager RCNReadout Control Network BCNBuilder Control Network BDNBuilder Data Network DSNDAQ Service Network RUReadout Unit

8 BU vs. FU allocate take (id, data) FUBU RU EVM

9 BU vs. EVM allocate confirm (id) FUBU RU EVM

10 EVM vs. RU readout (id) FUBU RU EVM trigger data

11 BU vs. RU send (id) FUBU RU EVM cache (id, data)

12 BU to EVM FUBU RU EVM clear (id)

13 DAQ Interfaces BU allocate RU cache send allocate take readout trigger data clear confirm EVM FU

14 Test Beams RUI RUO BUI BM RM FLT Generic reusable module Test beam specific Detector frontend...

15 BM RMFLT Test Beam Scenarios RUI RUO RUI RUO BU BM RMFLT Detector frontend... RUI RUO BU Detector frontend... CPU FU

16 Trigger and Readout trigger() busyLow() readout(id) readyToRead(id)readyToSend(id) FLTRM interrupt BM data RUM RUI

17 Interoperability RUOBM BUI Interfaces

18 Interoperability Detector Frontend Computing Services Readout Systems Filter Systems Builder (Control & Data) Networks Level 1 Trigger Run Control Readout Manager Filter Manager RCS BU

19 Interoperability Step 1 –Identify Interface requirements between components (RU vs. BU, EVM vs. BU, etc.) = IRS Step2 –Definition of Interface as messages and their format + configuration = IDD Step3 –Implementation of API supporting interfaces = XDAQ

20 Interoperability Support integration of independently developed DAQ sub-systems in step 3 Interoperability through –established application message formats (using I 2 O private message frames) –transport (available communication media) –configuration (physical/logical addressing scheme)

21 Configuration Management Version Control through CVS –Controlled software repository –Organised multi-platform releases Release notes document (SRN) with every software baselines to track status and changes –Build and installation instructions –Description and capabilities –Changes from previous releases –Reference to Documentation

22 Summary Support development of DAQ applications according to a common model Implement compatible interfaces between subsystems (interoperability) Assemble event builders from basic software components Common approach to DAQ in testbeams and CMS DAQ –smooth transition from testbeam to CMS DAQ –reuse software written for offline and testbeams

DAQ Column Software Internals

24 Topics Design Issues DAQ Column set-up and configuration Software internals Status and plans

25 Design Issues Reflect I 2 O core environment –Event driven model, peer operations, homogeneous message passing model –Software download, predefined commands –Fill the blanks approach Every subsystem maps to a self contained software module –Encapsulation of a subsystem behaviour –Clean and narrow interface from/to other subsystems Location transparency of components –Automatic mapping of physical to logical addressing

26 DAQ Column Configuration DAQ Executives tasks RUIO ( iop480) PPC(mvme2304) Personal Computer (pcPentium)

27 Idle Waiting for I2O configuration messages Zzz.. zzz … zzz..

28 Received Configuration Where (e.g. IOP 34) Who (e.g RUO 3) How (e.g TCP, DLPI) What (e.g. BM, BUI 1, etc.) Detector Frontend Computing Services Readout Systems Filter Systems Event Manager Builder Networks Level 1 Trigger Run Control

29 Downloading What (e.g.libRU.so, libEVM.so) BM RM FLT RUI RUO

30 RUO BUI Operational Send (eventId) BM

31 Local Communication Efficiency Through executive  transparent local and remote use  higher overhead  no infinite recursion Direct invocation  fast  only local  limits the reusability in a distributed environment  danger of infinite recursion RM RUI

32 Message Delivery I2O header data Buffer pool RUO Peer Transports TCP Myri DLPI FIFO Peer Transport Agent

33 Func() { f… do… } Message Access I2O header data Buffer pool RUO Peer Transports TCP Myri DLPI FIFO

34 BU FU XDAQ Deployment FUBURUOBM RUOBM

35 Buffer Policies Buffer pools of several sizes –1 KB, 4KB, 8KB … 256 KB Long messages are built from scatter-gather lists –SGL header contains a list of memory locations and data sizes –Quality of Service parameters in receiving peer-transport allow Building of SGL in one contiguous memory block (must be provided, e.g. buffer pool 1MB or dynamically allocated) Building of a new SGL consisting of several blocks

36 Transport Configurations TCP Myri DLPI FIFO TCP Myri DLPI FIFO Thread per Peer Transport - higher OS service overhead + no CPU monopolisation + allows integration with other software Polling Peer Transport Agent + low OS service overhead - executive uses CPU continuously - no mixing of polling/blocking PTs

37 I2O Overview An architecture to develop hardware and OS independent device drivers –Uniform message passing protocol –Hardware access wrapper API –Event driven execution model –Predefined parameters and protocol for configuration –Dynamic configuration –Space for private extensions –Split of processing responsibilities to offload host CPU Provides interfaces (.h files and annotation) –e.g. message headers, scatter-gather lists

38 I2O Issues Use of documented messages –for configuration, control and data Provide means for configuration I2O model maps well to distributed environments Allows hardware evolution without the need to change the application components Use parts of the I2O architecture for modelling distributed DAQ systems XDAQ toolkit uses a subset of I2O features

39 I 2 O Private Message Format MessageSize MessageFlagsVersionOffset TargetAddressInitiatorAddressFunction = 0FFh InitiatorContext TransactionContext XFunctionCodeOrganizationID = 100h PrivatePayload = Function Parameters PrivatePayload

40 I 2 O and the XDAQ Toolkit Every DAQ subsystems maps into a set of I2O private device classes –(RU = RUI + RUO) Each class can be instantiated several times –(RU#1, RU#2, BU#1) Each class exposes a private and an I2O interface that can be accessed by other class instances –(private, configuration and control messages) Instances exchange messages Toolkit takes care of send and delivery

41 The XDAQ Toolkit Is a framework with C++ API –Messages from network are received and delivered to registered applications (callback mechanism) Application class method is associated with a specific message (binding at compile time) Classes dynamically register with the framework when they are downloaded User provides implementations for the predefined interfaces (abstract functions) Access to remote classes through asynchronous method invocation –through various configurations (TCP, PCI, Myrinet,...)

42 I 2 O Based Configuration ExecSysTabSet –Tell the executive about the physical addresses of other machines ExecDeviceAssign –Tell the executive about remote software modules with which it can communicate ExecSwDownload –Tell the executive which software module it hosts(and initialise) ExecSysEnable –Tell the executive to start the local software modules (if required) ExecIOPReset –Tell the executive to put the system into its initial state

43 I 2 O Based Configuration UtilParamSet –Set parameters for a DAQ software element –Set executive parameters UtilParamGet –Get parameters from a DAQ software element –Retrieve executive settings

44 External Interface Issues I2O provides messages for –Application control –Application configuration –Executive configuration Systems that generate I2O configuration and control messages can communicate with XDAQ Interface design documents exist –I2O specification: –EVB data format:

45 Status and Plans Interoperability test with DAQ demonstrator and benchmarks (ongoing) Availability –Solaris, VxWorks, Linux –LynxOS by Frederic Drouhin XDAQ in small DAQ setup provides well founded input to the DAQ TDR –True user and system requirements and design options XDAQ data acquisition toolkit –as vehicle for smooth transition from testbeam to CMS DAQ –tools for stand-alone data acquisition test software