DUT integration DAQ buffers & data format Peter Fischer for Hans Krüger Physikalisches Institut, Universität Bonn Peter Fischer Institut für Technische Informatik, Universität Mannheim Presentation given at the EUDET / JRA-1 DAQ Meeting, 14.12.2005, CERN

Comparison of 3 possible approaches How can the DAQ of the telescope and the DUTs be merged ? First: study how hardware is connected Slides basically as presented by Hans during brainstorming @ DESY Then: look at data flow. DUT integration

Option1: Integration ‘on hardware level’ use special purpose hardware interface to read out everything DUT users must comply to hardware specs use one integrated DAQ software Problems: all users must use special interface & DAQ interface can have its limitations for future DUTs for new devices, existing DAQ software must be modified – this needs experts Probably large overhead when using in lab Where to put PC? in area: monitoring difficult in control room: long cables DUT telescope proprietary bus Trigger DAQ HW standard bus interface interface DAQ ctrl PC file DUT integration

Option 2: Integration ‘on software level’ DUTs provide their own DAQ hardware They can use ‘any’ PC interface stll use one integrated DAQ software Problems: ‘The’ DAQ PC must provide required interface for new devices, existing DAQ software must be modified – this needs experts Probably still overhead when using in lab Where to put PC? DUT telescope Trigger DAQ HW standard bus interfaces intfc intfc DAQ ctrl PC file DUT integration

Option 3: Integration ‘on data level’ Use completely different hardware for the DUTs Connect DUT to a seperate PC Readout Software is provided by the DUT user. It sends only DATA to DAQ. This can be on same PC or via TCP/IP Common DAQ is only on one PC Monitoring can be on ‘any’ PC Problems: How to make sure that devices are configured correctly at start of run. DUT telescope Trigger DAQ HW standard bus interfaces intfc PC intfc tel. ctrl IPC DUT ctrl file PC DUT integration

trigger interface first idea: 2(3) cables between trigger logic unit (TLU) and modules (incl. DUTs): (reset) - Set trigger counters to 0. - Broadcast from TLU to modules - May be replaced by software command, but this needs communication between run control task and modules. trigger - Read out one event. - Broadcast from TLU to modules - Modules increment trigger counter after each trigger. busy - Set by device as long as readout is active or when buffers are full. - One line from each module to TLU. (TLU can see who blocks readout!) physical format: NO NIM !!!! 3.3V TTL (?) LVDS (?, cannot use LEMOs) TLU reset module 1 trigger busy1 PC ??? module 2 busy2 maybe one reset/trigger output per module is better ! DUT integration

Present DEPFET ‘Mini-DAQ’ PC based, Windows based. Very light weight. Hardware interface to our present telescope: PCI Hardware Interface to the DEPFET: USB Software is divided into many parallel tasks: several Producer tasks read the hardware one Writer task bundles events, writes to file and sends subsets for monitoring several Online - Monitoring tasks. They request a fraction of the data from the writer one Buffer Monitor task and a main controller a Reader can re-inject data into the monitoring Central data flow element: ‘shared buffers’ Used between Producers and Writer and between Writer and Monitors Events have unambiguous trigger ID in the buffers for later sorting Mutex elements are used to control access of the buffers Uses begin-of-run (BORE), data (DATA), end-of-run (EORE) events. DUT integration

Present DEPFET ‘Mini-DAQ’ Hardware Hardware Hardware telescope ‘producer’ task DEPFET ‘producer’ task other ‘producer’ tasks DAQ buffers Writer task file Monitoring buffers Monitoring task 1 Monitoring task 2 DUT integration

BORE and EORE events BORE (Begin Of Run Event): time, date, … detector configuration data (bias values, thresholds, pedestals, …) EORE (End Of Run Event): error flags statistics for consistency checks (total number of data words,…) … DUT integration

Shared Buffers Organize data transport between tasks. Two types: DAQ buffers from DAQ tasks to WRITER Monitoring Buffers from WRITER to MONITORING tasks Two port buffer: one task can write, one task can read (no checks!) Very simple & efficient: 330 lines of source code for class implementation writing & reading can be done simultaneously, arbitration only needed at start / end of transfers. Can be extended to send data from PC to PC (via TCP/IP) Buffers are created by the writing task. (If they already exist, the task connects to the existing buffer) Buffers automatically disappear when nobody refers to them any more. DUT integration

Shared Buffer header (part) class TSharedBuffer { static const unsigned int MAX_SHARED_BUF; public: TSharedBuffer (AnsiString Name, unsigned int Size, SBufferType Type = BUFTYPE_DAQ); ~TSharedBuffer (); bool Created; // flags if buffer was newly created or already existed bool IsEmpty (void); float GetFillLevel (void); // return fill level unsigned int * GetWriteAddress (unsigned int BlockSize); // request a pointer to write void FinishWrite (unsigned int * NextPos); // flag that we are done unsigned int GetDataSize (void); // see how much data is there unsigned int * GetReadAddress (unsigned int * Size); // get data address (and size) void FinishRead (unsigned int * NextPos); // release data until NextPos void Clear (void); // delete all data ... } DUT integration

data structure in the shared buffers one word = unsigned integer (32 bit) data is organized in events with a 2 word header: word 0: event type begin of run BORE end of run EORE data DATA event size (including header) size event content group data GROUP detector data device / module / flags word 1: trigger number start with 0, magic numbers for BORE/EORE word 2…size: size-2 data words DUT integration

present bit allocation Device Type: 0x0: BORE 0x1: EORE 0x2: DATA 0x3: unused Event Type: GROUP BAT DEPFET 13 x unused Module Flags word 1 31 28 27 24 23 22 21 20 19 Size (including header) = N+2 word 2 31 trigger word 3 31 data word 1 31 data word 2 word N+2 31 data word N DUT integration

packets in the buffers DAQ buffer (single events): monitoring buffers and file (GROUP Events!): BORE - info word BORE - info word: GROUP type trigger = magic BORE word trigger = magic BORE word BORE data 1 BORE BORE 1 BORE       BORE data N BORE N DATA- info word DATA - info word: GROUP type 0x00000000 0x00000000 DATA data 1 event #1 DATA event 1 event #1       DATA data N DATA event N DATA- info word DATA - info word: GROUP type 0x00000001 0x00000001 DATA data 1 event #2 DATA event 1 event #2       DATA data N DATA event N       EORE - info word EORE - info word: GROUP type trigger = magic EORE word trigger = magic EORE word EORE data 1 EORE EORE event 1 EORE       EORE data N EORE event N DUT integration

GROUP events GROUP events are used to bundle data for ONE TRIGGER from several sources. They are written to file and sent to the monitoring tasks. data events within GROUP events are identical to full single events this gives a small overhead, as trigger ID is repeated but it makes copying and analysis much simpler DUT integration

summary Pipelined operation Small tasks keep units simple Clearly defined interface at buffer ends Extensions are simple flexible choice of operating system Several small monitoring tasks can run in parallel Later: Preprocessed data (tracks) can be included into the data flow and be used by monitoring tasks Send buffers through TCP/IP (work in progress) to relief main DAQ computer Still needed: Overall control task (flush all buffers, start/stop run,…) Sending stuff over TCP/IP Better control over detector configuration Logfile … DUT integration

example of possible ‘advanced’ data flow DUT DAQ PC - WIN (beam area, via VNC) main PC - WINDOWS (counting room) DUT1 hardware USB Producer TCP Mon 1 DUT2 hardware PCI Producer writer MON PC - LINUX (remote) telescope DAQ PC - LIN (beam area, via VNC) telescope hardware VME Producer TCP Mon 2 TCP TDC hardware CAMAC Producer DUT integration

further remarks Trigger hardware must guarantee same event ID’s in all detectors Must be able to inject dummy triggers. Therefore, TLU could also be read out to determine type of trigger Would be nice to get event time w.r. to main clock (TDC) DUT integration

from Hans: DEPFET Prototype System DEPFET sensors 64 x 128 pixels, 36,5 x 28 µm2 (Q1 2007: 512 x 512 pixels, 33 x 24 µm2) Event rate old system: 10 Hz (various limitations) new system: ~1 kHz (w/o zero supp.), data transfer limited (20 Mbyte/s, USB 2.0) with zero suppression: ~100 kHz (theor.), row clock rate limited (10 MHz) DAQ USB 2.0 interface to Win2k/XP PC independent processes for slow control, file writing and online-monitoring inter-process communication via ‘shared memory’ buffers DUT integration

from Hans: Read-out System Hardware Hybrid DEPFET matrix 2 Switcher + Curo 2 transimpedance amplifiers S3A Board Mixed signal board Dual 65 MHz ADCs FPGA + 256k SRAM USB 2.0 interface card DUT integration