1. DUT integration DAQ buffers & data format
Peter Fischer for Hans Krüger
Physikalisches Institut, Universität Bonn
Peter Fischer
Institut für Technische Informatik, Universität Mannheim
Presentation given at the EUDET / JRA-1 DAQ Meeting, , CERN

2. Comparison of 3 possible approaches
How can the DAQ of the telescope and the DUTs be merged ?
First: study how hardware is connected
Slides basically as presented by Hans during DESY
Then: look at data flow.
DUT integration

3. Option1: Integration ‘on hardware level’
use special purpose hardware interface to read out everything
DUT users must comply to hardware specs
use one integrated DAQ software
Problems:
all users must use special interface & DAQ
interface can have its limitations for future DUTs
for new devices, existing DAQ software must be modified – this needs experts
Probably large overhead when using in lab
Where to put PC? in area: monitoring difficult in control room: long cables
DUT
telescope
proprietary bus
Trigger
DAQ HW
standard bus interface
interface
DAQ ctrl PC
file
DUT integration

4. Option 2: Integration ‘on software level’
DUTs provide their own DAQ hardware
They can use ‘any’ PC interface
stll use one integrated DAQ software
Problems:
‘The’ DAQ PC must provide required interface
for new devices, existing DAQ software must be modified – this needs experts
Probably still overhead when using in lab
Where to put PC?
DUT
telescope
Trigger
DAQ HW
standard bus interfaces
intfc
intfc
DAQ ctrl PC
file
DUT integration

5. Option 3: Integration ‘on data level’
Use completely different hardware for the DUTs
Connect DUT to a seperate PC
Readout Software is provided by the DUT user.
It sends only DATA to DAQ. This can be on same PC or via TCP/IP
Common DAQ is only on one PC
Monitoring can be on ‘any’ PC
Problems:
How to make sure that devices are configured correctly at start of run.
DUT
telescope
Trigger
DAQ HW
standard bus interfaces
intfc PC
intfc
tel. ctrl
IPC
DUT ctrl
file PC
DUT integration

6. trigger interface
first idea: 2(3) cables between trigger logic unit (TLU) and modules (incl. DUTs):
(reset) - Set trigger counters to Broadcast from TLU to modules - May be replaced by software command, but this needs communication between run control task and modules.
trigger - Read out one event. - Broadcast from TLU to modules - Modules increment trigger counter after each trigger.
busy - Set by device as long as readout is active or when buffers are full. - One line from each module to TLU. (TLU can see who blocks readout!)
physical format:
NO NIM !!!!
3.3V TTL (?)
LVDS (?, cannot use LEMOs)
TLU
reset
module 1
trigger
busy1 PC
???
module 2
busy2
maybe one reset/trigger
output per module is better !
DUT integration

7. Present DEPFET ‘Mini-DAQ’
PC based, Windows based. Very light weight.
Hardware interface to our present telescope: PCI
Hardware Interface to the DEPFET: USB
Software is divided into many parallel tasks:
several Producer tasks read the hardware
one Writer task bundles events, writes to file and sends subsets for monitoring
several Online - Monitoring tasks. They request a fraction of the data from the writer
one Buffer Monitor task and a main controller
a Reader can re-inject data into the monitoring
Central data flow element: ‘shared buffers’
Used between Producers and Writer and between Writer and Monitors
Events have unambiguous trigger ID in the buffers for later sorting
Mutex elements are used to control access of the buffers
Uses begin-of-run (BORE), data (DATA), end-of-run (EORE) events.
DUT integration

8. Present DEPFET ‘Mini-DAQ’
Hardware
Hardware
Hardware
telescope
‘producer’ task
DEPFET
‘producer’ task
other
‘producer’ tasks
DAQ buffers
Writer task
file
Monitoring
buffers
Monitoring task 1
Monitoring task 2
DUT integration

9. BORE and EORE events BORE (Begin Of Run Event):
time, date, …
detector configuration data (bias values, thresholds, pedestals, …)
EORE (End Of Run Event):
error flags
statistics for consistency checks (total number of data words,…) …
DUT integration

10. Shared Buffers Organize data transport between tasks. Two types:
DAQ buffers from DAQ tasks to WRITER
Monitoring Buffers from WRITER to MONITORING tasks
Two port buffer: one task can write, one task can read (no checks!)
Very simple & efficient:
330 lines of source code for class implementation
writing & reading can be done simultaneously, arbitration only needed at start / end of transfers.
Can be extended to send data from PC to PC (via TCP/IP)
Buffers are created by the writing task. (If they already exist, the task connects to the existing buffer)
Buffers automatically disappear when nobody refers to them any more.
DUT integration

11. Shared Buffer header (part)
class TSharedBuffer {
static const unsigned int MAX_SHARED_BUF;
public:
TSharedBuffer (AnsiString Name, unsigned int Size, SBufferType Type = BUFTYPE_DAQ);
~TSharedBuffer ();
bool Created; // flags if buffer was newly
created or already existed
bool IsEmpty (void);
float GetFillLevel (void); // return fill level
unsigned int * GetWriteAddress (unsigned int BlockSize); // request a pointer to write
void FinishWrite (unsigned int * NextPos); // flag that we are done
unsigned int GetDataSize (void); // see how much data is there
unsigned int * GetReadAddress (unsigned int * Size); // get data address (and size)
void FinishRead (unsigned int * NextPos); // release data until NextPos
void Clear (void); // delete all data
... }
DUT integration

12. data structure in the shared buffers
one word = unsigned integer (32 bit)
data is organized in events with a 2 word header:
word 0: event type begin of run BORE end of run EORE data DATA event size (including header) size
event content group data GROUP detector data device / module / flags
word 1: trigger number start with 0, magic numbers for BORE/EORE
word 2…size: size-2 data words
DUT integration

13. present bit allocation
Device Type:
0x0: BORE
0x1: EORE
0x2: DATA
0x3: unused
Event Type:
GROUP
BAT
DEPFET
13 x unused
Module
Flags
word 1 31 28 27 24 23 22 21 20 19
Size (including header) = N+2
word 2 31
trigger
word 3 31
data word 1 31
data word 2
word N+2 31
data word N
DUT integration

14. packets in the buffers DAQ buffer (single events):
monitoring buffers and file (GROUP Events!):
BORE - info word
BORE - info word: GROUP type
trigger = magic BORE word
trigger = magic BORE word
BORE data 1
BORE
BORE 1
BORE
  
  
BORE data N
BORE N
DATA- info word
DATA - info word: GROUP type 0x 0x
DATA data 1
event #1
DATA event 1
event #1
  
  
DATA data N
DATA event N
DATA- info word
DATA - info word: GROUP type 0x 0x
DATA data 1
event #2
DATA event 1
event #2
  
  
DATA data N
DATA event N
  
  
EORE - info word
EORE - info word: GROUP type
trigger = magic EORE word
trigger = magic EORE word
EORE data 1
EORE
EORE event 1
EORE
  
  
EORE data N
EORE event N
DUT integration

15. GROUP events
GROUP events are used to bundle data for ONE TRIGGER from several sources.
They are written to file and sent to the monitoring tasks.
data events within GROUP events are identical to full single events
this gives a small overhead, as trigger ID is repeated
but it makes copying and analysis much simpler
DUT integration

16. summary Pipelined operation Small tasks keep units simple
Clearly defined interface at buffer ends
Extensions are simple
flexible choice of operating system
Several small monitoring tasks can run in parallel
Later:
Preprocessed data (tracks) can be included into the data flow and be used by monitoring tasks
Send buffers through TCP/IP (work in progress) to relief main DAQ computer
Still needed:
Overall control task (flush all buffers, start/stop run,…)
Sending stuff over TCP/IP
Better control over detector configuration
Logfile …
DUT integration

17. example of possible ‘advanced’ data flow
DUT DAQ PC - WIN
(beam area, via VNC)
main PC - WINDOWS
(counting room)
DUT1
hardware
USB
Producer
TCP
Mon 1
DUT2
hardware
PCI
Producer
writer
MON PC - LINUX
(remote)
telescope DAQ PC - LIN
(beam area, via VNC)
telescope
hardware
VME
Producer
TCP
Mon 2
TCP
TDC
hardware
CAMAC
Producer
DUT integration

18. further remarks
Trigger hardware must guarantee same event ID’s in all detectors
Must be able to inject dummy triggers. Therefore, TLU could also be read out to determine type of trigger
Would be nice to get event time w.r. to main clock (TDC)
DUT integration

19. from Hans: DEPFET Prototype System
DEPFET sensors
64 x 128 pixels, 36,5 x 28 µm2
(Q1 2007: 512 x 512 pixels, 33 x 24 µm2)
Event rate
old system: 10 Hz (various limitations)
new system: ~1 kHz (w/o zero supp.), data transfer limited (20 Mbyte/s, USB 2.0)
with zero suppression: ~100 kHz (theor.), row clock rate limited (10 MHz)
DAQ
USB 2.0 interface to Win2k/XP PC
independent processes for slow control, file writing and online-monitoring
inter-process communication via ‘shared memory’ buffers
DUT integration

20. from Hans: Read-out System Hardware
Hybrid
DEPFET matrix
2 Switcher + Curo
2 transimpedance amplifiers
S3A Board
Mixed signal board
Dual 65 MHz ADCs
FPGA + 256k SRAM
USB 2.0 interface card
DUT integration