1. Electronics for SuperB DAQ
Dominique Breton
(with the help of Christophe Beigbeder & Jihane Maalmi)
Rome SuperB meeting – December 17th 2008

2. Summary of BABAR implementation
FCTS
60MHz clock
60Mbits/s links
L1 accept
Read event
1Gbit/s
Optical links
ROM
FEC
60MHz clock
DAQ
Setup and control 16
Setup and control
16 serialized commands sent in parallel every clock cycle
Subsystems
control
Same command words serialized at 60MHz
(geographically dedicated lines)
Dominique Breton – Rome SuperB meeting – December 17th 2008

3. Dominique Breton – Rome SuperB meeting – December 17th 2008
BABAR implementation: main points
What is really specific to BABAR acquisition system : mainly the fact that the ROM is the only link to FEE, through a pair of optical fibers taking care of :
60MHz clock
Encoded fast commands (L1 accept, Read_event, …)
Event data readout.
Parameter setup and control.
Event data is not sent to ROM upon L1 accept but upon Read_event.
This permits an easier handling of ROM buffers
But this increases the size of the buffers on the FEE
Pros : simple unique bidirectionnal links between ROMs and FEE
Cons : in order not to add more jitter to the L1 accept, extra latency equal to its length has to be added if one wants to interleave it with other fast commands.
Concerning trigger: due to accelerator and detector constraints, the latency is rather long and has a certain jitter, this leading to a rather large (1µs) time window for data readout.
=> we can keep the same philosophy, but we need to make it run much faster.
Dominique Breton – Rome SuperB meeting – December 17th 2008

4. Dominique Breton – Rome SuperB meeting – December 17th 2008
BABAR philosophy
ROM-to-FEE “global” commands last 12 clock periods.
5 bits of trigger tag are included in the L1 accept command.
“Local” commands are sent on the same link. Their length can go up to 32 bits.
FE elements are addressed geographically through bit position in a 16-bit word.
This is fine for local commands because one avoids using address bits.
But this is a huge loss of bandwidth for global commands which are sent 16 times in parallel.
Seen the trigger rates in BABAR (a few kHz), this wasn’t a problem.
=> In order to cope with the new requirements, we’ll have to change the way of mixing commands in SuperB.
FCTS architecture on FEE could thus get closer to that of LHC experiments, where clock and L1 accept are decoded at the end of the fiber and distributed on dedicated links to the detector electronics.
FCTS-to-ROM L1 commands are the bottleneck for the L1 accept minimum spacing because of frame length (104 bits) and 60Mbit/s rate.
As this is really a problem, we should use multi-Gbits/s serialized links for FCTS-to-ROM links.
Dominique Breton – Rome SuperB meeting – December 17th 2008

5. Dominique Breton – Rome SuperB meeting – December 17th 2008
Moving to SuperB (1)
With the radiation, the smaller = the better for the FEE buffer size.
A solution with small buffers on the FEE is better if feasible (safe storage is especially hard close to beam, especially with radiation).
Reducing L1 trigger latency might be possible with modern electronics.
Making use of a 2.5Gbits/s link, 32 bits of command can be sent within 16ns between ROMs and FEE.
This can cover both global and local commands, even including address bits.
Then 2 options are feasible for distributing L1 accept to FE elements :
through a single pulse (« à la LHC ») => trigger tagging has to be performed in the FEE (by sampling a counter)
through a 60MHz serial command (« à la BABAR ») => takes more time but gives less work to FEE
In the FEE, once data has been transferred from the latency buffer to the multi-event buffer, only the mean trigger rate has to be considered for data transfer.
This offers up to 25kbits/event/link.
The multiplexing ratio of the FEE towards the links will then depend on the amount of data to read per channel.
It is important to define early enough the exact amount of necessary data time slices per event for each subdetector in order to scale the system.
Dominique Breton – Rome SuperB meeting – December 17th 2008

6. Example of possible SuperB electronics implementation
FCTS
Max jitter ~ 15ps rms
& no phase spread.
(+ specific solution for TOF ?)
60MHz clock
L1 accept
2.5Gbits/s
Optical links
Read event
ROM
FEC
60MHz clock
L1 accept
DAQ
Read event (?)
Setup and control 16
Setup and control
1 32-bit command word every clock cycle
Subsystems
control
Same command word serialized at 60MHz
(dedicated or broadcasted)
Dominique Breton – Rome SuperB meeting – December 17th 2008

7. Dominique Breton – Rome SuperB meeting – December 17th 2008
Clock and command distribution
FCTS -> ROM <-> FEE :
Mixing of clock and data in the same link is a source of jitter for the recovered clock
Moreover, the phase difference between input and output of commercial fast deserializers may not be constant each power-up: due to internal PLLs – not true with Glink)
Available COTS transceivers with a few Gigabits/s are not numerous
This is due to the fact that fast serializers are embedded inside all the new high-end FPGAs (they have the same phase and jitter problems)
In addition, these FPGAs are not safe where there is radiation on the detector side
The same problems are posed for Super-LHC
CERN started a work package about this subject, mixing FPGA off-detector and a set of new rad-hard ASICs on-detector, called “GBT” (GigaBitTransceiver)
Their solution covers the two directions (control and readout)
The solution chosen should also fit the requirements of ps-sensitive detectors (TOF PID)
Use of crystal-based jitter cleaning PLLs could clean up the deterministic jitter if necessary (like QPLL for LHC)
Otherwise, a dedicated clock distribution scheme will have to be foreseen for these detectors …
Dominique Breton – Rome SuperB meeting – December 17th 2008

8. Dominique Breton – Rome SuperB meeting – December 17th 2008
The GBT project
Single link serves:
Readout (DAQ)
High speed unidirectional (up-link)
Trigger data (up-link)
Timing Trigger and Control (TTC)
Clock reference and synchronous control (down-link)
Excellent timing accuracy
Low and fixed latency
Experiment control (SC/DCS/ECS)
Modest bandwidth (bidirectional link)
Custom ASICs in the detectors:
Radiation tolerant: Total dose & Single Event Upsets (SEU)
Commercial, Off-The-Shelf (COTS) in the counting room
FPGAs used to implement multi-way transceivers
Optical Link - April 2008
Dominique Breton – Rome SuperB meeting – December 17th 2008

9. Dominique Breton – Rome SuperB meeting – December 17th 2008
The GBT chipset
32 bits
@ 80MHz ?
CLK,
L1 Accept,
CALIB
GBTX
GigaBit TransImpedance Amplifier (GBTIA)
Detect the optical power
Amplify the detected signal
GigaBit Laser Driver (GBLD)
Modulate the laser diode current
Bias the laser diode
Works with singlemode (1310nm) and multimode (850nm) fibers
GigaBit Transceiver (GBTX)
Serialize/de-serialize the data
Interface with:
- Front-end ASICS
- GBT-SCA
GBT – Slow Control ASIC (GBT-SCA)
Function:
- Slow control
- Monitoring
Optical Link - April 2008
Dominique Breton – Rome SuperB meeting – December 17th 2008

10. From Commercial to Versatile link
It looks like the emerging SFP+ transceivers are worthy of attention
What needs to be customized?
Work with Industrial Partner
Based on the (tentative) requirements, look at those items where commercial specs deviate
Radiation resistance - all ASICs, Lasers, Photodiodes
Magnetic field tolerance
Materials and dimensions
Low temperature operation
Optical Link - April 2008
Dominique Breton – Rome SuperB meeting – December 17th 2008

11. GBTX – GigaBit Transceiver
Functionality:
Clock and Data Recovery (CDR)
De-Serializer
Serializer
Data Encoder/Decoder:
- Line coding (DC-Balance)
- Forward Error Correction (FEC)
Data and Slow control Links
Timing and Trigger functions
Still in the early specification phase: must be designed
according to the experiments
needs!
High speed critical serial communications building blocks, functionality well defined
and “independent” of the overall architecture.
(Currently under development)
Optical Link - April 2008
Dominique Breton – Rome SuperB meeting – December 17th 2008

12. GBT Project Collaborators
Institute
ProjectMgmt.
System Design
ASIC Design
FPGA Design
Opto-Electronics
Passive Compnts.
Radn. Testing
CERN x
(x)
CPPM Marseille
INFN Bari
INFN Bologna
INFN Torino
Oxford
SMU Dallas
Optical Link - April 2008
Dominique Breton – Rome SuperB meeting – December 17th 2008

13. Dominique Breton – Rome SuperB meeting – December 17th 2008
GBT Project Schedule
2008
Design and prototyping of performance critical building blocks:
- TIA, laser driver, PLL, serializer, de-serializer, phase shifter
First tests of optoelectronics components
- (e.g. to understand SEU in PIN receivers)
Proceed with the link specification meetings
General link specification
2009
Design/prototype/test of basic serializer/de-serializer chip
Design/prototype/test of optoelectronics packaging
Detailed link specification document
2010
Prototype of “complete” link SERDES chip
Full prototype of optoelectronics packaging
2011
Extensive test and qualification of full link prototypes
System demonstrator(s) with use of full link
Schedule of the final production version is strongly dependent on the evolution of the LHC upgrade schedule
Optical Link - April 2008
Dominique Breton – Rome SuperB meeting – December 17th 2008

14. Dominique Breton – Rome SuperB meeting – December 17th 2008
Moving to SuperB (2)
The main new problem on SuperB compared to BABAR is the channel occupancy.
This leads to two different types of problems :
For detectors with slow signals (like EMC barrel), physics events may sit on the queue of large background events (pile-up).
Two consecutive physics events may reside within the trigger time window (overlapping).
The question is : can we afford keeping the fixed trigger time window in such a situation, especially if the inter-trigger minimum delay can go down to ~100ns ?
Should the window be systematically larger for the EMC barrel ?
This increases the mean amount of data per event.
This makes the second problem worse.
The window length could then be defined on a per event basis, thanks to a few bits sent with the L1 accept command (Model1).
For overlapping events, FEE should be able to deal with close triggers, and send data in consequence (like reducing the size of posterior events)
The idea of directly addressing data in the ring buffer could be raised (Model2) if:
transmission links or FEE are not able to deal with the shortest time interval between successive L1 accept
the amount of data per event is too large because of the « fixed » latency
Dominique Breton – Rome SuperB meeting – December 17th 2008

15. Pros & cons of both FEE models
In terms of buffering size in the FEE :
Dealing with overlapping events is free.
Choosing the window length is cheap : buffer size has to take extra required window length into account.
Using the Read_event command in addition to L1 accept is costly, especially if the requested extra buffering is large (linked to trigger rate).
Having an addressable ring buffer is expensive : it may almost double the buffer length requirement.
In terms of complexity in the FEE :
Dealing with overlapping events is cheap : one just has to measure the distance between L1 accepts and send the corresponding time slices. This could also be done by FCTS, sending a shortened time window depending on previous L1 accept.
Choosing the window length is cheap : it’s just waiting for the right position in the ring buffer.
Having an addressable ring buffer is more expensive.
In terms of length of L1 accept command (and thus of transmission time) :
Choosing the window length is cheap (2-4 bits should be sufficient).
Using Read_event has a high cost in terms of link occupancy (one per L1 accept).
Having an addressable ring buffer is expensive.
Dominique Breton – Rome SuperB meeting – December 17th 2008

16. Temporary conclusion about FEE models
So, concerning FEE buffer size, FEE complexity, and L1 occupancy on the ROM to FEE link, addressing the ring buffers (« Model 2 ») is more expensive
this should lead us to study extensively the fixed trigger latency model (« Model 1 »), in order to check if it copes or not with the requirements before choosing any model as a baseline
if the Read_Event command doesn’t really prove to be necessary, it should be removed
Dominique Breton – Rome SuperB meeting – December 17th 2008

17. Simulation of Model1 Sequence:
The DAQ sends a L1 trigger command associated with a value corresponding to a time window. The FEE sends to the DAQ the data contained inside this window, embedded in a frame including status, trigger tag and time.
Trigger is defined by three parameters :
- The latency : L (fixed)
- The window width : W (event basis)
- The time distance between triggers : D
Constraints :
- Triggers with variable width windows
- Triggers with overlapping windows
- No dead time …

18. Dominique Breton – Rome SuperB meeting – December 17th 2008
Simulation of Model1 (2)
Description t0
L1 Trigger
Data to keep
Data to dump L W0 N0
Time M0
Baseline: latency pipeline always provides the oldest relevant data
L: fixed latency
W0: window containing the relevant data for trigger #0
N0: data to dump
M0: data finally kept for the trigger 0
Dominique Breton – Rome SuperB meeting – December 17th 2008

19. Dominique Breton – Rome SuperB meeting – December 17th 2008
N : data to be dumped
L : Latency
W : Window
D : Distance
between triggers
M : data to be kept
Two different cases can be identified
Trigger #0
Trigger #1
Case 1 : D ≥ W0 D L
D ≥ L W0 N0 W1 N1 M0 M1
Non overlapping latencies with 2
different windows (green): no problem
M1 = W1
Trigger #0
Trigger #1 Or D
Overlapping latency with non
overlapping windows. Still straightforward.
M1 = W1
D < L W0 N0 M0 W1 N1 M1
Trigger #0
Trigger #1
Case 2 : D < W0 D
Overlapping latency trigger with
overlapping windows: trickier …
The window W1 is then shortened!
M1 = W1 – (W0 – D)= W1- W0 + D W0 N0 M0 W1 N1 M1
Dominique Breton – Rome SuperB meeting – December 17th 2008

20. Dominique Breton – Rome SuperB meeting – December 17th 2008
Simple hardware implementation
Case 1 : Dn ≥ Wn :
Mn = Wn
Case 2 : Dn < Wn :
Mn = Wn – Wn-1 + Dn
Mn : amount of data to be kept for the trigger #n
Trigger
input Dn
Counter
Dn ≥ Wn?
Fifo
“M” Wn
!empty M U X Wn M
FSM S U B A D Wn
end
Dn+Wn-Wn-1
Wn-Wn-1 D
Wn-1
enable M
Counter
All synchronous pipelined operations
Start_flag
Wr_en
To serializer
Data
input
Event
Fifo
EVT_BUFFER
Latency Pipeline L
Dominique Breton – Rome SuperB meeting – December 17th 2008

21. Dominique Breton – Rome SuperB meeting – December 17th 2008
Conclusion
DAQ and trigger are based on BABAR experience but must evolve to cope with the new level of the requirements
ROM and L1 trigger processors have to be redesigned
=> there is a real need for new collaborators to work on DAQ and L1 trigger hardware
Control distribution has to be rethought
=> CERN has started a generic R&D for detector control and readout.
They propose us to join them …
The key point is the production if Super-LHC is delayed …
=> Do we have another solution in hand or in mind ?
We started working on the fixed latency DAQ model and will soon present high level simulations of this simple solution.
A Verilog/VHDL model will also be built.
We hope to get feedback from you whether it is adequate for physics requirements or not.
- Any question ?
Dominique Breton – Rome SuperB meeting – December 17th 2008