1. 2008-07-4LIU,Zhen'An, TriggerGroup,IHEP1  I would like to thank Prof. Y. Sakai and Dr. Y. Iwasaki for their kind help in BESIII trigger design, and I am happy that we have this chance to share our experience of BESIII in KEKB Trigger and DAQ design.

2. High Speed Signal/Data Transmission in BESIII Trigger and PANDA TDAQ Systems Zhen’An LIU Trigger Group, IHEP Beijing 2nd Open meeting for the KEKB proto-collaboration July 3-4th 2008

3. 2008-07-4LIU,Zhen'An, TriggerGroup,IHEP3 Outline  Background infor for BESIII trigger  Study of High speed signal transmission in BESIII trigger  BESIII trigger  PANDA TDAQ  Computer Node  Proof of Concept Application: HADES DAQ Upgrade  Comments and conclusion

4. 2008-07-4LIU,Zhen'An, TriggerGroup,IHEP4 Key points in BESIII trigger Design  Optical isolation with FEE, to prevent from ground loop current interference  Most latest FPGAs, Boards with simplicity, high reliability  FPGA online downloadable via VME  Generalized hardware, firmware for different function, for easy maintenance  Scheme optimization with simulation

5. 2008-07-4LIU,Zhen'An, TriggerGroup,IHEP5 Difficulties for MDC tracking Difficulties Bad number of wires for both axial and stero layers (trigger point of view, Hard to define Sector/board border for signal input) Hard to input signals via 9U front and rare pannels Too many sharing signals for neighbor boards Solution: RocketIO for input signals (32bits/ch, 8b/10b) private VME J3 Backplane for sharing signals 1234 SL 140444856/16 SL 2647280 /16 SL 376 88 /8 SL 4100 112 /16 SL 5128 /16 SL10128/256 /16

6. 2008-07-4LIU,Zhen'An, TriggerGroup,IHEP6 Study of High speed signal transmission in BESIII trigger

7. 2008-07-4LIU,Zhen'An, TriggerGroup,IHEP7 MFT (MDC Fiber Transmitter)  2796 hits signals from MDC QT boards are collected in MFT, 32 channels per MFT  Virtex-II Pro FPGA: XC2VP2  Fuctions: Stretching to 500ns Synchronization + alignment signals(private Protocol) Serialization(8b/10b)

8. 2008-07-4LIU,Zhen'An, TriggerGroup,IHEP8 TKF (TracK Finder) XC2VP40 : FF1152, 804 user IOs, 43,632 logic cells, 3,456Kbit BRAM, 12 RocketIOs, 2 PowerPCs, 192 multiplier blocks 10 layers 9UVME PCB XC2VP40HFBR5921L Functions: Deserialization Channel alignment TSF TF Track information

9. 2008-07-4LIU,Zhen'An, TriggerGroup,IHEP9 Clock correction  K codes COMMA IDLE  RXRECCLK  RXUSRCLK Bytes Allowed Between Clock Correction Sequences REFCLK Stability Remove1IDLE Sequences:Remove2IDLE Sequences:Remove3IDLE Sequences:Remove4IDLE Sequences: 100 ppm 5,000 （ √ ） 10,00015,00020,000 50 ppm10,00020,00030,00040,000 20 ppm25,00050,00075,000100,000

10. 2008-07-4LIU,Zhen'An, TriggerGroup,IHEP10 Data alignment  Continuous  Parallel bits transmitted in same clock => recovered also in same Clock  Special private protocol

11. 2008-07-4LIU,Zhen'An, TriggerGroup,IHEP11 Clocks  Crystal Must follow the recommendation  Use built-in DCM  Clocks REFCLK USRCLK USRCLK2

12. 2008-07-4LIU,Zhen'An, TriggerGroup,IHEP12 Example of RocketIO Instantiation  RocketIO_1 : GT_CUSTOM  generic map( CRC_END_OF_PKT => "K29_7",  CRC_FORMAT => "USER_MODE",  CRC_START_OF_PKT => "K27_7",  TX_CRC_USE => TRUE,  TX_DATA_WIDTH => 4,  …);  port map (REFCLK=>REFCLK_IN,  TXUSRCLK=>TXUSRCLK_IN,  TXUSRCLK2=>TXUSRCLK2_IN,  TXCHARISK(3 downto 0)=>TXCHARISK_IN(3 downto 0),  TXDATA(31 downto 0)=>TXDATA_IN(31 downto 0),  TXFORCECRCERR=>TXFORCECRC_IN,  TXN=>TXN_OUT,  TXP=>TXP_OUT,  …);

13. 2008-07-4LIU,Zhen'An, TriggerGroup,IHEP13 Transmission protocol  Transmitter side  Receiver side

14. 2008-07-4LIU,Zhen'An, TriggerGroup,IHEP14 Good reliability with RocketIO BER ＝ 1 3.0×10e13 × 40 errors periods × 40 <≈ 8.3×10 -16 Tx Data Rx data Rx random data # of errors Total periods TD+ TD- RD+ RD- HFBR5921L TD RD MGT TXP TXN RXP RXN TD+ TD- RD+ RD- HFBR5921L RD TD MGT TXP TXN RXP RXN MFT TKF TXDATA (31:0) PRBS Data Generate FPGA RXDATA (31:0) Data Aligned & Checker FPGA

15. 2008-07-4LIU,Zhen'An, TriggerGroup,IHEP15 MDC Sub-Trigger MTB MDC Fiber Transmitter MDCQ&T TrackFinder Inner- layer Track Finder Long Track Counter Short Track Counter 1.75Gb/s ×96 Optical Channels 80 ch 16 ch 2 3 4 128 MDC Trigger Crate MDC Electronic Crates ×8 TKF ×1 ITKF MDC Trigger Conditions 9 TO GTL ×1 LTKC ×1 STKC ×96 MFT ITKFTKF 108 1 1 1 8 1 96 # of boards 6 MTB STKC LTKC TKF ITKF MFT Board name 1MDC Trigger Backplane 1 124Total 1 1TracK Counter 8 1 1TracK Finder 11MDC Fiber Transmitter FPGA firmware Type of PCB

16. 2008-07-4LIU,Zhen'An, TriggerGroup,IHEP16 BESIII trigger Block diagram TCBA Global Trigger Logic 6.4  s TOF FEE MDC FEE EMC FEE MU FEE TOFPR MFT Mu track FastOR Track Counter Etotal Energy Hit/Seg Count TSF + TF BEPCII RF TTC TC Sum L1P 41.65 MHz Track Match Energy Balance Cluster Counting Fast Control FC Daughters Near DetectorsCounting Room 499.8 MHz High Lights: Optical Isolation-no ground loop current TKF

17. 2008-07-4LIU,Zhen'An, TriggerGroup,IHEP17 Installation Jumper Box MFTs in FEE crate

18. 2008-07-4LIU,Zhen'An, TriggerGroup,IHEP18 TOFPR

19. 2008-07-4LIU,Zhen'An, TriggerGroup,IHEP19 Installation TCBA: EMC preprocessor and transmitter

20. 2008-07-4LIU,Zhen'An, TriggerGroup,IHEP20 Installation cont’d TDC EMC/TOF/GTL Trigger MDC Trigger Optical Fibers Optical Cables CLK + FC Opt-Cable under

21. 2008-07-4LIU,Zhen'An, TriggerGroup,IHEP21 Status of BESIII trigger  Successful in Cosmic-ray test  Commissioning with BEPCII

22. 2008-07-4LIU,Zhen'An, TriggerGroup,IHEP22 PANDA: DAQ Requirements  Interactions: 10**7 Hz  Data: 200GB/s  Continuously Sampling ADC  No hardware trigger  Hi Speed Devices  Large Buffers  Large Bandwidth

23. 2008-07-4LIU,Zhen'An, TriggerGroup,IHEP23 PANDA DAQ & Trigger Detector’s Front-end Concentrator Board Switch L1 farm L1outL2outL3out Switch L2 farmL3 farm  2 Alternative DAQ Concepts Still Under Discussion

24. 2008-07-4LIU,Zhen'An, TriggerGroup,IHEP24 The Compute Node(CN)  5 Virtex4 FX60 FPGA Large Computer Power  10 GB DDR2 RAM (2GB per FPGA) Buffering capabilities  2 Embedded PowerPC in each FPGA  Slow control  32Gbit/s Bandwidth 13x RocketIO to backplane 5x Gbit Ethernet Front Pannel 1x Gbit Ethernet Backplane 8x Optical Links  ATCA Compliant Manageability

25. 2008-07-4LIU,Zhen'An, TriggerGroup,IHEP25 The Compute Node

26. 2008-07-4LIU,Zhen'An, TriggerGroup,IHEP26 Prototype of the CN by IHEP Beijing  Backplane  FPGA #O  FPGA #1-4  Front Panel Optical Links Ethernet Plugs

27. 2008-07-4LIU,Zhen'An, TriggerGroup,IHEP27 Proof of Concept Application: HADES DAQ Upgrade  Est.: 2009  Read Out – Trigger TRB Faster Readout  Trigger and Data Optical Links  Include Tracking in Trigger  12 Compute Node 1 Full ATCA shelf COMPUTE NODE  Online Tracking  RICH & TOF Matching with Tracking  Event Building on FPGA  Others Remote Upgrade IPMI

28. 2008-07-4LIU,Zhen'An, TriggerGroup,IHEP28 Motivation/Aim Produce a Configurable and Scalable Hardware Platform for Multiple Applications & Experiments Capable of High Performance Computing Large Bandwidth Real Time processing  Trigger Flexibility: Reusable  HADES - BESIII - PANDA Scalability  Flexible network topology

29. 2008-07-4LIU,Zhen'An, TriggerGroup,IHEP29 Comments and Conclusion  RocketIO can be used for HS transmission at KEKB  Awareness to KEKB TDAQ design: System clock Larger trigger latency  Latency by RocketIO (SEDES takes time )  Needs larger pipeline buffer in FEE Synch. + alignment  Uncertainty in recovered clock  Opt-cable length difference  Opt-transceiver difference

30. 2008-07-4LIU,Zhen'An, TriggerGroup,IHEP30