1. Presented by Jubin MITRA
CRU WORK SUMMARY
1st July, 2015
CRU WORKSHOP
1st July, 2015
Presented by Jubin MITRA

2. Team Members CPPM Jean-Pierre CACHEMICHE and the group Jubin MITRA
Variable Energy Cyclotron Centre (VECC)
Jubin MITRA
Shuaib Ahmad KHAN
Tapan Kumar NAYAK
Wigner Research Centre
for Physics
Erno DAVID
Tivadar KISS
CERN
Filippo COSTA
Peter CHOCHULA
Alex KLUGE
University of Jammu
Anik GUPTA
Bose Institute
Sanjoy MUKHERJEE
With active help and support from:
CPPM
Jean-Pierre CACHEMICHE
and the group
University Of Calcutta
Rourab PAUL
Amlan CHAKRABARTI
1st July, 2015
Presented by Jubin MITRA

3. Presented by Jubin MITRA
TABLE OF CONTENTS
What are we thinking of CRU Interfaces?
CRU Functional Block Diagram
What we have learned about PCIe40 ?
Participation in PCIe40 board testing and debugging
Firmware development status
Questions for Development Team
1st July, 2015
Presented by Jubin MITRA

4. WHAT ARE WE THINKING OF CRU INTERFACES?
CRU has three interfaces. It is marked as 1,2 and 3 respectively.
1 – GBT Link (radiation tolerant High speed Optical Link)
2 – To be decided (GBT/TTS/PCIe/10Gigabit PON)
3 – DDL3 link ( PCIe Gen 3 x16)
1st July, 2015
Presented by Jubin MITRA

5. CRU FUNCTIONAL BLOCK DIAGRAM
As can be seen in the functional diagram, the problem is broadly being classified into 5 super modules, which can then be divided into multiple sub-modules. It also includes a supervising module, to act as a debugging tool for design engineers. The CRU firmware specific to each detector will use different percentage of these modules.
Of the 5 functional modules only CIU module is visible to the CRU interfaces, rest of the 4 modules are internal functional ones and it will be like a black box to CRU users.
Development of CIU is our current priority
1st July, 2015
Presented by Jubin MITRA

6. A glimpse of GBT Test Setup at VECC on Stratix V board
1st July, 2015
Presented by Jubin MITRA

7. First prototype of the PCIe40 made by CPPM in Marseille
1st July, 2015
Presented by Jubin MITRA

8. Features of PCIe40 V1 board
FPGA used Arria 10 (10AX115S4F45I3SGES)
MAX V for Flash programming
1 PLX 8747 chip for converting two x8 PCIE into one x16 PCIe
2 Jitter cleaner: Si5338, CDCE62005
10 Gig PLL Clock: SI5315
4 Tx Minipod and 4 Rx Minipod – Each supporting 12 links
= 48 High Speed links available for data acquisition or distribution of timing, fast and slow control to front-ends
1 SFP+ or PON device for Timing and Trigger reception
With Arria10 Altera supports vertical migration to Stratix 10 same density device
1st July, 2015
Presented by Jubin MITRA

9. What we gain from Arria 10 with respect to Stratix V FPGA ?
5SGXEA7N2F45C3
10AX115S4F45I3SGES
Core voltage
0.85V
0.95V (For ES) else 0.9V
ALMs
234720
427200
Total I/Os
1064
960
GPIOs
840
624
GXB Channel
PMA and PCS/ HSSI channels 48 72
PCIe Hard IP Blocks 4
Memory Bits
DSP Blocks
256
1518
27 x 27 Multiplier
Fractional PLL 28 32
DLLs -
I/O PLLs 16
Global Clocks
HPS CPU Core
1st July, 2015
Presented by Jubin MITRA

10. Participation in PCIe40 testing and debugging
We were largely involved in the debug of serial links.
Most issues were corrected
THE RESULT IS THAT LINKS ARE ALL UP AND RUNNING
We still have some improvements to do (calibrate the channels after power up, or program few PLLs with LHC frequency).
Showing one of the many tests that have been conducted
Test setup : Transmit 5Gbps data from PCIe40 to AMC40 and vice verse. With pattern checker and generator on either side to check uplink and downlink data communication
1st July, 2015
Presented by Jubin MITRA

11. Participation in PCIe40 testing and debugging
Result:
We are able to transmit data from PCIe40 to AMC40, and vice verse
Eye Diagram of PCIe Tx side showing wide opening
Table showing a comparison the eye height and width for AMC40 Tx and PCIe40 Tx
Important point to note here:
There is a need of polarity inversion in the Tx side of the PCIe40 transmission.
1st July, 2015
Presented by Jubin MITRA

12. Participation in PCIe40 testing and debugging
Showing BER of data received from AMC40 to PCIe40
Showing eye diagram in EYEQ of data received in AMC40 from PCIe40 at 5Gbps
1st July, 2015
Presented by Jubin MITRA

13. PCIe40 firmware development status
GBT
Design migration from Stratix V to Arria 10
Complete Simulation of the firmware
Implemented it in PCIe40
Using presently available jitter free clock of PCIe (100MHz) to test 4Gbps data rate
Latency Measurement
Making of GBT QSYS wrapper
TO DO:
Run using 120 MHz clock, with required 4.8Gbps
Do board to board testing of GBT protocol between Stratix V and Arria 10
Do test with GBT chip
PCIe
Gen2 x8 +Gen2 x8 test for16 lane connectivity
Basic read write test on BAR
12 channel Transceiver tool kit design is made to test board to board communication between Stratix V and Arria 10
1st July, 2015
Presented by Jubin MITRA

14. Presented by Jubin MITRA
GBT : Design migration of version from Stratix V to Arria 10
Arria 10
Stratix V
The coding is modified keeping it in standard with the other GBT FPGA core.
So, it is easier to maintain and upgrade in future.
Good news: No Tx latency optimization problem in Arria10 GBT design
1st July, 2015
Presented by Jubin MITRA

15. Presented by Jubin MITRA
GBT : Complete Simulation of the firmware in modelsim
Direct launch of simulation
N.B. : “tx clkout” and “rx clkout” initially generates lower frequency and after some delay generates the required frequency. So, care should be taken when driving PLL from this signal.
1st July, 2015
Presented by Jubin MITRA

16. Presented by Jubin MITRA
GBT : Showing the In System Sources and Probes screenshot
1st July, 2015
Presented by Jubin MITRA

17. Presented by Jubin MITRA
GBT : Showing Signal Tap II Screenshot
1st July, 2015
Presented by Jubin MITRA

18. Presented by Jubin MITRA
GBT : Showing the signal quality of GBT operating at 4Gbps using 100 MHz PCIe clk
Test Setup
The bumps appearing on the left of the eye diagram are due to the de-emphasis of the minipod receiver. This is not an unwanted distortion of the signal. We can disable the de-emphasis at 4.8 Gigs because this feature is useful only at higher speeds (10 Gbits/s).
1st July, 2015
Presented by Jubin MITRA

19. Presented by Jubin MITRA
GBT : Showing the test setup for Latency Measurement
ISSP
1st July, 2015
Presented by Jubin MITRA

20. GBT : QSYS Wrapper for fast system integration
1st July, 2015
Presented by Jubin MITRA

21. GBT : QSYS Wrapper in use
1st July, 2015
Presented by Jubin MITRA

22. Presented by Jubin MITRA
PCIe Testing
Testing of Example Design for Arria 10 with Avalon-MM DMA
interface (ep_g2x8_avmm) with two modules of PCIe Gen2 x8 lanes.
Successfully detects 2 boards PCIe Gen 2 x8.
N.B.: Use 2 separate global ref clocks for two modules
PCI express example design Gen3 x8 lane in group of two is detected in PCIe40 board, but on reading the device information in Linux we found that it is detected as GEN1 (this is the issue with ES 1 of Arria10 FPGA, where Gen3 is falling back to Gen1). So the design is compiled again with GEN2 x8 in group of two which is detected in Arria10 and Read/write test on BAR is done
1st July, 2015
Presented by Jubin MITRA

23. Screenshot of PCIe detection in mindshare software
PLX 8747 Switch
PCIe gen2 x8
from Arria 10 FPGA
PCIe gen2 x8 from Arria 10 FPGA
1st July, 2015
Presented by Jubin MITRA

24. PCIe40 Tx signal as received in AMC40
12 channel Transceiver tool kit design for board to board communication between Stratix V and Arria 10 at Gbps
PCIe40 Tx side signal
PCIe40 Tx signal as received in AMC40
1st July, 2015
Presented by Jubin MITRA

25. Questions for Development Team
A realistic estimation of final firmware occupancy, speed and toggle rate necessary for optimizing the power supply tree of the final production card.
Specification of environmental conditions (air flow, ambient temperature)
Specification of mechanical space available
Next slide shows a graphical representation of this issue
N.B.: In Arria 10 Clock Calibration must be done before the firmware in loaded
1st July, 2015
Presented by Jubin MITRA

26. More Logic Occupancy and Toggle rate
PCIe40
Power ICs
More Logic Occupancy and Toggle rate
More Power
Required
More Power IC required
More Heat
Generated
More Cooling
Need
More
Mechanical Space
There is a trade off
It would be fine to get as soon as possible a realistic firmware in order to simulate the FPGA power consumption.
1st July, 2015
Presented by Jubin MITRA

27. Presented by Jubin MITRA
THANKS TO CPPM Team
(Marseille)
1st July, 2015
Presented by Jubin MITRA