Presented by Jubin MITRA CRU WORK SUMMARY 1st July, 2015 CRU WORKSHOP 1st July, 2015 Presented by Jubin MITRA

Team Members CPPM Jean-Pierre CACHEMICHE and the group Jubin MITRA Variable Energy Cyclotron Centre (VECC) Jubin MITRA Shuaib Ahmad KHAN Tapan Kumar NAYAK Wigner Research Centre for Physics Erno DAVID Tivadar KISS CERN Filippo COSTA Peter CHOCHULA Alex KLUGE University of Jammu Anik GUPTA Bose Institute Sanjoy MUKHERJEE With active help and support from: CPPM Jean-Pierre CACHEMICHE and the group University Of Calcutta Rourab PAUL Amlan CHAKRABARTI 1st July, 2015 Presented by Jubin MITRA

Presented by Jubin MITRA TABLE OF CONTENTS What are we thinking of CRU Interfaces? CRU Functional Block Diagram What we have learned about PCIe40 ? Participation in PCIe40 board testing and debugging Firmware development status Questions for Development Team 1st July, 2015 Presented by Jubin MITRA

WHAT ARE WE THINKING OF CRU INTERFACES? CRU has three interfaces. It is marked as 1,2 and 3 respectively. 1 – GBT Link (radiation tolerant High speed Optical Link) 2 – To be decided (GBT/TTS/PCIe/10Gigabit PON) 3 – DDL3 link ( PCIe Gen 3 x16) 1st July, 2015 Presented by Jubin MITRA

CRU FUNCTIONAL BLOCK DIAGRAM As can be seen in the functional diagram, the problem is broadly being classified into 5 super modules, which can then be divided into multiple sub-modules. It also includes a supervising module, to act as a debugging tool for design engineers. The CRU firmware specific to each detector will use different percentage of these modules. Of the 5 functional modules only CIU module is visible to the CRU interfaces, rest of the 4 modules are internal functional ones and it will be like a black box to CRU users. Development of CIU is our current priority 1st July, 2015 Presented by Jubin MITRA

A glimpse of GBT Test Setup at VECC on Stratix V board 1st July, 2015 Presented by Jubin MITRA

First prototype of the PCIe40 made by CPPM in Marseille 1st July, 2015 Presented by Jubin MITRA

Features of PCIe40 V1 board FPGA used Arria 10 (10AX115S4F45I3SGES) MAX V for Flash programming 1 PLX 8747 chip for converting two x8 PCIE into one x16 PCIe 2 Jitter cleaner: Si5338, CDCE62005 10 Gig PLL Clock: SI5315 4 Tx Minipod and 4 Rx Minipod – Each supporting 12 links = 48 High Speed links available for data acquisition or distribution of timing, fast and slow control to front-ends 1 SFP+ or PON device for Timing and Trigger reception With Arria10 Altera supports vertical migration to Stratix 10 same density device 1st July, 2015 Presented by Jubin MITRA

What we gain from Arria 10 with respect to Stratix V FPGA ? 5SGXEA7N2F45C3 10AX115S4F45I3SGES Core voltage 0.85V 0.95V (For ES) else 0.9V ALMs 234720 427200 Total I/Os 1064 960 GPIOs 840 624 GXB Channel PMA and PCS/ HSSI channels 48 72 PCIe Hard IP Blocks 4 Memory Bits 52428800 55562240 DSP Blocks 256 1518 27 x 27 Multiplier Fractional PLL 28 32 DLLs - I/O PLLs 16 Global Clocks HPS CPU Core 1st July, 2015 Presented by Jubin MITRA

Participation in PCIe40 testing and debugging We were largely involved in the debug of serial links. Most issues were corrected THE RESULT IS THAT LINKS ARE ALL UP AND RUNNING We still have some improvements to do (calibrate the channels after power up, or program few PLLs with LHC frequency). Showing one of the many tests that have been conducted Test setup : Transmit 5Gbps data from PCIe40 to AMC40 and vice verse. With pattern checker and generator on either side to check uplink and downlink data communication 1st July, 2015 Presented by Jubin MITRA

Participation in PCIe40 testing and debugging Result: We are able to transmit data from PCIe40 to AMC40, and vice verse Eye Diagram of PCIe Tx side showing wide opening Table showing a comparison the eye height and width for AMC40 Tx and PCIe40 Tx Important point to note here: There is a need of polarity inversion in the Tx side of the PCIe40 transmission. 1st July, 2015 Presented by Jubin MITRA

Participation in PCIe40 testing and debugging Showing BER of data received from AMC40 to PCIe40 Showing eye diagram in EYEQ of data received in AMC40 from PCIe40 at 5Gbps 1st July, 2015 Presented by Jubin MITRA

PCIe40 firmware development status GBT Design migration from Stratix V to Arria 10 Complete Simulation of the firmware Implemented it in PCIe40 Using presently available jitter free clock of PCIe (100MHz) to test 4Gbps data rate Latency Measurement Making of GBT QSYS wrapper TO DO: Run using 120 MHz clock, with required 4.8Gbps Do board to board testing of GBT protocol between Stratix V and Arria 10 Do test with GBT chip PCIe Gen2 x8 +Gen2 x8 test for16 lane connectivity Basic read write test on BAR 12 channel Transceiver tool kit design is made to test board to board communication between Stratix V and Arria 10 1st July, 2015 Presented by Jubin MITRA

Presented by Jubin MITRA GBT : Design migration of version 3.1.1 from Stratix V to Arria 10 Arria 10 Stratix V The coding is modified keeping it in standard with the other GBT FPGA core. So, it is easier to maintain and upgrade in future. Good news: No Tx latency optimization problem in Arria10 GBT design 1st July, 2015 Presented by Jubin MITRA

Presented by Jubin MITRA GBT : Complete Simulation of the firmware in modelsim Direct launch of simulation N.B. : “tx clkout” and “rx clkout” initially generates lower frequency and after some delay generates the required frequency. So, care should be taken when driving PLL from this signal. 1st July, 2015 Presented by Jubin MITRA

Presented by Jubin MITRA GBT : Showing the In System Sources and Probes screenshot 1st July, 2015 Presented by Jubin MITRA

Presented by Jubin MITRA GBT : Showing Signal Tap II Screenshot 1st July, 2015 Presented by Jubin MITRA

Presented by Jubin MITRA GBT : Showing the signal quality of GBT operating at 4Gbps using 100 MHz PCIe clk Test Setup The bumps appearing on the left of the eye diagram are due to the de-emphasis of the minipod receiver. This is not an unwanted distortion of the signal. We can disable the de-emphasis at 4.8 Gigs because this feature is useful only at higher speeds (10 Gbits/s). 1st July, 2015 Presented by Jubin MITRA

Presented by Jubin MITRA GBT : Showing the test setup for Latency Measurement ISSP 1st July, 2015 Presented by Jubin MITRA

GBT : QSYS Wrapper for fast system integration 1st July, 2015 Presented by Jubin MITRA

GBT : QSYS Wrapper in use 1st July, 2015 Presented by Jubin MITRA

Presented by Jubin MITRA PCIe Testing Testing of Example Design for Arria 10 with Avalon-MM DMA interface (ep_g2x8_avmm) with two modules of PCIe Gen2 x8 lanes. Successfully detects 2 boards PCIe Gen 2 x8. N.B.: Use 2 separate global ref clocks for two modules PCI express example design Gen3 x8 lane in group of two is detected in PCIe40 board, but on reading the device information in Linux we found that it is detected as GEN1 (this is the issue with ES 1 of Arria10 FPGA, where Gen3 is falling back to Gen1). So the design is compiled again with GEN2 x8 in group of two which is detected in Arria10 and Read/write test on BAR is done 1st July, 2015 Presented by Jubin MITRA

Screenshot of PCIe detection in mindshare software PLX 8747 Switch PCIe gen2 x8 from Arria 10 FPGA PCIe gen2 x8 from Arria 10 FPGA 1st July, 2015 Presented by Jubin MITRA

PCIe40 Tx signal as received in AMC40 12 channel Transceiver tool kit design for board to board communication between Stratix V and Arria 10 at 10.312Gbps PCIe40 Tx side signal PCIe40 Tx signal as received in AMC40 1st July, 2015 Presented by Jubin MITRA

Questions for Development Team A realistic estimation of final firmware occupancy, speed and toggle rate necessary for optimizing the power supply tree of the final production card. Specification of environmental conditions (air flow, ambient temperature) Specification of mechanical space available Next slide shows a graphical representation of this issue N.B.: In Arria 10 Clock Calibration must be done before the firmware in loaded 1st July, 2015 Presented by Jubin MITRA

More Logic Occupancy and Toggle rate PCIe40 Power ICs More Logic Occupancy and Toggle rate More Power Required More Power IC required More Heat Generated More Cooling Need More Mechanical Space There is a trade off It would be fine to get as soon as possible a realistic firmware in order to simulate the FPGA power consumption. 1st July, 2015 Presented by Jubin MITRA

Presented by Jubin MITRA THANKS TO CPPM Team (Marseille) 1st July, 2015 Presented by Jubin MITRA