1. M. Bellato INFN Padova and U. Marconi INFN Bologna
Status of the ROM
M. Bellato INFN Padova
and
U. Marconi INFN Bologna
SuperB meeting - Frascati 4/7 April 2011

2. SuperB meeting - Frascati 4/7 April 2011
Parameters.
The estimated event size at the FE is 500 kB.
The maximum trigger rate is 150 kHz.
We set the aggregate ROM input rate to 10 Gb/s.
Number of FEE 2Gb/s data links : 325
With these assumptions the number of ROMs needed to manage the SuperB data flux can be estimated to be of the order of: (8. × 500. ×103 ) × (150. × 103)/(10. × 109) = 60. boards
Each ROM will handle ∼10 kB of fragment size at 150 kHz.
SuperB meeting - Frascati 4/7 April 2011

3. 18 June 2009

4. SuperB meeting - Frascati 4/7 April 2011
ROM implementation.
We foresee two possible approaches to the implementation of the ROM.
Based on custom electronics and field busses: using FPGA to get data from the FEE, perform “simple” synchronous processing, and format data according to a suitable industrial standard.
Based on custom electronics and host PC/CPUs: FPGA being used to get data from the FEE (possibly perform synch. Processing) and inject them into the PC via PCIe. CPUs will perform complex data processing and data transfer, using standard protocol and on board network interface cards.
SuperB meeting - Frascati 4/7 April 2011

5. SuperB meeting - Frascati 4/7 April 2011
Implementation (I)
Based on a field bus (VME, xTCA, …)
5 x FEE optical links at 2Gb/s as input
1 x 10Gb/s as output
FCTS interface is not an issue
FPGA centric design
Inputs deserialization
Synchronous processing (feature extraction)
Fragment formatting according to event builder protocol (TCP/UDP, Converged Ethernet, custom)
Output serialization
Once design is fixed, changes are difficult
Improvements are limited, if not foreseen in advance
SuperB meeting - Frascati 4/7 April 2011

6. Implementation (I) – Block Diagram
SuperB meeting - Frascati 4/7 April 2011

7. SuperB meeting - Frascati 4/7 April 2011
Implementation (II)
Solution based on FPGA and PC/CPUs appears to be more versatile than the one based on stand alone custom board.
The required functionality implemented by means of high level languages.
Transmission protocol to the HLT farm is no more a constraint.
The trigger rate of 150 kHz constraints the available processing time per CPU to NCPU / 150. ms NCPU being the number of CPU cores per PC box.
A processing time of the order of 1 ms per CPU would require therefore order of 100 CPU per box.
Availability of many-cores CPUs …………? see next slides.
SuperB meeting - Frascati 4/7 April 2011

8. Implementation(II) – Block Diagram
Power Management
SFP RTX
FCTS I/F
SNAP12 RX
Front-Ends
FPGA – Virtex 6/7
PCIexpress 4x
Dual port RAM
Clock Management
SuperB meeting - Frascati 4/7 April 2011

9. + How to get High Performance and Energy Efficiency
for highly Parallel Workloads?
high F.P. performance (VPU/SIMD) +
many integrated small energy efficient and
high-performance cores
small extreme energy efficient core
The Newest Addition to the Intel Server Family.
Industry’s First General Purpose Many Core Architecture

10. Knights Ferry - Aubrey Isle Processor
Memory Controller
System & I/O
PCIe Interface
Function
Fixed
Multi-Threaded
Wide SIMD I$ D$
. . .
Shared coherent L2 Cache
GDDR
GDDR . .
GDDR
GDDR
Multiple IA cores 16-wide vector units (512b) bit ring bus
- In-order, short pipeline - Extended instruction set GDDR memory
- Multi-thread support Fully coherent caches - Supports virtual memory
Standard IA Shared Memory Programming
Future options subject to change without notice.

11. >50 Intel Architecture cores
The “Knights” Family
Future Knights
Products
Knights Corner
1st Intel® MIC product
22nm process
>50 Intel Architecture cores
Knights Ferry
Development Platform
Future options subject to change without notice.

12. “Knights Ferry” Development Platform
Software Development Platform
Growing availability through 2011
Up to 32 cores, up to 1.2 GHz
Up to 128 threads at 4 threads / core
Up to 8MB shared coherent cache
1-2 GB GDDR5 shared memory
PCIe Card
Bundled with Intel HPC SW tools
Software development platform for Intel® MIC architecture

13. SuperB meeting - Frascati 4/7 April 2011
ROM PCIe Architecture
The interface board (FEE to PCIe) plugged into the PC mother board.
PCIe Gen2 x4 lanes has enough bandwidth: (4.×5.×109 bit/s) (150 kHz)-1 = 16.7 KB ( > fragment size ~ 10 KB)
Adequate cooling of the interface board may be an issue (but estimated power consumption < 40 W).
Mechanical fit is poor
FCTS interface should be integrated in the ROM ?
SuperB meeting - Frascati 4/7 April 2011

14. ROM PCIe Architecture (II)
Event builder farm may fit the ROMs as plug-in cards
Exploit existing hardware and reduce costs
Event builder farm may fit also many-cores CPU cards as add-on
HLT may be performed on the same farm
+ filter farm
SuperB meeting - Frascati 4/7 April 2011

15. SuperB meeting - Frascati 4/7 April 2011
Ongoing Tests
Proof of concept performed on Xilinx ML605 with Virtex-6 LX240T and PCIe Gen2 4x
SuperB meeting - Frascati 4/7 April 2011

16. SuperB meeting - Frascati 4/7 April 2011
Ongoing tests
SuperB meeting - Frascati 4/7 April 2011

17. SuperB meeting - Frascati 4/7 April 2011
Plans
Legacy implementation based on field busses requires
investigation on the 10Gb/s output links in terms of protocol implementations on FPGA
PCIe based implentation requires
Development of a Linux driver to test the performance.
Evaluate process latency of getting and transmitting data: useful time for processing.
Evaluate Knights Ferry or GP-GPU for feature extraction ?
SuperB meeting - Frascati 4/7 April 2011