Possibilities for CPM firmware upgrade Juraj Bracinik, Richard J. Staley (University of Birmingham) L1Calo upgrade meeting, QMUL 3/2/2011 Introduction CPM Hardware Limits Possible data transfer schemes near future plans Medium term plans J. Bracinik, R.J.Staley UK L1Calo upgrade meeting 3/2/2011

Motivation for CPM upgrade Atlas trigger upgrade A long process Several stages (Phase 1, Phase 2, Phase 1.5 …) Time schedule (slowly) being defined, changing from time to time First part of upgrade Existing granularity of calorimeter signals the same definition of “trigger objects” (electrons, taus, jets, Et, Emiss) as in pre-upgrade L1Calo No change to CPM algorithms Main performance gain from adding topology Keep most of existing L1Calo HW Add CMM++ Or (hopefully) CMM++ + TP J. Bracinik, R.J.Staley UK L1Calo upgrade meeting 3/2/2011

UK L1Calo upgrade meeting 3/2/2011 I. CPMs in pre-upgrade L1Calo J. Bracinik, R.J.Staley UK L1Calo upgrade meeting 3/2/2011

CPM architecture – recapitulation I Most of work done in three types of FPGAs: 20 serializers- processing of input data streams, data distribution on-board and across backplane 8 CP chips – look for e/tau objects (RoIs) 8 EM and 8 EM/TAU thresholds 2 hit mergers – calculate total number of objects found by CP chips J. Bracinik, R.J.Staley UK L1Calo upgrade meeting 3/2/2011

CPM architecture – recapitulation II CP chips: Calculate energies in cluster, isolation and had veto energies Compare with pre-loaded thresholds There can be up to 2 objects (RoIs) on each chip (in left/right half of the chip) Information about these objects (which thresholds were passed) are sent on 2x16 data lines to 2 Hit mergers One Hit merger looks at EM thresholds, the other one at TAU/EM Hit mergers: Calculate total number of objects passing each threshold (saturate at 7) J. Bracinik, R.J.Staley UK L1Calo upgrade meeting 3/2/2011

II. limits of existing CPM hardware J. Bracinik, R.J.Staley UK L1Calo upgrade meeting 3/2/2011

Factors influencing CPM upgrade Speed of backplane transfer from CPM(s) to CMM(s) running at 40 MHz Bottle-neck in pre-upgrade L1Calo Transfer speed between CP chips and hit merger chips Running at 40 MHz Size of CP and Hit FPGAs J. Bracinik, R.J.Staley UK L1Calo upgrade meeting 3/2/2011

UK L1Calo upgrade meeting 3/2/2011 Backplane transfer I Tried to increase transfer frequency to 160 MHz Tested with pseudo-random patterns and by test patterns from serializer playback Built simple test board that allows to probe backplane signals in CMM slot With termination on both sides (CMM and CPM) the signal quality is very good! J. Bracinik, R.J.Staley UK L1Calo upgrade meeting 3/2/2011

Backplane transfer II Possible to run backplane at 160 MHz! Termination on CMM side improves signal quality, but increases dissipated power in CMM slot – can be a problem Tested signals with termination on source only Still very good Should work with termination on CPM only! Possible to run backplane at 160 MHz! J. Bracinik, R.J.Staley UK L1Calo upgrade meeting 3/2/2011

Transfer from CP chips to Hit mergers (on board) I Data lines between CP chips and Hit Mergers run currently at 40 MHz J. Bracinik, R.J.Staley UK L1Calo upgrade meeting 3/2/2011

Transfer from CP chips to Hit mergers (on board) II Different length of lines on the board limit possible rate J. Bracinik, R.J.Staley UK L1Calo upgrade meeting 3/2/2011

Transfer from CP chips to Hit mergers III Tested with dedicated firmware for CP chip and Hit merger CP chip sending parity protected data at 80 MHz Hit merger checking parity of received data Scan of parity errors as a function of phase difference between Deskew1 clock (used by Hit merger) and Deskew2 clock (used by CP chip) Parity error zone width dominated by spread of individual chips Possible to find error-free zone of reasonable width Possible to run these lines at 80 MHz! J. Bracinik, R.J.Staley UK L1Calo upgrade meeting 3/2/2011

CP and HIT FPGAs – available resources CP chip is rather full, but it is possible to add more functionality in Hit Merger (smaller FPGA)! J. Bracinik, R.J.Staley UK L1Calo upgrade meeting 3/2/2011

III. Possible CPM to CMM++ transfer schemes J. Bracinik, R.J.Staley UK L1Calo upgrade meeting 3/2/2011

Summary of available bandwidth Backplane at 160 MHz – 4x current data rate Each Hit FPGA to CMM => 96 bits/BC CP chips to Hit mergers at 80 MHz – 2x current rate Each CP chip to 1 HM => 32 bits/BC Some additional processig/routing possible in Hit FPGA J. Bracinik, R.J.Staley UK L1Calo upgrade meeting 3/2/2011

How to use available bandwidth? Backplane can have 25 lines run at 160 MHz Some information about RoIs that were found Parity bit Clock forwarding Possible to use one line for both parity bit and clock forwarding (line 24, used for parity bit by CMM) Rising edge gives clock phase information Parity bit corresponding to data in 4 160MHz ticks encoded in position of falling edge Leaves 24 lines for data (lines 0-23) J. Bracinik, R.J.Staley UK L1Calo upgrade meeting 3/2/2011

What to send – 10 RoI scheme First send map of locations where RoI's were found (16 bits, because there can be up to 2 RoIs in each of 8 CP chips) Then send hit bits (which trehslods were passed) for first 10 RoIs (8 bits/RoI) Remaining RoI simply discard (simulation shows it is very rare) Many advantages: Simple (involves only change to Hit Merger) Can do a lot of nice things (topology, overlap removal) Limited information about each RoI (8 bits) J. Bracinik, R.J.Staley UK L1Calo upgrade meeting 3/2/2011

10 RoI scheme – latency and resource usage Richard has implemented 10 RoI scheme in FW Checked with serializer playback feeding CP chips Looking at signals with scope Looks fine... Latency: Gate-level simulations :38 ns (1.5 Bcs) Standard hit-merging design: 30 ns Some latency penalty, but not big J. Bracinik, R.J.Staley UK L1Calo upgrade meeting 3/2/2011

10 RoI scheme – latency and resource usage Use of resources: Design fits into the FPGA Uses major fraction of resources This is about how complex we can go in Hit FPGAs A bit of resources still available.. J. Bracinik, R.J.Staley UK L1Calo upgrade meeting 3/2/2011

6+2 scheme (6RoIs and 2 highest ET ones) Simulations indicate most events have maximum of ~6 RoIs/CPM 10 RoI scheme doesn't allow for detailed energy and position information to be sent “Improved” 10 RoI scheme (6+2 RoI scheme): Send 16 bit RoI location map and threshold bits for 6 RoIs (8bits each) And detailed information about 2 highest ET RoIs: Transverse energy (ET) – 8 bits Detailed location – 5 bits Isolation – 3 bits EM information into one CMM++, HAD into the other one Our prefered scheme (for the moment): it gives us some backward compatibility allows for more energy thresholds (low ET thresholds in CPM, high in CMM++ or TP Use topology further downstream Seems to be possible, work (latency, resources …) in progress... J. Bracinik, R.J.Staley UK L1Calo upgrade meeting 3/2/2011

UK L1Calo upgrade meeting 3/2/2011 Energies only scheme In this scheme (almost) no thresholding in CPM All localized energy deposits (clusters) passing (low) threshold are transferred to CMM++ One CMM receives: 16 bin RoI map Detailed information about 6 RoIs: ET 8 bits Fine location 2 bits Isolation – 3 bits One CMM can receive EM information, the other one HAD All thresholding and topological processing done in CMM++ and Topological Processor J. Bracinik, R.J.Staley UK L1Calo upgrade meeting 3/2/2011

How to pick the transfer scheme ? Not sure if any of these schemes makes much sense :-( A lot of interesting studies needed: How many RoI do we really need Is energy sorting in Hit FPGA important (reasonable) thing to do? If yes, is one (two) highest ET deposits in CPM enough Do we need fine RoI location? All 8 energy bits? Would be nice to have a list of benchmark channels we would like to trigger on with these schemes... Are we looking for one scheme or rather thinking about gradual upgrade in several steps? Maybe start with 10 RoI scheme Next step 6+2 RoI Then energy only Maybe even start with existing CMMs and 80MHz backplane? J. Bracinik, R.J.Staley UK L1Calo upgrade meeting 3/2/2011

Future plans – CPM FW upgrade Continue playing with CPM FW, checking possibilities Several things haven't been touched yet: Proper testing of existing pieces of FW Readout Details of CP chip FW Need to have a look at how to send energies out Plan to look at these things with priority depending on CMM++ time scale and amount of new exciting ideas (either from simulation or from data-taking) J. Bracinik, R.J.Staley UK L1Calo upgrade meeting 3/2/2011

Future plans – middle term I Traditional areas of interest at Bham: High-speed PCB design Optical links FW development Online SW Physics studies and simulation It is not clear what type of HW we (UK L1Calo) are going to build and when Maybe we will know this afternoon :-) We know it won't be VME, probably ATCA A lot of new technologies and tools to learn J. Bracinik, R.J.Staley UK L1Calo upgrade meeting 3/2/2011

Future plans – middle term II Started discussion about building an ATCA based HW demonstrator: complementary to GOLD, possibly learning from GOLD experience Look at various options for board control: One board connected via Ethernet, serving as master, other boards in crate connected via PCIe Each module having Ethernet access Controlled by a dedicated processor (PIC, ARM) on board Controlled by hard/soft core in one of big FPGAs Running embedded code or full (Linux) OS? Some big, fast and expensive FPGAs! Some optical links! Designed at Bham or in collaboration with other L1Calo institute(s) J. Bracinik, R.J.Staley UK L1Calo upgrade meeting 3/2/2011

Future plans – middle term III Possible shopping list for this year: Buy ATCA crate Which type? Would be nice to use the same technology within L1Calo Buy some test board(s), get experience with modern FPGAs and microcontrollers experiment/research with module control schemes, Ethernet and PCIe Try to build an ATCA based demonstrator towards the end of 2011 Exact design depending on the experience with GOLD and L1Calo UK plans Feedback welcome !!! J. Bracinik, R.J.Staley UK L1Calo upgrade meeting 3/2/2011

UK L1Calo upgrade meeting 3/2/2011 Backup slides J. Bracinik, R.J.Staley UK L1Calo upgrade meeting 3/2/2011

UK L1Calo upgrade meeting 3/2/2011 J. Bracinik, R.J.Staley UK L1Calo upgrade meeting 3/2/2011

UK L1Calo upgrade meeting 3/2/2011 J. Bracinik, R.J.Staley UK L1Calo upgrade meeting 3/2/2011

UK L1Calo upgrade meeting 3/2/2011 J. Bracinik, R.J.Staley UK L1Calo upgrade meeting 3/2/2011