Fysikalisk Systemteknik 1 Instrumentation seminar Fysikalisk Systemteknik Christian Bohm Overview: About the group Overview of projects What is an FPGA Our major projects

Fysikalisk Systemteknik 2 Instrumentation seminar Fysikalisk Systemteknik Personell: Professor Christian Bohm Lecturer Sam Silverstein Part time lecturer Magnus Engström Adjunkt Eddie Ahlestedt Forskningsingenjör (emeritus) Hans Eriksson Forskningsstuderande: Jonas Klereborn Abdelkader Bousselham Attila Hidvegi Florian Bauer (external) New We collaborate with experimental physics groups, focusing on the development of new instruments. Make it easier to develop and maintain useful engineering skills while retaining an active grasp of the relevant physics. Use experience from projects to solve new problems. Look for general solutions and methodologies which are easier to carry over to new problems. Experimental PhysicsTechnology Instrumentation Physics

Fysikalisk Systemteknik 3 Instrumentation seminar Different instrumentation projects: In collaboration with particle physics, SU RD-16 FERMI and RD27 first level trigger Digitizing electronics for ATLAS TileCal The Jet/Energy-sum processor for the ATLAS first level trigger In collaboration with physicists at KI Development of a SPECT camera Development of a PET-Camera In collaboration with molecular physics, SU Frequency stabilisation of semiconductor lasers for laser trapping and cooling of atoms In collaboration with astroparticle physics, SU Participation in the development of IceCube

Fysikalisk Systemteknik 4 Instrumentation seminar What is a Field Programmable Gate Array? Configuration memory Logic with Data path switches Firmware

Fysikalisk Systemteknik 5 Instrumentation seminar Configurable Logic Blocks FPGA Programmable interconnect a d fgclkh e c b Programmable IO-blocks Configuration memory

Fysikalisk Systemteknik 6 Instrumentation seminar Configuration memory pattern defines circuit a d fgclkh e c b a d e c b

Fysikalisk Systemteknik 7 Instrumentation seminar Modifying the memory content changes the circuit a d fgclkh e c b a d e b

Fysikalisk Systemteknik 8 Instrumentation seminar FPGAs have been around since mid-1980s Early components were programmed at a bit-level using graphic editors Increased complexity required better methods: High level languages (VHDL), or Schematic specifications

Fysikalisk Systemteknik 9 Instrumentation seminar When designing complex circuits with FPGAs one has to consider: Does the design fit? Is it fast enough? Is it too expensive?

Fysikalisk Systemteknik 10 Instrumentation seminar FPGA design process High level description (VHDL) Functional simulation (no timing) Select FPGA type Synthesize – translate to simple primitives (compilation) Simple timing simulation Place and route Full timing simulation

Fysikalisk Systemteknik 11 Instrumentation seminar State-of-the-art FPGAs Very complex (Xilinx, Alterra) many gates ~ 8 million gates many i/o pins ~ 800 (400 diff) flexible interconnects ~ 7 metal layers high costs ~ 50 kkr Multiple clocks - 8 Embedded memories – 4 MB Embedded multipliers Embedded processors – 4 PowerPC High speed IO – 16 x 3 GB/s

Fysikalisk Systemteknik 12 Instrumentation seminar Re-use of previously developed code blocks Intellectual Property blocks IP-blocks can be: In-house developed Commercially available Freely available – open-core Part of the design can be accomplished by assembling compatible IP-block Processors (embedded or IP) Memories Busses Interfaces Etc. Efficient tools required

Fysikalisk Systemteknik 13 Instrumentation seminar What to implement in logic What to implement in processor software Hardware software co-design VHDL  System-C or Handle C When designing complex FPGA modules one must decide:

Fysikalisk Systemteknik 14 Instrumentation seminar ATLAS data flow LHC physics looks for rare events – 1 in  High event rates and  High selectivity 1 event in new data every 25 ns Since all data must be stored while waiting for the L1 decision the L1 processing must be quick – 1ns Data from entire detektor but with low spatial resolution and reduced dynamic range from calorimeters and muon detector 10 Hz 1 kHz kHz 40M Hz About 100 million channels 1 event in 100 Data from ROIs with high spatial resolution and full dynamic range from all subdetector Entire detector with high spatial resolution and full dynamic range from all subdetector

Fysikalisk Systemteknik 15 Instrumentation seminar Muon Trigger (Italy) Calorimeter trigger Central Trigger (CERN) L1 accept ROI info The calorimeter trigger is a Birmingham-London -Rutherford-Stockholm-Heidelberg-Mainz collaboration Looks for typical features for event selection Preprocessor (Heielberg) Calorimeter trigger Electron/Tau Processor (GBR) Jet (Sthlm) and Missing energy (Mainz) processor Analog Input signals 64x64x2 Analog signals Digitizes determines amplitudes and pulse starts Looks for isolated clusters resembling single Electrons/hadrons in the ECAL and HCAL 64x64x2 8-bit 32x32x2 8-bit Looks for energy clusters ATLAS first level trigger collaboration with particle physics SU Looks for energy balance

Fysikalisk Systemteknik 16 Instrumentation seminar Cluster Processor (e/  and  /had) Cluster Finding Cluster Processor (e/  and  /had) Cluster Finding Calorimeter LAr, Tile Calorimeter LAr, Tile   Tower 0.1 x 0.1 Pre-Processor Timing alignment 10-bit FADC FIFO BCID Lookup Table BC-MUX Sum 2x2 Pre-Processor Timing alignment 10-bit FADC FIFO BCID Lookup Table BC-MUX Sum 2x2 Analog Count Jet/Energy-Sum Processor Jets ET EX EYET EX EY Count  E T, E T Pre-Processor RODs (DAQ) Pre-Processor RODs (DAQ) CP/JEP RODs (DAQ) CP/JEP RODs (DAQ) Region Of Interest Builder (L2) Region Of Interest Builder (L2).1 x.1.2 x.2 Level-1 CTP Level-1 CTP Realtime data path

Fysikalisk Systemteknik 17 Instrumentation seminar The JET/energy sum trigger Look for.4x.4,.6x.6 and.8x.8 energy clusters centered around a local.4x.4 maximum Form global sums of total Et and missing Et Process ~1024.2x.2 jet elements in parallel All requiring neighborhood information 32 processor boards with large FPGA for Jet and missing energy processing, sharing overlapping environment data Latency (processing time) 200 ns Many different module types Standardized modules

Fysikalisk Systemteknik 18 Instrumentation seminar The JET trigger We have built a 18 layer backplane with > pins for the Jet and the E/  processors VME - - Communication with neighbors Report results

Fysikalisk Systemteknik 19 Instrumentation seminar The JET trigger We have participated in the design of the JEM Processor board. And developed firmware for the algorithms and control functions

Fysikalisk Systemteknik 20 Instrumentation seminar Experiences from the trigger project: Large scale system design Massive pipelined parallel processing Reliability Large FPGA design (> 1 Mgates) Draw on experience from earlier bit-serial trigger project to do pipelined processing of multiplexed data -> more efficient use of logic and interconnects!

Fysikalisk Systemteknik 21 Instrumentation seminar Many prototypes – test beam tests – earliest ATLAS subsystem – lots of firsts – production experience – 2000 boards this year The ATLAS TileCal Digitizer collaboration with particle physics SU

Fysikalisk Systemteknik 22 Instrumentation seminar Task: to digitize pre-amplified PMT-pulses and to transfer data selected by the L1-trigger to the higher level triggers. The ATLAS TileCal Digitizer 16-bits dynamic range with limited precision L1 buffer memory – 2.5 us Storage of selected data Format data send to level 2 Physical layout Noise control Radiation tolerance Reliability (physical chain – electrical star) We also made a optical link with matching reliability

Fysikalisk Systemteknik 23 Instrumentation seminar Analog part Digital part Noise control Reliability 16-bits dynamic range with limited precision 10 L1 buffer memory – 2.5 us L1 buffer Physical layout ADC L1a L1 buffer Event storage Format and send Storage of selected data Event storage L1a Trigger and Timing Circuit System clock – level 1 accept Format data Format and send Send to level 2 Data till second level trigger Radiation tolerance Radiation hard ASIC Radiation tolerant custom ASIC - no FPGA Components Of The Shelf - COTS 10 ADC Hi gain Lo gain

Fysikalisk Systemteknik 24 Instrumentation seminar Experiences from the digitizer project: Large scale system design System aspects – timing and grounding Reliability Radiation tolerant design Production Even if did not use FPGAs in the Digitizer we used them extensively when building prototypes and testbenches.

Fysikalisk Systemteknik 25 Instrumentation seminar SU – SPECT Collaboration with Karolinska hosptal 72 PMTs around crystal – position determination via light sharing Earlier design based on transputers discontinued Pulse detection + sampling ADCs Digital pulse processing + digital trigger Firewire network Xilinx FPGAs Texas Instrument DSP – TMS family The design of a SPECT camera with an innovative cylindrical crystal

Fysikalisk Systemteknik 26 Instrumentation seminar ICE-CUBE Collaboration with the astroparticle physics group at SU 80 strings Volume 1 km 3 Self-triggers on each pulse Captures waveforms Time-stamps each pulse Digitizes waveforms Performs feature extraction Buffers data Responds to Surface DAQ Set PMT HV, threshold, etc Noise rate in situ: ≤500 Hz 1400 m 1000 m 60 modules/string Digital Optical Module (designed by D. Nygren)

Fysikalisk Systemteknik 27 Instrumentation seminar ICE-CUBE The DOM circuit board 2 DOMs share 1 twisted pair for power supply and communication 2 ATWD - 4 channel transient waveform recorder 300 MHz 256 samples 2 channels – hi and lo gain from PMT Symmetric timing pulses between hub and DOM sampled at 20 MHz 10bits Supports a higly stable local clock 3.3 ns rms FPGA and CPU combined in new Altera FPGA Our part feature extraction

Fysikalisk Systemteknik 28 Instrumentation seminar ICE-CUBE Experimental Requirements IceCube Time resolution:<5 ns rms Waveform capture: >250 MHz - for first 500 ns ~40 MHz - for 5000 ns Dynamic Range: >200 PE / 15 ns >2000 PE / 5000 ns Dead-time:< 1% OM noise rate:< 500 Hz ( 40 K in glass sphere) -- DOM Pair 20 kB/sec String Processor N x 20kB/sec All Hits MB/sec 80 Strings String Subsystem: 60 DOMs N pairs 100BaseT Total traffic: 1.6 MB/sec String Coincidence Messages Global TriggerEvent Triggers / Lookback Requests for all Strings- 0.8 MB/sec Event Builder Built events ~ 1 MB/sec ( all event builders) SAN (Network Disk Storage) "DOM HUB" Lookback Requests String Coincidence Messages - 170kB/sec Fulfill Lookback Messages 0.6 MB/sec Fulfill Lookback Messages Online LAN 100 BaseT Total traffic: 1MB/sec Proposed IceCube DAQ Network Architecture String LAN 100BaseT Total traffic: 0.6 MB/sec Offline Data Handling Tape Satellite Event LAN

Fysikalisk Systemteknik 29 Instrumentation seminar Digital Laser Control Collaboration with Anders Kastbergs group Frexghi Habte Aim: to design a simple laser control that can manage a large number of units Our solution: use an FPGA based lock-in module Cordic algorithm to produce sine and cosine waveforms + second order Butterworth low pass Filter (4Hz) Hardware design based on SPECT module laser Absorption cell Modulation detector Lock-in amplifier Lock on low frequency component 0  lock on maximum x Asin(  t+  )Asin(2  t+  )-Asin  Asin  cos(  t)/2

Fysikalisk Systemteknik 30 Instrumentation seminar Future Our involvement in the ATLAS projects will eventually decrease – the digitizer during 2004 and the trigger during Now they are quite intense The SPECT camera project should terminate In its present form this year. We are participating in a EU application Coordinated by Anders Brahme at KI. Our part here would be in the development of a high resolution whole body positron camera. Lars Eriksson from KI and CPS would be partner in this project. There will surely be other exciting new projects coming up.