1. Project Overview Ted Liu Fermilab Sept. 27 th, 2004 L2 Pulsar upgrade IRR Review http://hep.uchicago.edu/~thliu/projects/Pulsar/L2_upgrade_meeting.html Materials for the review can be found at:

2. Q&A Are we ready for operation?  No Why are we having this review then?  To help us getting ready for final operation. To review - what we have achieved so far - our plans for final commissioning - is there anything missing on our to-do-list? - is manpower enough? - Is the schedule realistic? - … Will have another review early next year

3. Qutline Baseline design for operation and overall status Project Organization: who is working on what General testing&commissioning methodology Firmware Overview and necessary improvements Firmware Version Control Comments on possible future system optimization Manpower issues Summary

4. Baseline for operation Pulsar-based system is very flexible Should only focus on the baseline for operation  single algorithm node  deliver non-SVT and SVT data on separate PCI path  simple algorithm in pre-processor Pulsars  simple data merging  use S32PCI64 cards instead of FILAR … Will comment on some system flexibility later  further possible improvements/optimization depend on initial operation experience, not required for operation

5. L1 muon L1 XTRP L1 trigger L2 CAL (CLIST/Iso) PreFred ShowMax (RECES) PC SLINK TS SVT Muon Cluster Electron Merger SVT L2toTS Commissioning strategy: (1)Use Pulsar Tx  Rx in teststand (2)Split all input signals to Run IIa system, run parasitically with detector and beam Baseline for operation Pulsar pre-processors

6. Main tasks for the project Hardware: custom: Pulsar,mezzanine&AUX, LVDS & fiber active splitter commercial: fiber passive splitter, SLINK mezzanine, SLINK-PCI cards, Linux processors Firmware: transmitters and receivers for all data paths SLINK merger and L2 to trigger supervisor interfaces  about 14 different types of Pulsar firmware Software: board testing, VME DAQ readout, online/offline monitoring, software for algorithm&control node, interface with rest of world …

7. Overall Status Hardware: All custom hardware in hand, and all fully tested except fiber active splitter (see Chris’s talk) need to order two Algorithm PCs soon, one in hand Firmware: All firmware in place and tested in beam…some needs fine tuning for either performance optimization or robustness  more on firmware later Software: main work left is related to algorithm and control node see later talks for details

8. Who is working on what (I) Muon/XTRP/L1 + SVT + Merger + RECES + L2TS  Tx Firmware: Tomi (work done, left)  Rx Firmware: Sakari  Testing: Burkard, Frans (left), Cheng-Ju (L2TS) + Sakari  Commissioning: Burkard + Cheng-Ju  Monitoring for Muon/XTRP/L1/SVT/Merger: Shawn  Monitoring for L2TS: Cheng-Ju  Monitoring for RECES: Chris with Daniel’s help earlier  Active Splitter for RECES: Chris  LVDS Splitter Board: Wojtek (only the main soldiers are shown here)

9. Who is working on what (II) Cluster/PreFred  Tx Firmware: Tomi (work done, left)  Rx Firmware: DataIO FPGA: Vadim Control FPGA: Sakari (similar to other cases)  Testing/commissioning/monitoring: Vadim and Wojtek Algorithm node software: Kristian Control node software: Daniel (only the main soldiers are shown here)

10. Who is working on what (III) General  Pulsar Hardware: Burkard  VME readout software: Jane from DAQ + Burkard’s help  PulsarMon design: Vadim  RunCTRL/TriggerDB etc interfaces: Daniel  Firmware coordination: TL  Commissioning coordination: in the past: TL in the future: Trigger SPLs (See Cheng-Ju’s talk on plan and schedule)

11. Testing and Commissioning Methodology Pulsar is self-testable, at both board and system level This design feature is the key to success  Pulsar Hardware testing all done with self-testing capability  Pulsar firmware testing mostly done with self-testing capability  Software development all done with Pulsar self-testing capability  Even at system commissioning, a few Tx boards are used  This has made the commissioning work much easier System commissioning relies heavily on input signal splitting  Saved us LOTs of dedicated beam time  So far, only used a few hours dedicated beam time  Input signal splitting has been and will be extremely helpful  Will keep input signals split even after final commissioning  to have a “copy” of the new system running parasitically  for hot spares, for firmware/software improvements

12. Pulsar Firmware: BIG effort Pulsar has three main FPGAs L2 decision upgrade:14 different type Pulsars (Tx/Rx) Key to success:  Design: common re-usable modular design  Simulation: no simulation  no testing  Dedicated testing support  Version Control  Careful firmware update procedures …

13. 081 L2 Pulsar Muon/XTRP Rx IIa 083 L2 Pulsar SVT Road Warrior 085 L2 Pulsar Muon/XTRP/L1 Tx or SVT XTRP-emu 086 L2 Pulsar Muon/XTRP/L1 Rx IIb 087 L2 Pulsar RECES Tx 088 L2 Pulsar RECES Rx 089 L2 Pulsar Cluster/PreFred Tx 090 L2 Pulsar Cluster/PreFred Rx 091 L2 Pulsar SVT Tx 092 L2 Pulsar SVT Rx 093 L2 Pulsar Merger Tx 094 L2 Pulsar Merger Rx 095 L2 Pulsar L2TS “Tx” (reserved spare ID) 096 L2 Pulsar L2TS 097 L2 Pulsar L1 Scaler 098 L2 Pulsar SVT Track Fitter 099 L2 Pulsar test Tx 100 L2 Pulsar test Rx 101 L2 Pulsar XFT Stereo Tx 102 L2 Pulsar XFT Stereo Rx 103 L2 Pulsar SVT Hit Buffer 104 L2 Pulsar SVT AMSRW Board ID One Hardware -- many applications 14 Pulsar types/firmware for L2 decision upgrade Sakari Tomi Sakari Tomi Sakari Tomi Vadim/Sakari Tomi Sakari Tomi Sakari people other types for SVT/XFT or future improvement

14. Interfaces (Tx/Rx): SVT/XTRP/L1/PreFred TS & SLINK (P3) Interfaces (Tx/Rx): Muon Reces Cluster Iso Slink FPGAs and data interfaces

15. General simplified data flow Mezz interface DAQ buffers algo slink formatter Same as above VME response CDF CTRL DataIO DAQ buffers merge slink formatter DAQ buffers VME response CDF CTRL DAQ buffers XTRP/SVT

16. Well organized firmware effort is the key: clean/easy-to-understand/robust with minimal maintenance effort Input interface DAQ buffers algo Output FIFO Slink formatter VME response CDF CTRL DAQ buffers all common parts are shared by all firmware also share many low level routines (latency timer, bunch counter etc) DataIO or Control FPGA

17. VME responses SLINK formatter DAQ buffers CDFCtrl responses Mezzanine interfaces Latency timer Bunch counter … Common core firmware (library) Knowing one data path Pulsar  knows all Pulsars Data specific algorithm difference is just minor details hotlink muon cluster Taxi Iso showermax LVDS + ext. FIFO SVT track LVDS L1 TS core firmware developed by people dedicated to the task

18. Format version Data source Region ID Reserved Bunch count Buffer # 842828 Data elements Latency (previous)Latency (current) 16 Data sizeError flags 16 Keep it simple SLINK data format was defined as data size before, which means one has to wait until all data arrives before sending out SLINK package. will remove it, and use it as latency stamp instead.

19. SLINK Merger  fragments into event package Slink inputs AUX card Slink output Four Slink mezz cards Merger header 2 Merger trailer Input 1 header 1 Input 1 trailer Input 2 header 1 Input 2 trailer Input 3 header 1 Input 3 trailer Input 4 header 1 Input 4 trailer Slink input 1 data Slink input 2 data Slink input 3 data Slink input 4 data Input 1 header 2 End of fragment Input 2 header 2 Slink merger output End of fragment Input 3 header 2 End of fragment Input 4 header 2 End of fragment 0xE0F00000 Merger header 1

20. 32-bit SLINK to 64 bit PCI interface card: S32PCI64 highly autonomous data reception 32-bit SLINK, 64-bit PCI bus 33MHz and 66 MHz PCI clock speed Slink to PCI Slink to PCI mem CPU PCI to Slink data L2 decision

21. General comments on Firmware Firmware is not hard if it just has to be “sort of working” The hard part is to get it “rock solid” Firmware is not a place to show how smart one is rather to show how many mistakes one could make  Firmware testing is as important as writing/simulating  Firmware testing is often much more time consuming than the actual writing  How good the firmware depends on how well it is tested Our general strategy: firmware developer supported by people dedicated to testing

22. Files kept VHDL source code Files necessary for recreating this version Readme with revision history FPGA configuration files the FPGAs Source code (working version) under CVS control Firmware archive updated on different PCs

23. How to keep Firmware in order We have so many different types of firmware Keeping them in order is non-trivial:  under CVS control: source VHDL code, pin files etc  Firmware version ID VME register per FPGA (has to update by hand)  README file update  Follow careful firmware update procedures There has been no confusion about firmware versions when the procedure has been followed

24. te

25. slot 05: IDPROM: 0016 086 PULSAR MUON RX CTRL: 04408060,DataIO1: 08407020, DataIO2: 08407020 slot 13: IDPROM: 0004 092 PULSAR SVT RX CTRL: 05408060, DataIO1: 09407280, DataIO2: 09407280 slot 15: IDPROM: 0028 093 PULSAR SLINK TX CTRL: a0406300, DataIO1: 08407230, DataIO2: 08407230 slot 17: IDPROM: 0061 096 PULSAR TS INTERFACE CTRL: 0a407280, DataIO1: 09408030, DataIO2: 09406040 slot 19: IDPROM: 0012 094 PULSAR SLINK MERGER CTRL: 02408080, DataIO1: 01407220, DataIO2: 01407220 slot 21: IDPROM: 0014 094 PULSAR SLINK MERGER CTRL: 02408080, DataIO1: 01407220, DataIO2: 01407220 Example of Pulsar crate ID&version scan by Burkard on Aug 19 th after Beam test: Not all boards are shown due to limited space Allow us to go back to the same configuration after shutdown How can we check these against database when in operation mode?

26. Necessary firmware optimization before operation Reces: zero-suppression + latency optimization  Reces data comes into Pulsar 8 us after L1A  current firmware is not optimized  needed to reduce Reces path latency by a few us  Status: in progress, estimated time: ~ one month Slink formatting: sending data words out upon arrival (not waiting for all data to arrive)  improving Slink merging latency (~ a few us)  estimated time: ~ one month (including testing) More timing measurements  add latency timer in a few other places (relatively easy) Cluster path improvement (see Vadim’s talk)

27. Future possible system optimization after operation Use FILAR cards instead of S32PCI64  Remove the final merger, data directly goes into FILAR  Possibly also remove Reces merger with another FILAR  Also allow us to run SLINK faster: 40 MHz  ~60 MHz  Should reduce the latency by ~ few us  Need PCI software modification (no firmware change) Move L1 scaler into firmware if needed Move to 4 PCs instead of single PC (how much do we gain?)  Additional firmware (L2TS) work and some software work Muon and track matching in firmware Bring upgrade XFT stereo info into L2 (this will happen) …much more, depends on what we learn&need by then

28. FILAR can improve the system latency  Higher bandwidth, less latency overhead 64-bit/66MHz PCI bus, four 2 Gbit/s S-LINK channels Reduced PCI protocol overhead (compare to S32PCI64)

29. L1 muon L1 XTRP L1 trigger L2 CAL (CLIST/Iso) PreFred ShowMax (RECES) PC SLINK TS SVT Muon Cluster Electron Merger SVT L2toTS System with FILARs Pulsar pre-processors Send all data paths directly into 2 FILARs…

30. Manpower Issues Why should we worry?  Tomi and Frans left (two key people on firmware/testing)  Only10% Burkard’s time available from now on (key person on hardware/VME software/firmware testing/system commissioning  need replacement SOON)  Kristian needs to graduate … (key person on algorithm/PCI software  need replacement SOON …)  Need new L2 software coordinator (Daniel?) …  anyone else will be full time on the project? This is very serious …need this committee to help!

31. Summary Review should focus on baseline for operation Keep in mind that system is flexible Hardware is in good shape (still need to fully test the active splitters) Firmware is in reasonable shape, needs fine tuning/testing in some cases  key is to follow our testing methodology Software needs more work, no show stoppers  clear spec is the key Manpower issues serious (in a few areas): need new people! Details are in the rest of talks …