1. For more information on Pulsar board: http://hep.uchicago.edu/~thliu/projects/Pulsar/ Burkard Reisert (FNAL) Nov. 7 th, 2003 PULSAR Production Readiness Introduction Testing the PULSAR Tools involved People

2. Gigabit Ethernet RF clock SRAMs Pulsar is designed to be: Modular, universal & flexible, fully self-testable (board &system level) All interfaces are bi-directional (Tx & Rx) Lego-style, self test capability Spare lines one for all and all for one user defined interfaces Personality cards Standard link another Pulsar or S-LINK to PCI Has ALL interfaces L2 decision crate has In God we trust … …the rest we test

3. Bottom view Top view Mezzanine slots AUX card S-LINK General purpose, useful within & outside CDF Pulsar

4. AUX Card Pulsar Hotlink Tx/Rx Prod./testing done Taxi Tx/Rx Prod./testing done SLINK LSC/LDC (have some, need to order more) ANL SLINK->GBE (production) Pulsar Hardware testing started Six S32PCI64 in hand Ready for Production Mezz. Cards Mother Of All Boards

5. History 4 prototype Pulsar boards (late02/early03) 4 pre production Pulsar boards (August 03) (thin PCB 63mil instead of 96mil, vendor admitted error, boards still useful) 2 boards new pre production version (October 03) Not a single blue wire  No revision Prototype  Production

6. How We Tested TxRx 1. Standalone Test Stand Mode 2. Test Stand: Pulsar  PC 3. Pulsar in beam self test all interfaces Pulsar AUX PC CERN Slink Cards UofC Slink HOLA Card ANL Slink to Giga Bit Ethernet  XTRP RX CDF Muons Tracks CDF Level2 Trigger 4. Test procedure for Pulsars: See next slides millions of events

7. Registers 3 Power LEDs 3.3, 2.5, 5V Control FPGA DataIO FPGA 1 VME chip Visual Inspection Put fuses, jumpers, Oscillators Power up JTAG-detect and program VME Chip, DataIO1&2, Control FPGA First VME Access to -- R only registers -- R/W register All Ok DataIO FPGA 2 Initial Checkout of New Pulsar Boards 3 VME LEDs JTAG

8. DataIO FPGA 1 VME chip DataIO FPGA 2 VME access to SRAMs & internal RAMS SRAM Two 128Kx36 SRAMs on board Each DataIO FPGA has control of one SRAM Write and read SRAMs through VME Load SRAMs with 128K test patterns internal RAMs implemented in both DataIO+ControlFPGA Run Test Loop ALL OK Control FPGA iRAM

9. Control FPGA DataIO FPGA 1 VME chip DataIO FPGA 2 P2 CDF control signals Pulsar FPGAs see ALL P2 CDF control signals Used Testclk to toggle all signals Signals recorded by circular buffer RAM in each FPGA Also checked with Logic Analyzer Data matches RAM P2

10. Control FPGA DataIO FPGA 1 VME chip DataIO FPGA 2 Pulsar P2 inter-communication lines Pulsar has five SVT style inter- communication lines on P2 Data is sent from Control FPGA on one Pulsar and received by all three FPGAs on the other one Run test loop All Ok P2 In/out registers Input register Input register

11. Control FPGA VME chip Pulsar P3 spare lines Pulsar has 25 spare lines to P3 Data sent from Control FPGA initial Check with Logic Analyzer from P3 back plane later with SVT input from AUX mapped to the spare lines All Ok Output register LA P3

12. Control FPGA VME chip TSI interface connection TS cable Data sent and received by Control FPGA Run test loop All Ok In/out Registers

13. SVT data path SVT cable SVT data out SVT data in Control FPGA DataIO FPGA 1 DataIO FPGA 2 RAM VME chip Input data is uploaded to Control FPGA RAM using VME Control FPGA sends SVT data out from SVT output All FPGAs receive incoming SVT data Data is read from each FPGA using VME Run test loop All Ok Receiver FIFO

14. L1 data path: input and output Data is sent out from one Pulsar and received by another one Control FPGA sends data from internal RAM Data is received by all three FPGAs Read-out through VME Run test loop All Ok DataIO FPGA 1 VME DataIO FPGA 2 Control FPGA Input register VME Control FPGA Output RAM Input register

15. Hotlink TX Mezzanine Cards Interface Test Pattern loaded into Tx DataIO FPGA Data is driven out on L1As Data recorded into Rx DataIO FPGA Fifos Run test loop All Ok Tested with --Hotlink 20Mhz & 4x CDFClk -- TAXI 12MHz & CDFClk RX 16x

16. SLINK formatting Control FPGA DataIO FPGA 1 VME chip DataIO FPGA 2 Spy FIFO Spy FIFO Spy FIFO P3 Load data into TX Receive data on Rx mezzanine cards DataIO FPGAs send merged input data in SLINK format to Control FPGA Control FPGA merges incoming data and sends it out in SLINK format from P3 Outgoing data is stored into a Spy FIFO in each FPGA, and it can be read from the FIFOs using VME Data in the Control FPGA Spy FIFO matches the data uploaded to input Tx RAMs LALA Data matches

17. SLINK Interface Load Test Pattern / Event data into output RAM Outgoing SLINK data goes from P3 to AUX card, with LSC (link source card) LOOP BACK -- Receive data back on LDC (link destination card) SLINK  PC via CernSLINK/HOLA/GBE -- Send event to PC -- Run L2 algorithms -- send decision back Timing measurement Run test loops PulsarAUX card SLINK Source Card Destin. Card PC Control FPGA VME chip Output Ram Receiver Fifo Loop back ALL OK Thanks to Kristian Hahn (Upenn)

18. Muon Input Data XTRP input Output to TrkList Pulsar as Level 2 Muon Interface Board  Alpha processor 12 matchbox & 4 prematch fibers Pulsar receives -- XTRP data -- L2 Muon data Pulsar appends Zero suppressed Muon data to Tracks Pulsar sends Track & Muon data To L2 Alpha processor (alpha code: unpacking of tracks & Muons L2 algorithm)

19. RunIIa L2 Muon path commission: methodology at work  from discussion to error free path: < 3 months June  Sept. 2003 L1 Muon (16 hotlink fibers) and L1 track input Pulsar muon+track data transmitter upstream Data source Pulsar Muon Board Legacy L2 decision crate downstream Data sink Pulsar Receiver We didn’t waste one second of beam time The FULL chain test with collision beam works on the first try (error free) Fully self-tested before put in the running exp. up to 1 Billions events in self-test mode.

20. Muon Input Data XTRP input Output to TrkList Pulsar as Level 2 Muon Interface Board 12 matchbox & 4 prematch fibers What has been tested with beam? Mezzanine Card Interface XTRP/SVT INPUT FPGA Algorithms at 80MHz Board internal data transfer XTRP/SVT OUTPUT CDF Control Signals DAQ readout VME CDF Signals

21. Test Runs Summary STORE 2985 2003/09/03 21:49 2003/09/04 14:51 Run# L3A Lumi_live Max. rate Avg. rate Inst.Lumi (E30) (nb-1) L1 / L2 / L3 L1 / L2 / L3 start/end 168766 335,930 196.220 15.0k/268/58 13.3k/255/53 35.0/30.2 168767 155,974 85.949 13.7k/240/53 13.0k/230/50 29.5/28.0 168775 596,981 258.197 12.5k/190/45 10.6k/169/39 19.7/15.8 STORE 2988 2003/09/04 21:26 2003/09/05 10:21 Run# L3A Lumi_live Max. rate Avg. rate Inst.Lumi (E30) (nb-1) L1 / L2 / L3 L1 / L2 / L3 start/end 168819 90,076 76.570 16.5k/148/30 13.8k/132/27 24.7/23.1 168820 1,101,959 471.469 14.7k/235/50 12.0k/193/47 22.7/14.8 168821 166,146 114.954 9.6k/ 94/22 8.6k/ 86/20 14.5/13.2 168822 121,685 50.591 8.4k/134/34 7.3k/121/30 13.2/12.6 STORE 2988 2003/09/04 21:26 2003/09/05 10:21 Run# L3A Lumi_live Max. rate Avg. rate Inst.Lumi (E30) (nb-1) L1 / L2 / L3 L1 / L2 / L3 start/end 168889 2,816,155 1,359.049 15.5k/300/60 12.2k/205/45 40.0/13.7 ALL data collected for Pulsar beam test are good for physics analysis. No single Pulsar hardware/firmware problem was seen (~ 5M evts.)

22. PULSAR TEST TOOLS

23. PULSAR PRODUCTION TEST SETUP PC AUX Crate Controller: TSTL2TRG1TRACER Pulsar Receiver (Rx) Pulsar Transmitter (Tx) NEW BOARD Hardware and Firmware to test all paths in place Automated software testing procedures in place Ready to fully test new Pulsars XTRP/ SVT Hotlink/Taxi : TSI L1

24. Pulsar TEST VME Software Test Pattern Generation & Analysis TestClock Utility Functions Pulsar Data Path Tests & Utility Routines Infrastructure: vxworks, VISION Receive & Analyze Data on PC by Kristian (Upenn) Pulsar Standalone Tests Run Control Java GUI FER-code Front End Readout Same package used for testing system firmware Easy access through Java GUI (  version by Mikko)

25. Automated Test Procedures TestRAM() reading list of RAMs from steering file ram.test TestSVT(1000000, 1, 1, 15, 15) SVT/XTRP input&output TestHotlink(12345, 1, 1, 7, 15) mezzanine card interface TestL1(111111111, 1, 1, 7, 15) L1Trigger bits TestIntCom(98763, 1, 7, 15) Internal Communication lines : Pulsar_forever_andever(15) “infinite” SLINK loop back test iterations pattern type L1A mode Tx Slot Rx Slot Tested interface Too lazy to type?  Use the PulsarTestGui !

26. Pulsar Test GUI 0. select crate 2. Load code 1. reboot 3. Map crate 4. Select test, Tx Rx slot, start test Display results for experts: command line entry read/write to reg. resets JavaGUI Light weight interfacing to Pulsar test c-routines

27. Man Power for Testing People being trained for production testing: Vadim Rusu (New UC PostDoc) Chris Neu (New Upenn PostDoc) Mikko Hakala (FNAL, already trained) Consultants: Burkard Reisert (FNAL) Ted Liu (FNAL)

28. We have: tested prototype & pre production boards -- all in test stand self test mode, all interfaces -- some as MUON/XTRP Rx in beam condition well defined test procedures necessary infrastructure in place -- Hardware (Crate, Tx/Rx Pulsar, Mezzanine cards) -- Software (VME c-code, JavaGUI) well trained people  We are ready for production  Waiting for pulsar hurricane to come Conclusion