1. D. Saltzberg, 7 Dec 01 L2 Review Level-2 Interface Board Status David Saltzberg for L2 Group Level-Two Trigger Review December 7, 2001

2. D. Saltzberg, 7 Dec 01 L2 Review Overview l Phase 1: L1interface, Clist, XTRPlist, SVTlist l Phase 2: ISOlist, RECES l Phase 3: Muon board (not in this talk) (Phase 1 and Phase 2 have been done in parallel)

3. D. Saltzberg, 7 Dec 01 L2 Review Responsible Physicists l L1 interface: Greg Feild * l Clist: Monica Tecchio, Heather Ray * l XTRPlist, SVTlist: Matt Worcester *, Jane Nachtman, D.S. l RECES: Masa Tanaka *, Karen Byrum * l ISOlist: Steve Kuhlmann *, Bob Blair * * = lives within 50 miles of Fermilab ANL engineers (L1,reces,isolist): John Dawson, Bill Haberichter Special operatives: Stephen Miller, Ted Liu, Peter Wittich

4. D. Saltzberg, 7 Dec 01 L2 Review Theory of Operation - I l Input data from “Clients” äL1 interface, RECES - one word/event, no handshake äClist, XTRPlist, SVTlist, Isolist: - variable length data, buffered by FIFO’s - terminated by EE word - Some info transfer about BC, L2B or event count for sync checking

5. D. Saltzberg, 7 Dec 01 L2 Review Theory of Operation - II l Output: Control and Data signals via Magic Bus äMaster mode (currently all boards except Reces) - L2P issues “STARTLOAD” - When ready, Interface board requests “Boss” - Board is granted Boss from upstream - Board drives block mode data-transfer on Bus - Boss is released by interface board, and MOD_DONE asserted - When all MOD_DONE bits set, L2P begins processing äSlave mode - Board is addressed over Magic Bus and read in single-word transfers l Alternate Output (TRKlist boards) äVME readout

6. D. Saltzberg, 7 Dec 01 L2 Review General Error Detecton & Handling l In L2P (every event--10’s kHz) äL2P has 600  sec timeout for all MOD_DONE signals äBC, L2B, or counters checked where possible event by event äChecks for exactly 1 magic bus word from L1 board äIf error, pull CDF_ERROR (or equivalent) and ask for automatic Halt- Recover-Run to resynch FIFO’s. l In TrigMon (~ 2 Hz) äCheck Number of words transferred for each board äCheck BC across system äExact bit-for-bit comparison of data vs. emulation and/or alternate source l Offline äRun select parts of TrigMon & Monica’s validation code on look area, stream-g, stream-b, l2-torture runs (~1M events lately)

7. D. Saltzberg, 7 Dec 01 L2 Review Testing Performance of system l Without Beam ä“L2 torture” nominally runs at ~20 kHz äOccasionally have run system at ~40 kHz äRuns system with high L2B occupancy äTest patterns in for COT tracks, SVT tracks, emulate clusters ä9 interface boards & up to 3 alphas connected l With Beam äSame config. äGet real XFT tracks but often have to run SVT test patterns (no SVX) äHave found other problems (sometimes systemwide) that tests w/o beam do not show. (Real world stuff that no teststand will anticipate) äExtensive tests before Oct. shutdown, preliminary Dec. tests.

8. D. Saltzberg, 7 Dec 01 L2 Review Current Boss Arb. Kludge l Glitch on BOSSGROUT (pecl) when taking BOSS can lead to two boards taking boss. Since in hardware (not firmware), cannot make simple glitch protection l Solution: äReduce collision rate by putting different delays in boards’ receiving of STARTLOAD (limits deadtimeless L1A rate at 20kHz--we should have such problems.) äHandle remaining collisions with L2P error handling äNew Backplane äIn a pinch, could it be fixed with TTL

9. D. Saltzberg, 7 Dec 01 L2 Review Overview Plot of L2 crate

10. D. Saltzberg, 7 Dec 01 L2 Review Board -by-Board Status (follows...) l Status of “best” board äHighest rate tested & error rate äLimit on (or measurement of) bit error rate äCooperation with other boards äPlans for further work l Status of spares äNumber and status of spares äknown problems? l Status of Documentation l Debugging tools, here and elsewhere l Plans l Other comments

11. D. Saltzberg, 7 Dec 01 L2 Review L1 Interface Board l L2 torture tests ätested at 20-40 kHz no problems ätested ~1M events, no errors tested offline äno collisions with other boards (by construction) l Known problems änoisier than others, but protected in time ästill have to connect ground sheild & see äSolving noise here may solve it elsewhere

12. D. Saltzberg, 7 Dec 01 L2 Review L1 Interface Board Plots No errors in bit-for-bit comparison

13. D. Saltzberg, 7 Dec 01 L2 Review L1 Interface Spares & Debugging tools l Spares äS/N 1 OK äS/N 2 OK (in crate) äS/N 3 3/4 stuffed l Debugging tools äBit for bit check available offline äIf more or less than one word is sent, L2P pulls error ä(Pretty simple board, no need for complex diagnostics) äTeststand: Can set bit patterns, check in realtime or later - data source: FRED - data sink: MB to emulator board

14. D. Saltzberg, 7 Dec 01 L2 Review L1 Interface Documentation/Plans l DOCS äCDFNOTE 4971 äWebpage: http://hepwww.physics.yale.edu/www_info/yale_cdf/l1crate.html äSchematics have control room hardcopy äPDF files recently sent to Greg-- will put on web and in trigger room l Plans äKeep running äFinish stuffing board #3 (2nd spare) and test äLook into noise problem, not urgent. Wait until after new MB installed

15. D. Saltzberg, 7 Dec 01 L2 Review CList Board l Responsibles: Monica Tecchio, Heather Ray l Gets data by fiber from each Locos board l L2 torture tests äworks at 20-40 kHz no errors äno errors found in ~1 M events offline l Known problems äcrate 04-- had bit 02 is stuck low (probably trivial)

16. D. Saltzberg, 7 Dec 01 L2 Review Clist board plots No errors in bit-for-bit comparisons

17. D. Saltzberg, 7 Dec 01 L2 Review L2 cutting on Jets

18. D. Saltzberg, 7 Dec 01 L2 Review Clist Debugging tools l Bit-for-bit comparisons done in online/offline monitoring l If L2 buffer number disagrees L2P pulls error l Clusters can be set äpulling cable in DCAS crate makes a known cluster äin principle software exists to make arbitrary cluster pattern at B0 (need to verify) l Michigan teststand capabilities: äStandalone board tests using VME äData source: Locos äData sink: MB & L2P äTest full clustering chain DCAS ---> L2P via MB w/ tracer generating multiple L1A’s

19. D. Saltzberg, 7 Dec 01 L2 Review Clist Spares/Documentation/Plans l Spares äS/N 1 OK (in system) äS/N 2 flaky VME, otherwise works. äS/N 3 being stuffed l Documentation äwebpage for aces, experts & non-experts - http://www-cdf.fnal.gov/internal/cdfoperations/trigger/level2/my.html äwill become general L2 webpage (need more disk space) äschematics online in Michigan ähardcopies in trigger room l Plans äKeep running stably with board #1, monitor robustness äFix flaky VME on board #2 äMake board #3 a second “hot spare”

20. D. Saltzberg, 7 Dec 01 L2 Review SVTlist Board Tests l Responsibles: Jane Nachtman Matt Worcester, D. Saltzberg l L2 Torture Testing: ä20-40 kHz L1A no errors (SVX off, running SVT test pattern) äTested with ~1 M events no bit errors äSpecial run with checks inside alpha: BER<10 -6 äNo collisions with other boards l Problems äGets confused if no EE word from SVT; L2P pulls error. - Due to SVX not sending info to SVX - Known problems in SVX have been fixed, others? - Bill A. thinking about an SVT timeout to pull error - Only happens with beam. Checked (painfully) before shutdown & it worked (could even have taken special oct. SVT runs with it.) l No firmware changes to TRACKlist boards in last 2 months!

21. D. Saltzberg, 7 Dec 01 L2 Review Some SVTList Plots l No errors in bit-for-bit comparisons

22. D. Saltzberg, 7 Dec 01 L2 Review L2 SVT Cutting (before shutdown)

23. D. Saltzberg, 7 Dec 01 L2 Review XTRPlist Board Tests l Responsibles: Jane Nachtman Matt Worcester, D. Saltzberg l L2 Torture Testing: ä20-40 kHz L1A noerrors äTested with 1 M events no detectable errors - XTRD bank has known errors that cause Ntracks mismatch - Correct at L2, wrong in readout - No errors when cut on Ntrack agreement - Handscan of other events looks okay äNo collisions with other boards l Problems ä Illinois to fix XTRD bank filling errors äOne bad pT bit from one XTRP board

24. D. Saltzberg, 7 Dec 01 L2 Review XTRPlist plots No errors in bit-for-bit comparisons when number of tracks agrees.

25. D. Saltzberg, 7 Dec 01 L2 Review Spares for TRACKlist l SVTList & XTRPlist are both instances of one board: TRACKlist äCPLD change with JTAG connector äone jumper change l Six production TRACKlist boards äCurrently 2 in L2P crate--permanent äCurrently 2 in SVT crate --1 or both temporary? - one makes nominal SVTD bank. Convenient for booking SVT crate for test runs - having separate boards effectively makes a cable check - another board in SVT crate makes XTRP list---could be removed soon? l Six production boards, at least 2 required in system, maybe 3. Right now using 4.

26. D. Saltzberg, 7 Dec 01 L2 Review TRACKlist spares l S/N 1 & 2: (Prototypes, no longer used.) l S/N 3 XTRPlist OK (in L2P crate) l S/N 4 SVTlist OK -- used for SVTD bank l S/N 5 XTRPlist OK --”hot spare” l S/N 6 SVTlist MB not working, bad connection l S/N 7 SVTlist stuck chisq bit for MB -- used for SVTD bank l S/N 8 SVTlist OK (in L2P crate) All boards work for VME readout

27. D. Saltzberg, 7 Dec 01 L2 Review TRACKlist debugging tools l Can send arbitrary pattern from SVT easily l Can send arbitrary pattern from XTRP (more difficult) l Bit-by-bit checking in TrigMon l Can test BC from XTRP & SVT on every event l UCLA teststand: ädata source: merger board ädata sink: MB and emulator board and/or VME

28. D. Saltzberg, 7 Dec 01 L2 Review TRACKlist plans l Keep running stably l Fix one SVT spare (bad connection makes MB error) l Fix one bad bit on another SVT spare l Wean SVT off of second SVT board l Make sure all six boards are “hot spares” l Print hardcopies of schematics & firmware

29. D. Saltzberg, 7 Dec 01 L2 Review TRACKList Documentation l Web-pages: äSpecs http://buggs.physics.ucla.edu/~nachtman/board/specifications_v1.ps äTIB instructions: http://www-b0.fnal.gov:8000/level2/tib/tib_main.html äTIB database: http://www-b0.fnal.gov:8000/level2/tib/tib_status.html äTIB schematics etc: http://buggs.physics.ucla.edu/~nachtman/tib.html l Schematics on web in.eps format l Need updated hardcopies printed out

30. D. Saltzberg, 7 Dec 01 L2 Review ISOlist status l Responsibles: Steve Kuhlmann, Bob Blair l Calculates 5 isolation sums äDCAS->Iso Pick -->ISOlist äClique ->Isoclique-> ISOlist l L2 Torture tests (or cosmics) äneed to require eta-phi match (~1-3% failure) äperfect at 20-40 kHz in all 5 sums l Problems äwith collisions see eta-phi match (still 1-3% failure), but L2P can check and pass the event äIn 0.5% of events also scatter of expected vs. seen in all 5 sums (less than analog jitter in Run 1) N.B. the whole scatter comes from crate 1, eta=17.

31. D. Saltzberg, 7 Dec 01 L2 Review ISOlist plots

32. D. Saltzberg, 7 Dec 01 L2 Review ISOlist spares l In DCAS crates äNeed 1 ISOclique (have 2) äNeed 6 isopicks (have 8, 1 with stuck bit) l In L2P crate äNeed 1 ISOlist (have 2) l All spares are “hot spares” except for 1 isopick with stuck bit.

33. D. Saltzberg, 7 Dec 01 L2 Review ISOlist Debugging Tools l Standard running äISOpick times out if DCAS does not send data l Standalone code: äwrites to ISOclique (only board with VME) a seed ätell it to read out fixed values to ISOlation system äcan load different values for different buffer numbers äwith a switch, can read energies from DCAS. Essentially this “factors” the problem. l TrigMon & Offline Code äIncorporated isolation variables into Monica’s code äNeed to debug some boundary values against the hardware l Teststand at ANL ädata source: ISOpick ädata sink: MB to emulator board

34. D. Saltzberg, 7 Dec 01 L2 Review ISOlist Documentation/Plans l DOCS äCDFnote 5788 äSchematics in hardcopy in binders at ANL but will come to trigger room äPDF files of schematics (firmware & hardware) are available, will be placed on web by Heather l Plans äContinue running & monitor robustness äGo after eta/phi mismatch (needs coordination between ANL and Michigan) äFind & fix flaky bit in DCAS crate

35. D. Saltzberg, 7 Dec 01 L2 Review RECES status l Responsibles: Masa Tanaka, Karen Byrum l Four boards in L2P crate receive information from SMXR by fiber l During L2 Torture tests (36 kHz) äIn crate, on backplane, but not used by default table äNo negative interactions l Special L2 executable (TEST_RECES table) äL1 input is crossing trigger and 4 GeV elec, 8 GeV photon äruns at 20kHz L1 input, 100 Hz L2A äMaybe small bit errors -- few thousand events äAll SMXR to RECES is okay (at end of shutdown) l Problems äAccidental collisions on Alpha readout äSol’ns: Arnd’s special retry readout code. Stephen will modify FPGA äpossible bit errors (10 -3 )

36. D. Saltzberg, 7 Dec 01 L2 Review Reces Plots

37. D. Saltzberg, 7 Dec 01 L2 Review RECES Spares/Docs/Plans l Need 4 Reces boards in system ä4 in top crate OK ä2 spare boards OK l Docs äCDF 5132 äNeed to put schematics on web & hardcopies in trigger room. l Plans äKeep RECES on backplane during default running äFix readout problem äSearch for BER < 10 -4 in standard datataking & fix

38. D. Saltzberg, 7 Dec 01 L2 Review Reces Debugging tools l Special standalone code äVME based. Set trigger threshold, load SMXR’s äSend bit patterns to RECES board, Alpha reads through VME äCheck bit-for-bit (checks all bits) ä10 Hz (tens of thousands of events OK) l ANL teststand ä Not needed any more l TrigMon plots ätemperature plots ächecks bit-for-bit errors

39. D. Saltzberg, 7 Dec 01 L2 Review Interface Board status by run (documented for collaboration)

40. D. Saltzberg, 7 Dec 01 L2 Review Interface Boards: The Bottom Line l L2 crate with Clist, XTRPlist, SVTlist, L1 interface, ISOlist all work at up to full speed 20 kHz as-is. l Their bit-error rates are measured < 10 -6 (RECES not tested to this level yet.) l Essentially all documentation exists. Some tweaks in progress l There is at least one working spare for every board. l Every board has a real expert living close by l Work in progress fixing up extra boards’ bad bits etc. l In current configuration we can fulfill the charge of running jets, electrons and SVT at 5e31 right now, as-is (assuming all clients are working)---”backups” will only distract.

41. D. Saltzberg, 7 Dec 01 L2 Review Goals of Sept. workshop (for interface boards) äsync errors <10 -6 DONE äcut on jets/ “reliable Clist” DONE ä“reliable L1 board” DONE äautomated HRR DONE ä“solve XTRP problem” DONE (don’t remember what is was, but it works) äreliable SVTlist DONE äSVT kludge path DONE äalpha code for cutting on SVT: Simple code DONE, complete cdf4718-lite underway äSolve clist eta/phi errors for electrons: DONE for electrons (iso needs work) äalpha electron code Debugging äprepare firmware without delays for MB testing DONE ätest boards on new MB NOT DONE ätest isolist and reces DONE ä“improve documentation” DONE -- more to do, as always

42. D. Saltzberg, 7 Dec 01 L2 Review Suggestions-I l Spares should not be kept in lower crate unless being used. Otherwise water leak (it has happened before!) will destroy all boards. Currently squatting on other spare space...could use space allocated specifically for L2 spares l Need more disk space for L2 webpages on B0 machine. l SVT group should use XTRP list in TL2D and free up spare TRACKlist board l “Clients” should be kept in stable configuration l D-sized plotter in B0 for printing updated Firmware schematics (.eps or.pdf)

43. D. Saltzberg, 7 Dec 01 L2 Review Suggestions-II l Need more of the “good” jumpers (white) l Make MagicBus document a CDFNOTE l File cabinet for all L2 docs. Can be different sized schematics and also text documents so folders would work better than one binder. l web “clearing house” for all L2 web documents. Good documentation exists for all boards, just need a list of links (Heather is working on this.) I think we should not over- structure this at this point...leave the microstructure to the individual groups l When given choice of testing kludge path vs. real path, try real first

44. D. Saltzberg, 7 Dec 01 L2 Review Suggestions -III l In next 3-6 months, experts (and their supervisors) should think about training their successors. l Need to implement bit-for-bit emulation SIXD--> TL2D into TrigMon l Need someone to write/ implement XFLD-->XTRD emulation l A MB “display” module would be a critical debugging tool (LED’s on each line) much like the old Fastbus display module