1. Tevatron IPM status Oct 2009 Andreas Jansson K.Bowie, T.Fitzpatrick, R.Kwarciany, C.Lundberg, D.Slimmer, L.Valerio, J.Zagel Also thanks to: T.Anderson, M.Bowden, A.Bross, A. Chen, R.Dysert, S.McCormick, S. Suleimani, H.Nguyen, C.Rivetta, H.Glass, D.Harding, B.Hively, V. Kashikin, D.Miller, Z.Tang, J.Volk, T.Zimmermann, …

2. Outline Brief overview history of Tev IPMs –Summary of Review talk from 2007 –Long version available at: http://indico.fnal.gov/conferenceDisplay.py?confId=1272 What happened then? –Progress reports from 2007 to now –Summary of partial reports given at Tev Dept & Vaia’s meetings Current status 10/15/2009IPM status October 2009

3. DESIGN Brief overview of IPM history 10/15/2009IPM status October 2009

4. 9/21/07IPM Review, September 21, 2007 A. Jansson 4 Goal GOAL: Measure protons and pbar beam size turn by turn at injection and ramp during normal operation to diagnose and mitigate emittance blow-up.

5. 9/21/07IPM Review, September 21, 2007 A. Jansson 5 Challenges in the Tevatron Two small beams separated by helix. Separate protons from pbars, injected from circulating beam Beam induced parasitic signals. Low vacuum pressure Fine granularity and many channels Single bunch resolution and gating Improved shielding and matched cables Local pressure bump with controlled leak Solutions: Challenge:

6. 9/21/07IPM Review, September 21, 2007 A. Jansson 6 IPM detector 0.2T MCP 1kV 100V 0V 9.9kV 10kV All signal cables are enclosed in a Faraday cage!

7. 9/21/07IPM Review, September 21, 2007 A. Jansson 7 Anode board and internal cabling ¼ mm strip pitch 200 channels (128 instrumented) Board mounted series resistor for back- termination and LP filtering. In-vacuum signal cabling using UHV-compatible flex-circuits High resolution area can be moved by swapping connectors

8. 9/21/07IPM Review, September 21, 2007 A. Jansson 8 Microchannel plate Max gain with 36 proton bunches is ~1e4 to avoid saturation. Can be achieved with single plate With dual plates, each plate would run at a very low gain and low bias current. Use single MCP with extra-high bias current.

9. Front end cabling 40 channels in each vacuum DB50 feed- thru (flex circuit) Transition board from feed-thru. Mini-coax bundles with external shield. Custom (CMS) DB coax connector in FE board. 9/21/07IPM Review, September 21, 2007 A. Jansson 9

10. 9/21/07IPM Review, September 21, 2007 A. Jansson 10 DAQ system CMS-QIE chip digitizes signal in tunnel. Serial data uplink on optical fiber. Receiver and data buffer in upstairs PC Timing + QIE clock + QIE clock supplied from PC thru cat-5E cable Timing card (PCI) Timing fanout QIE cards (16x 8 ch) Data Buffer (2x 8 links) (PCI) Host PC (LabView) Service building Tunnel

11. 9/21/07IPM Review, September 21, 2007 A. Jansson 11 QIE card 8 channels (CMS QIE8) per board. Achieved noise ~1.8fC with 4’ cable. Data is combined with timing information, serialized by CERN GOL ASIC (rad hard) and sent thru optical fiber at 1.1Gbps data rate Timing fanout board cleans up and distributes clock and timing signals

12. FE board input/output format 9/21/07IPM Review, September 21, 2007 A. Jansson 12 76543210 QIE0 1514131211109823222120191817163130292827262524 QIE1QIE2QIE3 QIE5QIE6QIE7 QIE4 CAPID0 Timing bits TX_EN/RX_DV QIE0QIE1QIE2QIE3 QIE5QIE6QIE7 QIE4 CAPID1 Timing bits QIE0QIE1QIE2QIE3 QIE5QIE6QIE7 QIE4 CAPID2 Timing bits Proton Revolution CounterError bits QIE mode 2RF/7 Clock Counter 11000No effect 11001Proton Injection Marker 11010Antiproton Injection Marker 11011QIE Mode0 (QIE in Calibration Mode) 11100QIE Mode1 (QIE in Mode) 11101QIE Mode2 (Link test - outputs a 32-bit counter to GOL) 11110QIE Mode3 (Reset QIEs) 11111Board Reset 15MHz clock frequency Proton and pbar revolution markers Encoded signal: INPUT (Cat5 cable) OUTPUT (1.6Gbps serial link)

13. 9/21/07IPM Review, September 21, 2007 A. Jansson 13 Data buffer card Handles 8 incoming optical links (64 channels, 1.1 GB/s of data) Can sparsify data on–the- fly based on timing masks 512MB RAM allows for –20.000 turns of continuous data –90.000 turns for 72 bunches –6 million turns for a single bunch Read out thru PCI64 bus. Two boards are used to handle 128 channels. Sync via jumper cable IPM buffer board doubled as prototype for BTeV L1 data buffer. Considered for use in MICE experiment.

14. 9/21/07IPM Review, September 21, 2007 A. Jansson 14 Timing card Produces the 15MHz (2/7 RF) QIE clock Decodes and transmits beamsync revolution marker + injection and trigger events Controls QIE settings. 396 ns (21 buckets) p-pbar separation ~120ns RF 2/7

15. 9/21/07IPM Review, September 21, 2007 A. Jansson 15 Labview Software

16. Installation timeline Magnets installed shutdown ‘04 Vertical detector installed Dec’05 –Arcing problems inhibited operation at full B field. High outgassing from retrofit retainer. Vertical detector fixed, horizontal installed May’06 –reduced DAQ system with single buffer board due to power supply limits and firmware problems). Full DAQ, larger leak installed Oct07 –80 channels per plane, 2 buffer boards 10/15/2009IPM status October 2009

17. 9/21/07IPM Review, September 21, 2007 A. Jansson 17 System Status 2007 Both IPMs had ‘new’ software and firmware. Vertical IPM works pretty well, basicall since installtion. Horizontal IPM has some hardware problems (bad FE baord) that required an access. Horz IPM also requires two buffer boards to measure beam size at injection NB. REAL DATA!

18. 9/21/07IPM Review, September 21, 2007 A. Jansson 18 Beam size at IPM location By moving detector, able to fit beam in active area, except horizontal beam size at 150GeV

19. EARLY MEASUREMENTS Brief overview of IPM history 10/15/2009IPM status October 2009

20. 9/21/07IPM Review, September 21, 2007 A. Jansson 20 Protons at low beta Proton bunch #1 at low beta during store #4758. Measured beam size 0.55mm, turn- by-turn variation (noise) 20µm. Total signal per bunch ~1.3pC. 30 turn average single turn Magnet at 200A

21. 9/21/07IPM Review, September 21, 2007 A. Jansson 21 Injection – Full B field Proton bunch #21 turn-by-turn 1 cm Store #4772Magnet at 200A RMS profile width 0.9 1.0 1.1 1.2 1.3 [mm] 0.00.51.0 01020 FFT 0 35 turns

22. 9/21/07IPM Review, September 21, 2007 A. Jansson 22 100 turns Four injection measurements superimposed

23. 9/21/07IPM Review, September 21, 2007 A. Jansson 23 Measured mismatch vectors Nominal settings Q701 at -22A Q701 at -42A Q711 at -25A Need to increase Q711 by ~15A (saturation?) Should result in 5-10% smaller vertical emittance Should have very little effect on horizontal plane. Should have been tested this morning…

24. 9/21/07IPM Review, September 21, 2007 A. Jansson 24 Flying wire comparison Known electronics problem (cross talk from revolution marker pulse) Comparison of vertical beam size from IPM and nearby Flying Wire. Tuning of abort gap cleaner timing had caused blow-up of certain bunches. From MAD lattice file, expect a 13% wider beam at Flying Wire. See ~1%.

25. 9/21/07IPM Review, September 21, 2007 A. Jansson 25 Pbars at 980 Gev Store #5172, Jan 07 HorizontalVertical

26. 9/21/07IPM Review, September 21, 2007 A. Jansson 26 Pbars at 150 GeV Also saw coasting pbars (with vertical system) at injection Requires moving the detector to pbar orbit. Triggering did not work at first attempt (timing event found to be disabled). Fri Jan 12 2007, shot #5175

27. 9/21/07IPM Review, September 21, 2007 A. Jansson 27 Shot 5186 – vertical plane Two consecutive beam revolutions in Tevatron, just at injection of batch 9.

28. 9/21/07IPM Review, September 21, 2007 A. Jansson 28 Vertical pbar injection data (batch 9) Beam position gives consistent oscillation ~0.2mm. Need to look closer at beam size data, but… There is a clear frequency component at twice the tune! 2.4 3.2 2.8 0.3 0.6 0.4 0.5 Beam size [mm] Beam position [mm] 100 0

29. ISSUES Brief overview of IPM history 10/15/2009IPM status October 2009

30. 9/21/07IPM Review, September 21, 2007 A. Jansson 30 Parasitic signals Despite very good shielding, beam related signals still present with MCP gain off. Very stable from turn to turn -> reference subtraction possible Present even on “broken” channels (where no ionization electrons are seen) 1x1 store at 980GeV ~center anode strip Timing markers Beam pulse

31. Buffer board sync With two boards, capacitive background appeared unaligned, and background subtraction didn’t work well –Found and fixed software data alignment bug –Verified that buffer board sync check works properly 9/21/07IPM Review, September 21, 2007 A. Jansson 31

32. PCI card poll glitching SERDES Polling card via PCI during measurement was found to glitch the SERDES reciever. “Solved” by using interrupt instead of polling, and reorganising code to minimize PCI traffic during acquisition. 9/21/07IPM Review, September 21, 2007 A. Jansson 32

33. INTEGRATION What happened more recently? 10/15/2009IPM status October 2009

34. IPM Review 2007 IPM Review Committee Report The charge to the committee has consists of the following two items: 1. What is the current status of the Tevatron IPM systems - the front-end electronics and DAQ in particular? Identify any known problems or issues. 2. Can the IPM system be made fully operational - complete detector readout with production electronics and DAQ issues resolved - by Jan 1, 2008? What additional resources ($$, effort) would be needed to meet that date? The committee was positively impressed with quality of the work and the part of the presentation devoted to the item 1. Nevertheless very little was said about the second half of the charge. It is unclear: What does it mean “to make IPM system fully operational?” Which additional resources, if any, are required to finish the project timely and which additional expenses are required? Taking the above into account the Committee suggests to convene an additional short session devoted to discussion of item 2 of the charge. Andreas should settle with the Tevatron department what they consider for the project to be operational and to present a clear answer to the part 2 of the charge. Having this session in about one or two weeks should leave enough time to elaborate details. 10/15/2009IPM status October 2009

35. Tev dept wish list In General the IPM controls and data retrieval should be very similar to the flying wire front end. It should be able to perform the following: –Decode V:CLDRST state and setup the instrument based on a state to instrument spec mapping owned by the IPM front end. The TFW instrument has an example of this. –Should provide plane and particle sigma data on event $75 (profile timer) There are 35 profiles that currently go out up the ramp and through the lowbeta squeeze. –Should provide data to SDA. The instrument should own a “data ready” device similar to TFW that communicates the state of pending data. This can also be used for lumber jack Client Logging. –The instrument should be free running providing data during a collider store at ~ 1/30 min. (similar to TFW). –The instrument should write bunch by bunch data into ACNET array devices following the standard convention.. The ACNET array devices that hold bunch by bunch information should be 37 elements long. The enumeration of array devices should have the [0] element of the array as the average value of the 36 array members to follow. There may have to be some special code to handle updating the average to reflect and average of elements with non zero values. The [1 – 36] would store the data where bunch 1 would correlate to [1] of the ACNET device respectively. –ACNET parameter representing data and setup should be placed in the lumberjack utilizing Client Logging. –The IPM should have an ACNET application that can perform the following functions (similar to T46 for TFW): Standard user interface. View and Configuration of State to IPM Spec Transition Map for pre-loading specs. View and Configuration of IPM specs. Ability to access and plot profile data correlated to data taken on profile event and periodic sampling with capability to have fitting options for plots. –IPM ACNET devices should attempt to follow a naming convention. 10/15/2009IPM status October 2009 Tevatron IPM Specs for Collider II Operation General Collider II IPM Specifications:

36. ACNET control Since early 2008 the Tevatron IPMs are state driven in much the same way as flying wires. Selected output is logged when data ready since mid- January 2008 9/3/2008IPM status A. Jansson 36

37. Tom’s ACNET interface 9/3/2008IPM status A. Jansson 37 Available summer 2008 (no plotting)

38. PSPEC settings 9/3/2008IPM status A. Jansson 38

39. Pbar jacking 10/15/2009IPM status October 2009 Jan 31, 2008 Store #3874

40. Pbar jacking 2 10/15/2009IPM status October 2009 Feb 1, 2008 Store #3876

41. IPM–FW comparison 9/3/2008IPM status A. Jansson 41 Jan 2008

42. How to improve data quality? Fluctuations from measurement to measurement much bigger than turn- by-turn fluctuations for coasting beam. One known source of bad data: synchronization problems! 10/15/2009IPM status October 2009

43. Example: Synchronized 9/3/2008IPM status A. Jansson 43

44. Example: Unsynchronized 9/3/2008IPM status A. Jansson 44 Easy to detect and reject by eye for manual data-taking!

45. One manifestation of sync issues 9/3/2008IPM status A. Jansson 45 Histogram of proton counter in first buffer data frame. Expect single cluster of three bins around zero. Saw wider cluster + scatter + unexpected peaks

46. Sync saga We discovered relatively early on that PCI access during an acquisition would sometimes glitch one or more of the high-speed data links. –Appears to be a FPGA internal problem, now believe it is caused by does not like 5V input signals on pins close to high speed inputs. –Work-around: re-order setup sequence to limit access to board, use interrupt instead of polling. –Seemed to work fine. 9/3/2008IPM status A. Jansson 46

47. Board layout 9/3/2008IPM status A. Jansson 47

48. Sync saga cont’d Even without PCI access during acquisitions, loss of synchronization still occurred relatively frequently, particularly for injection measurements (long wait time) –Improved the initial synchronization logic –Added firmware logic to allow links to resynchronize dynamically if needed, while waiting for a trigger (In original implementation, the entire chain was synchronized once by a timing card reset just before an acquisition) 9/3/2008IPM status A. Jansson 48

49. Sync saga cont’d Early resync logic worked in test stand, but not in the Tevatron systems. Used simulated TVBS in lab5 teststand. At this point, FY08 omnibus budget came out, furloughs started and we lost Kwame (our FE board engineer) to Florida. Rick (Buffer board engineer) took over debugging of both boards, and we moved the test stand to FCC (closer and real TVBS available). –Real TVBS available 9/3/2008IPM status A. Jansson 49

50. Sync saga cont’d Found occasionally missing AA markers, and that FE board and buffer board treated this unspecified case differently. –Fixed with buffer firmware modification. Also discovered “unexplained” PCI accesses during acquisition? –Don’t know where they come from, or how long they have been there (system patch?). –Tried to have board ignore PCI requests during acquisition, but this caused total PC lock-up. 9/3/2008IPM status A. Jansson 50

51. Status ~sept ‘08 Improved re-sync logic got residual error rate down to per mil level (in test stand). Couple of “full firmware versions” tested OK in test stand but crashed the horizontal IPM host PC. Moved horizontal system to FCC for easier debugging. –Source of problem related to new inter-board sync logic –Found bad power supply in horizontal system Swapped test stand PC with Horz system 9/3/2008IPM status A. Jansson 51

52. Summary of sync issues Spontaneous front end resets (early on) –Revert to default mode and loss of synchronization. –Due to Single Event Upsets (or just noise on slow control channel?) –Fixed with new front end firmware version plus Labview software changes Trigger/Counter synchronization (affects both systems) –Links synchronized, but bunch data not in the correct time-bins. –Due to “missing” AA markers –Fixed by having both boards treat missing markers in same way. Intra-board synchronization (affects both systems) –Loss of SERDES lock or illegal character on serial link. –Due to “unexplained” PCI accesses –Effect minimized by re-sync logic. Errors during acquisition flagged. Inter-board synchronization (affects mainly horizontal) –Data from two buffer boards not aligned (varying from measurement to measurement) affecting bunch profiles and background subtraction. –Was hard to debug in the presence of other sync issues… –Fixed with a combination of firmware and software changes 9/3/2008IPM status A. Jansson 52

53. LabView software update Changed software to not issue “data ready” when a acquisition problem is detected –Only data without hardware errors datalogged. Fixed ”total signal charge” readback, and chi-square readback. 9/3/2008IPM status A. Jansson 53 Nov’08

54. Observations with “new” firmware The beam related background, which used to be significant but very stable, is now fluctuating a lot in horizontal plane. –Likely due to some sort of timing jitter Also, the signals appear to be smaller than before –Possibly due to new x-collimator ion pump? 9/3/2008IPM status A. Jansson 54

55. Beam background 9/3/2008IPM status A. Jansson 55 “Average beam background” “Fluctuation in beam background” VIPM and HIMP both looking at the vertical signal

56. Various failed attempts to fix jitter Swapped timing cards Changed timing card slot location (possible interference with buffer board) 9/3/2008IPM status A. Jansson 56

57. Status – Jan’09 Observed jitter in horz clock/timing signals –Not reproducible in test stand –Swapping timing boards back and forth gave inconclusive results. –Possibly related to fluctuating signal? In the meantime found and fixed bug in background subtraction for pbars. –Pbar data much improved 9/3/2008IPM status A. Jansson 57 Jan’09

58. February power failure After the power outage on February 11, IPMs stopped reporting data to ACNET –Software detected persistent buffer errors bits, marking data as bad VIPM: Buffer 1/2 mismatch HIPM: Pbar marker error Swapped cables, to locate error –Buffer mismatch followed upstairs PC –Pbar error followed downstairs crate. Pulled buffer boards from vertical system for bench test. –Left horizontal PC monitoring vertical signals Bench tested boards, finding nothing out of the ordinary –Upgraded test stand labview code and re-tested, saw nothing Reinstalled, buffer mismatch problem gone –Had to distable signals from horizontal front end board 1, which was causing the pbar marker error. Mechanics now seems to be working properly 9/3/2008IPM status A. Jansson 58

59. Signal strength issue Signals are smaller than they used to be, fluctuating a lot and occasionally disappear altogether –Not due to stochastics of gas interaction –HV rise time ruled out as a culprit(?) –Gas pressure fluctuating? –Not currently understood 9/3/2008IPM status A. Jansson 59 April’09

60. Fluctuating data 9/3/2008IPM status A. Jansson 60

61. OK data 9/3/2008IPM status A. Jansson 61

62. Marginal data 9/3/2008IPM status A. Jansson 62

63. Bad data 9/3/2008IPM status A. Jansson 63

64. Before shutdown ’09 Tried turning off some ion pumps –Signals strength improved somewhat –Signal fluctuations still there Took vertical tbt data at injection –No obvious mismatch Found almost 1V ripple on 3.3V supply to HIPM timing card –Replaced power supply –Timing jitter now essentially gone 9/3/2008IPM status A. Jansson 64

65. Signals just before shutdown 10/15/2009IPM status October 2009

66. HIPM at injection 9/3/2008IPM status A. Jansson 66

67. Improved error detection 10/15/2009IPM status October 2009 During shutdown, improved the buffer error Detection to reduce false positives Error rates in machine (without beam) consistent with test stand with few exceptions (weak boards). Swapped two boards.

68. STATUS Current 10/15/2009IPM status October 2009

69. Data availability While we have been working on improving the measurement reproducibility, data from ramp, squeeze and store have been available in ACNET and logged since beginning of 2008. Injection and “pbar jacking” data taking have been enabled occasionally –Not enabled permanently due to concerns about depleting MCP if stuck with high voltage on for long times. 10/15/2009IPM status October 2009

70. Hardware, firmware and software On-the-fly re-sync logic seems to work Intra-board sync seems to work Improved the error detection, minimizing false positives Small residual error rate (due to SERDES glitches) remain. Any flawed data is discarded 10/15/2009IPM status October 2009

71. Signal Quality Signal is weaker than before, in particular for horizontal system –Horizontal pbar signal no longer visible –Leak no longer as effective? Signal level is fluctuating significantly, causing fluctuation in measured beam sigma. –Gas puffs from pumps? Noise level also seem somewhat higher, at least in some channels 10/15/2009IPM status October 2009

72. E0 gas pressure 10/15/2009IPM status October 2009 Last 4 years TSPs were fired during the shutdown Degassing TSP filaments at IPMs (spoiling the TSP) Helped bring the signal back up somewhat

73. TSP degassing test 10/15/2009IPM status October 2009 Trying to raise local pressure by degassing TSP filaments gave little or no effect on the measured signal (might have worked better if pumps were off)

74. Horz signal then and now 10/15/2009IPM status October 2009 2007, before shutdown and now

75. IPM vs FW 9/3/2008IPM status Store #7243 Oct 10, 2009 Consistent to 10% with FW, but weak IPM signals

76. Turn-by-turn data 10/15/2009IPM status October 2009 Enabled injection measurement for store #7265 Got data! No apparent vertical mismatch (horizontal signal too weak) Some trigger issues (only a few bunches captured, rest of data stale) To be looked into further.

77. Summary Tev IPMs are delivering data since almost two years Compares quite well to FW, but some fluctuations in the sigma readings. All known hardware issues have been resolved, data affected by any remaining occasional hardware glitches discarded. Remaining issues related to weaker signal with larger fluctuations than before. Also seems there are some triggering issues for injection measurement (not used regularly) 10/15/2009IPM status October 2009