Status of Tev IPMs September 3, 2008 Andreas Jansson

Status (short version) The Tev IPMs can be controlled from and deliver data to ACNET in much the same way as the flying wires The vertical system is operational since a long time. –Old version of the firmware/software. –Occasional data synchronization problem (not a problem for manual measurements) The horizontal system is currently at FCC the help debugging sync issues. 9/3/2008IPM status A. Jansson 2

ACNET control Since some time the Tevatron IPMs are state driven in much the same way as flying wires. Selected output is logged when data ready. 9/3/2008IPM status A. Jansson 3

Tom’s ACNET interface 9/3/2008IPM status A. Jansson 4

PSPEC settings 9/3/2008IPM status A. Jansson 5

Datalogged averaged data 9/3/2008IPM status A. Jansson 6 PSPEC 5 June 19 During store

Sync issues Spontaneous front end resets (early on) –Revert to default mode and loss of synchronization. –Due to Single Event Upsets (or just noise on slow control channel?) –Fixed with new front end firmware version plus Labview software changes Trigger/Counter synchronization (affects both systems) –Links synchronized, but bunch data not in the correct time-bins. –Due to “missing” AA markers –Fixed by having both boards treat missing markers in same way. Intra-board synchronization (affects both systems) –Loss of SERDES lock or illegal character on serial link. –Due to “unexplained” PCI accesses –Effect minimized by re-sync logic. Errors during acquisition flagged. Inter-board synchronization (affects mainly horizontal) –Data from two buffer boards not aligned (varying from measurement to measurement) affecting bunch profiles and background subtraction. –Was hard to debug in the presence of other sync issues… –Still working on this one. Expect new firmware in about a week. 9/3/2008IPM status A. Jansson 7

Status Long version

Example: Synchronized 9/3/2008IPM status A. Jansson 9

Example: Unsynchronized 9/3/2008IPM status A. Jansson 10 Easy to detect and reject by eye for manual data-taking!

One manifestation of sync issues 9/3/2008IPM status A. Jansson 11 Histogram of proton counter in first buffer data frame. Expect single cluster of three bins around zero. Saw wider cluster + scatter + unexpected peaks

IPM timing 9/3/2008IPM status A. Jansson 12 master slave Timing boardFront-end boardsBuffer boardsLabView TVBS APTVBS

System Debugging FE board data streams have embedded timing and status information. Due to limitations in the memory bandwidth, most error checking is done in buffer board firmware –Debugging mode writing raw data from some links available to debug front end boards. Firmware quickly grew to become resource limited, making debugging iterations slow. –Painfully slow “place and routes” –Reduced version required to fit chipscope 9/3/2008IPM status A. Jansson 13

Known buffer board issues The buffer board was designed according to “best practices” for high speed signals –No hardware design issues found to date. However, we discovered relatively early on that PCI access during an acquisition would sometimes glitch one or more of the high- speed data links. –Appears to be a FPGA internal problem –Work-around: re-order setup sequence to limit access to board, use interrupt instead of polling. –Seemed to work fine. 9/3/2008IPM status A. Jansson 14

Board layout 9/3/2008IPM status A. Jansson 15

Sync saga cont’d Even without PCI access during acquisitions, loss of synchronization still occurred relatively frequently, particularly for injection measurements (long wait time) –Improved the initial synchronization logic –Added firmware logic to allow links to resynchronize dynamically if needed, while waiting for a trigger (In original implementation, the entire chain was synchronized once by a timing card reset just before an acquisition) 9/3/2008IPM status A. Jansson 16

Sync saga cont’d Early resync logic worked in test stand, but not in the Tevatron systems. At this point, FY08 omnibus budget came out, furloughs started and we lost Kwame (our FE board engineer) to sunny Florida. Rick (Buffer board engineer) took over debugging of both boards, and we moved the test stand to FCC (closer and real TVBS available). –Real TVBS available (used simulated before) 9/3/2008IPM status A. Jansson 17

Sync saga cont’d Found occasionally missing AA markers, and that FE board and buffer board treated this unspecified case differently. –Fixed with buffer firmware modification. Also discovered “unexplained” PCI accesses during acquisition? –Don’t know where they come from, or how long they have been there (system patch?). –Tried to have board ignore PCI requests during acquisition, but this caused total PC lock-up. –With improved re-sync logic, error rate in test stand is now a few per mil. 9/3/2008IPM status A. Jansson 18

Status Intra-board sync fixed to the extent possible (residual rate is per mil level). Last couple of “full firmware versions” tested OK in test stand but crashed the horizontal IPM host PC. –Hardware difference? –Moved horizontal systems to FCC for easier debugging. –Likely related to new inter-board sync logic that was not yet fully tested (In test stand only timeouts were observed). 9/3/2008IPM status A. Jansson 19

Summary Found and fixed the sync issue between the front-end & buffer board Traced remaining sync problem to occasional non-Labview PCI access –With new (intra-board) re-sync logic, error rate is few per mil. Further reduction would require new board layout. Investigating occasional timeout errors –Likely related to inter-board sync logic that was put in some time ago but never fully tested. –Seems benign in test stand, but causes the horizontal system to crash. 9/3/2008IPM status A. Jansson 20

Forecast With the intra-board sync issues fixed, work is proceeding on the inter-board sync logic Expect to have a new firmware version ready for testing in about a week. 9/3/2008IPM status A. Jansson 21