1. ABCD DAQ Timothy Phung

2. Topics 1.Introduction 2.Status of Software 3.Overall plan of the software 4.Hardware Setup 5.Threshold Scan 6.Details of how Threshold Scan works 7.Analysis of Data 8.Known problems or issues 9.Comparison of old and new system 10.How we should split up the software development 11.Further things to look into 12.Lessons Learned so far

3. Introduction We are working on a DAQ to read out several modules on a stave The DAQ uses a digital I/O board from National Instruments (the pxi 6561) which sits in a pxi chassis We use labview to program the board to run these tests Labview is also used for processing and analysis of data

4. PXI 6561 Triggers are on the back RSTI We send clock and commands at the DDC connector. Data is also acquired at the DDC connector The interface to the computer is through the PXI bus and through a MXI-4 connection Generation and acquisition memory are independent

5. PXI 6561 cont. There are 16 channels of LVDS input/output We will use 1 channel for output (commands) The other 15 channels will be later used for acquiring data from modules Clocks are also sent from the 6561 to the chips using the onboard clock

6. Hardware Setup PXI chassis with MXI –4 and the 6561 board Power from SC2001 support card and the SCT LV Break out panel from NI

7. Panel Connections Clock goes to chip Generation channel and Acquisition Channels

8. Modules on a Stave readout case Several channels of data coming out One channel for generating data distributed to all modules One clock signal coming in to all modules Data Clock and commands

9. ABCD chip We readout and decode the bit streams of the ABCD chip to find the occupancy of the 128 channels. Based on this we can make measurements of gain, noise, etc., ABCD chip has an amplifier, shaper, and comparator at the front end. We simulate the particle tracks by using the calibration circuit

10. Threshold Scan > Comparator V_threshold V_in If V_in >= V_threshold, the comparator gives a true value If V_in < V_threshold, the comparator gives a false value As V_threshold increases and V_in stays constant, then less channels will respond true V_threshold occupancy An important point is the 50% point VT_50

11. Threshold Scan cont. V_threshold occupancy Increasing charge As charge injected increases, the Vt_50 value increases Charge injected Vt_50 Gain measurement We fit a 2nd order polynomial to the vt-50 values as a function of charge injected. The derivative of the curve at each charge point gives us the gain at that point

12. Status of Software Currently working on Noise occupancy

13. Overall Plan of Software Configure pxi 6561 hardware clocks, channels, resource names, etc Configure generation session, write waveform and scripts Acquisition of Data in a loop Parsing of Raw Data Bit streams from ABCD chip Done in parallel File.xls or.bin file of raw data occupancy as a function of scan parameter and channel number Each test follows the same basic structure

14. Overall Plan of Software cont. Raw data files in.xls or.bin format are read in and analysis is performed We plan to use the reporting feature in labview Analysis Plots, Histograms on Front Panel Reports Raw Data Files

15. Example Program: Threshold Scan Hardware configure Writing waveforms and scripts Data Processing Acquire Data Queue structure allows for execution on multiple threads so both processes can occur in parallel

16. Configure Hardware Configure resource and channels The stream delay is for now placed with a default value of 0.5 * 25ns = 12.5ns The clock is at 40 MHz I export the on board clock to chip through the DDC connector

17. Configuration of Triggers Triggers are sent along the RSTI lines at the back of the 6561. There are 4 triggers Triggers are setup to do multirecord acqusition 1.Ready to Start Event 2.Ready to Advance Event 3.Start Trigger 4.Advance Trigger

18. Scripts Part of the threshold scan script Wait for ready to start event Generate configuration“X”“Y” where X = cal line and Y = threshold value rounded to decimal integer Generate trigger marker0(160) is the command sequence with the level 1 trigger. The marker event starts the multirecord acquisition. Repeat for (N-1) where N = number of (triggers/4) Wait for scripttrigger 1 to get a ready to advance event Generate trigger marker1(160) to advance to the next record. script myScript1 Wait until ScriptTrigger0 Generate config050 Generate trigger marker0(160) Repeat 999 Wait until ScriptTrigger1 Generate trigger marker1(160) end repeat Generate config150 Generate trigger marker0(160) Repeat 1000 Wait until ScriptTrigger1 Generate trigger marker1(160) end repeat Generate config250 Generate trigger marker0(160) Repeat 1000 Wait until ScriptTrigger1 Generate trigger marker1(160) end repeat Generate config350 Generate trigger marker0(160) Repeat 1000

19. Multirecord Acquisition For a threshold scan, we acquire for 4000 records with 3500 samples on each record to roughly account for the maximum data packet length that comes at each threshold step There is one trigger from a level 1 command to start the multirecord acqusition and subsequent level 1 commands are used to advance to the next record. These are exported as marker events in the script.

20. Multirecord Acquisition cont., We setup multirecord acquisition and then fetch records inside of a loop and then place them into a queue. Another loop parses out data after pulling it out of the queue.

21. Processing of data Processing of data occurs by pulling data out of the queue and into a subroutine to do the parsing. The parsing is done by using bit wise operations in labview For now the parsing is able to keep up with the fetching with data, as the queue size stays zero throughout the entire scan. For the case of many modules, the parsing may take longer than the fetching of data and the queue may build up. We should look into faster and more efficient bit stream parsing algorithms

22. Results from this system Threshold scan from 50 to 400 mV for 1.5 fC injected charge Above plots show the occupancy as a function of channel number and threshold The occupancy at 50mV S-Curve for a channel

23. Results continued A strobe delay plot Occupancy as a function of the channel number and strobe delay setting

24. Analysis of Data: Finding the Vt_50 point We have two cases, rising edges and falling edges Currently, I am using the “sparsification threshold measurement” approach to find the vt_50 values The vt_50 values are given below: Plots to the right are for threshold scan and strobe delay

25. Analysis cont. Other analysis routines are just statistical caculations or fits to curves. I use the routines from labview to perform these

26. Speed, Optimization The threshold scan for the case of 50 mV to 450 mV with 1.5fC of injected charge is much slower than the old system. On a fast dual processor Xeon computer, it is slower by a factor of 2. Old system just takes 1 min to do a threshold scan, it takes the new system 2 minutes to do this on a fast computer

27. Speed Bottlenecks Using the profile vis feature it has been identified that the transfer of data from the pxi 6561 to the host pc and the parsing of data are the most time consuming parts of the program

28. Differences between old and new system Old system parses data in hardware, we parse in software Also we wait a fixed amount of time between triggers, old system decreases time between triggers as occupancy decreases MUSTARD VME readout card

29. Relative Rates of Fetching Data and Parsing Data Fetching of data stays the same throughout, since we fetch the same amount of data always Rate Time or increasing threshold or time as test progresses Data Parsing Rate decreases because the parsing of data occurs in a while loop that stops when the trailer bit stream is found. As we move the threshold out this is found sooner since there is less occupancy.

30. Time Between Triggers Compared to old system Time Between Triggers Time or increasing threshold or time as test progresses Old sct test daq the time decreases since there is hardware parsing of data. When it finds the end of the bit stream for one event, it instructs through vme for another trigger to be sent next New system: Time between triggers is the same throughout, it has to wait for 3500 clock cycles and a ready to advance trigger to start on the next trigger

31. Suggested Solution Rate Time or increasing threshold or time as test progresses Data Parsing Rate Current System Fetching Rate Make Fetching Rate decrease by taking data packet size from previous scan and using it as the record size of next scan

32. Suggested Solution cont. Configuration of record sizes and data parsing Boolean T/F Trailer Bit stream not found. Taking Data packet length from previous scan and feed it back as the record size of the next scan

33. Are results consistent between two systems? The values for the gain, input noise, vt50 values, and the extrapolated offset are consistent

34. Continue comparison

35. Style of vis It will would aid the software development and debugging if we could have a consistent style of the vis throughout For example we should have a color code for the vi icons Also consistent names for all the vis This will help when we have many people working on the software

36. A Proposed Color code Blue = Threshold Scan Yellow = 3pt gain Purple = Strobe Delay Etc. All related vis analysis,data processing should be the same background color as well

37. Documentation Right now most vis are documented inside of themselves We should also have some outside documentation Vi to the right is for 3 pt. gain

38. User Interface It has not been decided and should be discussed what the user interface of the control program, reports, plots, histograms should look like A full response curve

39. Version Control Software NI recommends to now use perforce or Microsoft visual source safe I have been using the built in source control, but it has limitations, you cannot rearrange vis into files after you place them into the source code control NI has discontinued its built in source code control in newer versions of labview Version control software will definitely help if more people are going to work on this project

40. How we should split up software Mother vi to control everything Unfinished tests and reports

41. There are lots of additional things to work on Integration with software to control power (Code has already been written by Evan Wulf to control the SCT LV and HV modules) A configuration file format A fast parsing algorithm for the case of many modules in parallel Planning of clusters to pass data around instead of passing 20 controls or indicators to subvis We still also need to cleanup the code, document it, and optimize it as well

42. Things to look into further Embedded controller on the pxi chassis to do the parsing FPGA module from NI to do the parsing Streaming the data from an entire test to the disk and parsing out streamed data The speed of the system for the case of many modules on a stave Whether building the program into an executable and turning off all debugging increases speed a lot What the maximum data packet size for the ABCD chip is. I assumed a value of 3500 samples for 4 chips using a rough estimate.

43. Brief Overview on Streaming vs. Multirecord Multiple record acquisition allows for setting up complex trigger sequences on the pxi 6561. In streaming you just have one trigger to start and another to end. The idea behind streaming is to read in data from the 6561 to the host pc at faster rates than it comes into the 6561 from the device under test. For us this means we need to read in data at faster than 80 MB/s for 2 bytes * 40 MHz. Streaming is highly dependent upon the computer system. I did not look into using streaming in detail yet, because we are more interested in getting a working system for now.

44. Streaming cont. Streaming data is very memory intensive on the computer. If we operate a test for 1 min we will have 4800MB or 4.8GB of data on the computer. 80MB/s * 60s. Also, we have to write out the entire script before we are beginning and makes sure that we wait sufficient time between all trigger sequences before sending the next trigger sequence. There is a lot of data to parse, we have to sift through a lot of data after it is streamed in. A lot of it is useless data Streaming requires less overhead than multirecord It will be hard to implement such tests as noise occupancy with streaming, where we have to decode some data before we send the next commands. For tests where the decoding of data and the next set of commands are not dependent on each other this may be a possible alternative. One should also use the benchmark vi before using streaming. Faster pcs usually are able to stream better. It is also dependent upon what other things are on the pci bus, since devices on the pci bus all share the same resources.

45. Water Analogy of Multirecord and Streaming Data from the ABCD chip is like a bucket of dirty water Multirecord Streaming 6561 storage container Host pc With multirecord acquisition it is like you turn on and off (based on a trigger) the faucet of dirty water coming in to the 6561 with a filter in front. Refined Data comes into the 6561 and host pc. Less parsing to do in the pc With streaming, you turn on the faucet once, and leave it on with no filter in front, all the dirty water comes in and must be filtered in the pc. This is much faster, but the parsing at the end may be slower. Also need a large hard drive or else host pc will overflow. Filter Large Filter Green is the dirty water, blue is cleaner, light blue is the cleanest

46. Lessons learned so far Placing data into queues to process in a producer/consumer loop increases performance slightly by putting the two processes on separate threads There is some overhead into going into subvis. For example if a subvi resides in a loop, it is better to place the loop in the subvi. In cases where there is not possible, turn the execution priority of the subvi to subroutine to minimize the calling time to the subvi Making the vis reentrant also speeds up execution a little because multiple data sets can enter into the same subvi and execute along different threads. The stream delay setting on the PXI –6561 does not allow for setting the stream delay to be 0.2 to 0.3 * 25ns or 0.7 to 0.8 * 25ns See NI Application Note 168 Labview Performance and Memory Management for more details on optimization

47. Other useful references from NI Application Note 199 LabVIEW™ and Hyper- Threading Application Note 114 Using LabVIEW to Create Multithreaded VIs for Maximum Performance and Reliability LabVIEW Development guidelines describes application of software engineering principles to development of large labview applications Lots of references on NI website as well.

48. An iterative software development lifecycle Developing the software in iterations looks like a good idea at this point. We are on iteration 1 at this point 1.Version 0.1 (iteration 1) get a complete working system going without worrying about all the small details and optimize later, solution is not the most elegant at this point. We are more or less learning about how building this application in labview works, how to communicate with the abcd chips to perform this test, and how to analyze data 2.(iteration 2) we will start to optimize the application and make the solution look cleaner and more modular

49. Working Session Results Suggestions after meeting with NI engineer and our own discussion How software will be split up Things we can do to make the development uniform

50. Suggestions We spoke to an engineer from NI (Matt Thompson). We also came up with a list of ideas talking amongst ourselves about how to increase performance. Two main suggestions categories Immediate suggestions Extra Hardware suggestions

51. Immediate suggestions 1.Prioritize loops 2.Make the record size smaller and smaller (presented earlier) for threshold scans. 3.Streaming Prioritizing loops means making the code that is used to fetch data from the pxi 6561 run at a higher priority than other parts of the code. Prioritizing loops can be quickly and easily implemented.

52. Immediate suggestions (cont.) Adjusting the record size appears also to be able to give large performance gains. This has to be implemented in the code, and we must code it so that it can recover from the case where the data packet size does not monotonically decrease. Matt said that streaming of data would probably not improve performance much

53. Extra Hardware suggestions 1.Faster Host Computer 2.Embedded controller w/ Windows 3.Embedded controller w/ RTOS 4.PCIe MXI Remote Controller 5.FPGA Card from NI 6.New NI Products PXIe NIHSDIO card HSDIO card with a FPGA to do onboard signal processing

54. Extra Hardware Suggestions cont. Faster Host Computer definitely helps. Matt said that an embedded controller w/ Windows won’t help much. The MXI connections make the host pc think that there are more pci slots, so basically the pxi chassis and the host pc can all be thought to be a single entity. An embedded controller w/ Real time operating systems may be faster because the computer is dedicated to doing one task. It doesn’t run the windows background tasks.

55. Extra Hardware Suggestions cont. PCIe MXI express connection might be faster (BNL already has this setup). We have a PCI connection at LBL. FPGA card from NI (This can be very fast and also scalable for the case of more than 15 modules). It can have up to 160 digital I/O lines and operate up to 40 MHz(this is frequency that are clock is operating at). One problem is that it uses TTL inputs. We would need LVDS/TTL converters. We also would have to look how commands sent to the chip would work.

56. Extra Hardware Suggestions cont. New NI Products They have a pci express nihsdio card (it operates up to 50 MHz), but uses ttl inputs Matt said that he will look into other products as well. They are going to later have a hsdio card with a fpga on board to do onboard signal processing.

57. Software Split-up I will continue on analog tests Hucheng will work on digital tests Andrew said they are beginning their setup and will mimic the setup at BNL.

58. Cont. Analysis is a major component. This can take a long time to get all the analysis that was performed in Peter Phillips code. We do not feel that we are ready to discuss user interface, and plots at this point. Hucheng will also look into version control with Perforce.

59. Agreements We should all use a consistent version of labview. The latest version 8.20 was recommended because this will optimize the application more for multithreaded applications executing on multicore processors. Labview 8.20 also integrates object oriented programming (this may make our code cleaner and more modular). We should look into this further.