1. Stefano Magni I.N.F.N Milano Italy - NSS 2003 Pomone, a PCI based data acquisition system Authors: G.Alimonti 3, G. Chiodini 1, B. Hall 2, S. Magni 3, D.Menasce 3, L. Uplegger 3, D. Zhang 2 1 I.N.F.N. Lecce, 2 FNAL, 3 I.N.F.N. Milano

2. Stefano Magni I.N.F.N Milano Italy - NSS 2003 Project goals Validation of the benefits of an Object Oriented design in a small scale project in the BTeV collaboration Development of a general purpose DAQ system based on PCI technology suited for test beams and laboratory measurements Test of a pixel detectors DAQ in untriggered mode and of an on-line, time-stamp based, event reconstruction algorithm

3. Stefano Magni I.N.F.N Milano Italy - NSS 2003 A PCI Test Adapter (PTA) plug-in card,compliant with the PCI protocol Pomone has been designed to meet the requirements of a general Test Stand hardware for testing detectors both in a laboratory environment and at test beam facilities for the BTeV experiment at Fermilab. Current implementation focuses on following hardware components: A Programmable Mezzanine Card (PMC) An FPIX read-out chip (ROC) External data source subsystem, a Fermilab Pixel readout chip (FPIX) The PMC is intended to work in conjunction with a PTA and serves as an hardware interface between the the PTA card and the external detector. PCI-compliant card featuring: a programmable Altera FPGA controlling all functions two RAM banks (1 Mb each) Schematics of the PMC card Schematics of the PTA card Hardware components DAQ computer Esternal subsystem

4. Stefano Magni I.N.F.N Milano Italy - NSS 2003 PTA working mechanism ● A data driven DAQ: data are produced by an external subsystem (PMC+ROC) at a variable rate while the host computer receives them at its own rate (depending upon the CPU clock and the processor current activity). ● The PTA FPGA has been programmed to get data from the external subsystem and to act as an intermediate rate-balancing buffer. This is accomplished by a periodic switching of the data flow between two RAM banks (details later) ● The architecture has been designed to allow for a continuous, lossless sustained data-rate

5. Stefano Magni I.N.F.N Milano Italy - NSS 2003 Software design guidelines Components of the system should be loosely coupled: this allows for upgrades and changes of individual elements with little effort. It further insures smoother deployment due to minimal cross dependencies. Code must be robust, in principle able to whitstand change of operating system environment, extensive refurbishing and additions of algorithms. Code must be highly modular, able to accommodate different detectors, with different hardware and software specifications. Since the incoming data rate is asynchronous from the host computer clock, a data-rate compensation buffer must be provided to accommodate for fluctuations in the data rate. Functionality of the code must be guaranteed also in environments with minimal resources (eg. no X11graphics is available). The system should be able to perform even without a GUI for user interaction.

6. Stefano Magni I.N.F.N Milano Italy - NSS 2003... solutions Communications between processes has been implemented througth a lightware message bus, based on system V message queues To allow for robustness and modularity the object oriented paradigm has been chosen. Particular care has been exercized in the design phase in order to efficiently achieve an optimal decoupling of components. Where possible, the system uses already existing (open source) packages or component libraries. Don’t reinvent the wheel ! In order to accomodate an intermediate data-rate compensation buffer, the system is split in two main processes with a shared memory in the middle: the first process (called producer) gets data out of the PTA card memories placing them in the shared memory, while the second (called consumer) continuously browses the shared memory to fetch completed blocks for the event-builder to assemble hits into events. Events are finally written by the consumer on an external data-storage. System logging is performed by means of a centralized logging utility, collecting messages from different cooperating DAQ processes

7. Stefano Magni I.N.F.N Milano Italy - NSS 2003 Producer DAQ objects collaboration PTA card Hardware Software Circular buffer Event builder Event buffer Message Bus Controller Logger Writer System monitor PTA card PCI Bus

8. Stefano Magni I.N.F.N Milano Italy - NSS 2003 Read-out process DAQ time evolution (0/15) Basic mechanism of operation of the PCI card and the read-out: Bank0Bank1 FPGA 0 Interrupt handler Reset interrupt Shared memory Disk stager Consumer Process A Consumer Process B

9. Stefano Magni I.N.F.N Milano Italy - NSS 2003 DAQ time evolution(1/15) Basic mechanism of operation of the PCI card and the read-out: Bank0 FPGA Time 0 Interrupt handler Bank1 Read-out process Shared memory Reset interrupt Disk stager Consumer Process A Consumer Process B

10. Stefano Magni I.N.F.N Milano Italy - NSS 2003 DAQ time evolution(2/15) Basic mechanism of operation of the PCI card and the read-out: Bank0 FPGA Time 0 Interrupt handler Bank1 Read-out process Shared memory Reset interrupt Disk stager Consumer Process A Consumer Process B

11. Stefano Magni I.N.F.N Milano Italy - NSS 2003 DAQ time evolution(3/15) Basic mechanism of operation of the PCI card and the read-out: Bank0 FPGA Time 0 Interrupt handler Bank1 Read-out process Shared memory Reset interrupt Disk stager Consumer Process A Consumer Process B

12. Stefano Magni I.N.F.N Milano Italy - NSS 2003 DAQ time evolution(4/15) Basic mechanism of operation of the PCI card and the read-out: Bank0 FPGA Time 0 Interrupt handler Bank1 Read-out process Shared memory Reset interrupt Disk stager Consumer Process A Consumer Process B User limit

13. Stefano Magni I.N.F.N Milano Italy - NSS 2003 DAQ time evolution(5/15) Basic mechanism of operation of the PCI card and the read-out: Bank0 FPGA Time 0 Interrupt handler Bank1 Read-out process Shared memory Reset interrupt Disk stager Consumer Process A Consumer Process B

14. Stefano Magni I.N.F.N Milano Italy - NSS 2003 DAQ time evolution(6/15) Basic mechanism of operation of the PCI card and the read-out: Bank0 FPGA Time 0 Interrupt handler Bank1 Read-out process Shared memory Reset interrupt Disk stager Consumer Process A Consumer Process B

15. Stefano Magni I.N.F.N Milano Italy - NSS 2003 DAQ time evolution(7/15) Basic mechanism of operation of the PCI card and the read-out: Bank0 FPGA Time 0 Interrupt handler Bank1 Read-out process Shared memory Reset interrupt Disk stager Consumer Process A Consumer Process B

16. Stefano Magni I.N.F.N Milano Italy - NSS 2003 DAQ time evolution(8/15) Basic mechanism of operation of the PCI card and the read-out: Bank0 FPGA Time 0 Interrupt handler Bank1 Read-out process Shared memory Reset interrupt Disk stager Consumer Process A Consumer Process B

17. Stefano Magni I.N.F.N Milano Italy - NSS 2003 DAQ time evolution(9/15) Basic mechanism of operation of the PCI card and the read-out: Bank0 FPGA Time 0 Interrupt handler Bank1 Read-out process Shared memory Reset interrupt Disk stager Consumer Process A Consumer Process B

18. Stefano Magni I.N.F.N Milano Italy - NSS 2003 DAQ time evolution(10/15) Basic mechanism of operation of the PCI card and the read-out: Bank0 FPGA Time 0 Interrupt handler Bank1 Read-out process Shared memory Reset interrupt Disk stager Consumer Process A Consumer Process B

19. Stefano Magni I.N.F.N Milano Italy - NSS 2003 DAQ time evolution(11/15) Basic mechanism of operation of the PCI card and the read-out: Bank0 FPGA Time 0 Interrupt handler Bank1 Read-out process Shared memory Reset interrupt Disk stager Consumer Process A Consumer Process B

20. Stefano Magni I.N.F.N Milano Italy - NSS 2003 DAQ time evolution(12/15) Basic mechanism of operation of the PCI card and the read-out: Bank0 FPGA Time 0 Interrupt handler Bank1 Read-out process Shared memory Reset interrupt Disk stager Consumer Process A Consumer Process B

21. Stefano Magni I.N.F.N Milano Italy - NSS 2003 DAQ time evolution(1315) Basic mechanism of operation of the PCI card and the read-out: Bank0 FPGA Time 0 Interrupt handler Bank1 Read-out process Shared memory Reset interrupt Disk stager Consumer Process A Consumer Process B

22. Stefano Magni I.N.F.N Milano Italy - NSS 2003 DAQ time evolution(14/15) Basic mechanism of operation of the PCI card and the read-out: Bank0 FPGA Time 0 Interrupt handler Bank1 Read-out process Shared memory Reset interrupt Disk stager Consumer Process A Consumer Process B

23. Stefano Magni I.N.F.N Milano Italy - NSS 2003 DAQ time evolution(15/15) Basic mechanism of operation of the PCI card and the read-out: Bank0 FPGA Time 0 Interrupt handler Bank1 Read-out process Shared memory Reset interrupt Disk stager Consumer Process A Consumer Process B

24. Stefano Magni I.N.F.N Milano Italy - NSS 2003 A data driven (untriggered) DAQ Hits from pixel detectors are continuously read in untriggered mode. Single hit is tagged by PMC cards with a hardware generated time-stamp. An event, defined as “a collection of equal time-stamp hits from different detectors”, is built by the consumer process using a efficient sorting algorithm. The DAQ framework allows replacing and adding different event- building algorithms by defining an abstract interface to the appropriate implementation. This allows for different detectors, with different event-building strategies, to be read-out by the same DAQ system. Frequenze tipiche sono …

25. Stefano Magni I.N.F.N Milano Italy - NSS 2003 Event builder Event Builder Hit Buffer (Unordered data) Event Buffer (Ordered data) Time-stamps: Event

26. Stefano Magni I.N.F.N Milano Italy - NSS 2003 DAQ control The complete control of the DAQ system is performed through User Interfaces Both a command line and a graphical user interface (based on the QT library, http://www.trolltech.com) have been developed

27. Stefano Magni I.N.F.N Milano Italy - NSS 2003 DAQ monitor Acquired data and related DAQ information are streamed through sockets to the network, where client code on remote machines can suitably make use of those informations for histogram and event display. The graphics and histogramming package of choice is ROOT (http://root.cern.ch). No burden is placed on the host CPU for monitoring An unlimited number of people can remotely monitor the system with high efficiency

28. Stefano Magni I.N.F.N Milano Italy - NSS 2003 A successful Object Oriented approach During the development of the DAQ new versions of read-out chip have been available: thanks to the OO approach and the design of a virtual read-out chip class, inclusion of these new chips, along with additional detectors, became trivial. The first version of the system used a commercial general purpose PCI driver. After about 2 years of development of the DAQ system we decided to replace the commercial PCI driver to get free of licence issues; we developed a custom linux kernel device driver specifically suited for our cards. Thanks to the OO design the switching to the new driver required just one line of code change in the DAQ package !

29. Stefano Magni I.N.F.N Milano Italy - NSS 2003 A successful integration of open source components In this project we followed the general philosopy to take advantage of all the existing open source libraries and components and to develop just the specific DAQ related parts. The following open source packages has been used: Xerces : an xml parser by the Apache project (http://xml.apache.org/xerces-c). XML is used to provide contextual content to configuration files. Qt : a multiplatform, C++ framework with a set of powerful graphical widgets to develop sophisticated GUI applications (http://www.trolltech.com)http://www.trolltech.com Root : a data analysis package (http://root.cern.ch)http://root.cern.ch QTRoot: a set of graphical analysis widgets to integrate QT and ROOT (http://www-linux.gsi.de/~go4/qtroot/html/qtroot.html)http://www-linux.gsi.de/~go4/qtroot/html/qtroot.html Doxygen: a Web auto-documentation system for C++, C, Java, IDL. (http://www.doxygen.org)

30. Stefano Magni I.N.F.N Milano Italy - NSS 2003 Conclusions A low cost PCI DAQ framework has been developed The system is highly flexible – new detectors can be added just by providing a suitable mezzanine card – virtual functions for detector initialization e calibration are available in the framework A complete test of an untriggered DAQ for pixel detectors has been carried on We exploited the advantages of a good object oriented design