1. The PHENIX Event Builder David Winter Columbia University for the PHENIX Collaboration CHEP 2004 Interlaken, Switzerland

2. 28 Sept 2004David Winter - The PHENIX Event Builder2Overview Introduction –Heavy Ion Collisions –The Relativistic Heavy-Ion Collider –The PHENIX experiment and its DAQ The Event Builder –Hardware –Software –Monitoring & Performance Present and Future Development Summary

3. 28 Sept 2004David Winter - The PHENIX Event Builder3 Torturing the Nucleus: Heavy Ion Collisions “Cartoon” of what we imagine to be phase diagram of hadronic matter (Temp vs. baryon density) Lattice QCD calculations have long indicated existence of phase transition

4. 28 Sept 2004David Winter - The PHENIX Event Builder4 Relativistic Heavy Ion Collider 3.83 km circumference Two independent rings Capable of colliding ~any nuclear species on ~any other species Energy: è 500 GeV for p-p è 200 GeV for Au-Au (per N-N collision) Luminosity –Au-Au: 2 x 10 26 cm -2 s -1 –p-p : 2 x 10 32 cm -2 s -1 (polarized) As if that weren’t enough, it’s visible from space!

5. 28 Sept 2004David Winter - The PHENIX Event Builder5PHENIX The Pioneering High-Energy Nuclear Interaction Experiment Two central arms for measuring hadrons, photons and electrons Two forward arms for measuring muons Event characterization detectors in center

6. 28 Sept 2004David Winter - The PHENIX Event Builder6 Data Collection: The Challenge Blue and Yellow Beams cross at ~10MHz Collisions occur at ~10kHz hadrons photons leptons High rates Large event sizes (Run-4: >200 kb/event) Interest in rare physics processes => Big Headache How do we address these challenges? –Level-1 triggering –Buffering & pipelining: “deadtime-less” DAQ –High Bandwidth (Run-4: up to 400 MB/s archiving) –Fast processing (Level-2 triggering, for example)

7. 28 Sept 2004David Winter - The PHENIX Event Builder7 The PHENIX DAQ EvB A granule is a detector subsystem, including readout electronics A partition is one or more granules that receives the same triggers & busies

8. 28 Sept 2004David Winter - The PHENIX Event Builder8 Event Builder Overview Data stream from granule SEB GBit S W I T C H ATP SHORT TERM STORAGE SEB ATP EBC Data stream from granule Control provided by Run Control Three functionally distinct programs derived from the same basic object SubEvent Buffer (SEB): Collects data for a single subsystem – a “subevent” Event Builder Controller (EBC): Receives event notification, assigns events, flushes system Assembly Trigger Processor (ATP): Assembles events by requesting data from each SEB, writes assembled events to short-term storage, can also provide Level- 2 trigger environment

9. 28 Sept 2004David Winter - The PHENIX Event Builder9CPUs 105 1U rack-mounted dual CPU servers 28 Dell PowerEdge 1000 –1.0 GHz Dual PIII –256 MB ECC RAM –8 GB Seagate U160 SCSI HDD 77 Microway machines –2.4 GHz Dual P4 Xeon –1 GB ECC RAM –37 GB Seagate ATA100 HDD –45 Tyan MB with 400 MHz FSB –32 Supermicro MB with 533 MHz FSB

10. 28 Sept 2004David Winter - The PHENIX Event Builder10 Network Interfaces Dell PowerEdge servers –On-board dual Intel Fast Ethernet (10/100 Mbps) controllers –17 retrofitted with Gigabit adapter add-on cards 16 Intel PRO/1000 MT Server Adapters 1 Syskonnect SK-98xx Gigabit Adapter Tyan MB based Microways –On-board combination Intel Fast Ethernet and Gigabit network controllers Supermicro MB based Microways –On-board dual Intel PRO/1000 MT Server Adapter controllers

11. 28 Sept 2004David Winter - The PHENIX Event Builder11 Gigabit Switch Foundry FastIron 1500 Gigabit Switch Total switching capacity = 480 Gbps Jumbo Frame capable (used 9014 MTU in Run-4, Run-5 configuration under study) 15 Slots (10 in use) –2 control modules - each includes 24 100 Mbps ports (module spans two slots) –6 16-port Gigabit modules (95 ports in use) 39 SEBs 52 ATPs 1 EBC, phnxbox2, phnxbox3, corba

12. 28 Sept 2004David Winter - The PHENIX Event Builder12 JSEB Interface Card PLX 9080 PCI controller Input for JSEB cable from Partitioner Module PCI 2.1 interface: 32-bit 33 MHz with 3.3v or 5v signaling 2 banks of 1 MB static RAM Altera FLEX10k FPGA JTAG port for programming FPGA

13. 28 Sept 2004David Winter - The PHENIX Event Builder13 JSEB Cont’d Interfaces the Partitioner Module (PM) to the SEB Data from DCMs transmitted via 50-pair 32-bit parallel cable (Pseudo) Driver provided by Jungo DMA burst mode not available in Run 4 –New firmware fixes this problem Firmware contained in FPGA –Versions 0x11 and 0x13 used in Run 4 –Version 0x27 written for Run 5 DMA Burst Mode off DMA Burst (4-word) DMA Burst (continuous)

14. 28 Sept 2004David Winter - The PHENIX Event Builder14Software Run 4 Windows NT/2000 Visual C++ 6.0 Iona Orbix ASP 6.0 Java 1.[34].x –Run Control –Monitoring Boost 1.30.x –Threads, Mutexes, etc. Code developed on dedicated machine, distributed to cluster via FTP Run 5 Linux 2.4.21 (SL 3.0.2) GCC/G++ 3.2.3 Iona Orbix ASP 6.1 Java 1.[34].x Boost 1.30.x Code developed on any machine with proper versions of CORBA and GCC –Common file systems NFS mounted, use make install

15. 28 Sept 2004David Winter - The PHENIX Event Builder15CORBA Common Object Request Broker Architecture The networking protocol by which the run control software components talk to each other. Based on a client/server architecture through a heterogeneous computing environment. –VxWorks, Linux, Windows NT/2000 Servers: implement “CORBA Objects” that execute the functionality described in the member functions of the object Clients: invoke local CORBA Object’s methods. This causes the server to execute the code of its corresponding member function

16. 28 Sept 2004David Winter - The PHENIX Event Builder16 Basic Component Design Booted Configured Initialized Configuration Data from Run Control State Machine whose transitions are controlled by CORBA methods Running Monitor Data to client(s) Specific EvB components are derived from this model Event-driven Multithreaded UDP sockets for data TCP sockets for control

17. 28 Sept 2004David Winter - The PHENIX Event Builder17 Performance Monitoring Each component keeps track of various statistics Data served via CORBA calls Java client displays stats in “real time” Strip charts display data as function of time Histograms display data as function of component

18. 28 Sept 2004David Winter - The PHENIX Event Builder18 Run Control

19. 28 Sept 2004David Winter - The PHENIX Event Builder19 Where does the future lie? The most important improvement we can make: Port to Linux Linux is a “sane” platform (saner than Win32) –Win32 WAS the right platform when using ATM –At the limit of what Win32 can provide us (Peak stable performance: ~2.2kHz, DAQ is spec’d to at least 8kHz) –Common platform among PHENIX developers Re-evaluate architecture (keeping basic design) –Integrate the improvements made in Run-4 but never used –Win32 implementation relied heavily on overlapped I/O –Win32 used timers to rerequest dropped UDP packets All control messages converted to TCP Preliminary timeout mechanism under development

20. 28 Sept 2004David Winter - The PHENIX Event Builder20 The Impact of a Linux Port Win32 Asynch Socket Win32 Synch Socket Linux Synchronous Socket Linux beats Win32 hands-down in simple socket tests JSEB read test Without network writing (top curves) With network writing (bottom curves) Win32 Completion Socket A picture is worth a thousand words

21. 28 Sept 2004David Winter - The PHENIX Event Builder21 The Payoff To date, all components have been ported Approximately 35% of the servers converted –Distcc is a god-send: 90k lines of code in <3 min Preliminary tests with 1 & 2 granule partitions have been performed

22. 28 Sept 2004David Winter - The PHENIX Event Builder22Summary Heavy Ion Collisions at RHIC are an ideal means to study hot, dense quark matter The PHENIX experiment is designed to make high statistics measurements of rare probes of the QGP The PHENIX Event Builder is used to assemble events from parallel data streams, enabling high rates of archiving. –Three multithreaded programs originally implemented on Win32 –Win32 EvB has done a respectable job so far, but we need more Linux is the future –Synchronous I/O superior to even Win32’s overlapped I/O –OS overheads much lower (in general) –Various issues when porting from Win32 to Linux I/O, timers, threading Bottom line: Run-5 will have a Linux Event Builder that early tests show will improve performance by as much as a factor of 10.