The PHENIX Event Builder David Winter Columbia University for the PHENIX Collaboration CHEP 2004 Interlaken, Switzerland

28 Sept 2004David Winter - The PHENIX Event Builder2Overview Introduction –Heavy Ion Collisions –The Relativistic Heavy-Ion Collider –The PHENIX experiment and its DAQ The Event Builder –Hardware –Software –Monitoring & Performance Present and Future Development Summary

28 Sept 2004David Winter - The PHENIX Event Builder3 Begin at the Beginning Big Bang: Formation of the universe approximately years ago Condensation of “normal” matter (protons and neutrons) s later Water: steam  Water  Ice Hadronic matter: free quarks and gluons  protons, neutrons, etc. $64,000 Question: Can we recreate the hot and dense matter that existed at the beginning of the universe?

28 Sept 2004David Winter - The PHENIX Event Builder4 Torturing the Nucleus: Heavy Ion Collisions “Cartoon” of what we imagine to be phase diagram of hadronic matter (Temp vs. baryon density) Lattice QCD calculations have long indicated existence of phase transition

28 Sept 2004David Winter - The PHENIX Event Builder5 Relativistic Heavy Ion Collider 3.83 km circumference Two independent rings Capable of colliding ~any nuclear species on ~any other species Energy: è 500 GeV for p-p è 200 GeV for Au-Au (per N-N collision) Luminosity –Au-Au: 2 x cm -2 s -1 –p-p : 2 x cm -2 s -1 (polarized) As if that weren’t enough, it’s visible from space!

28 Sept 2004David Winter - The PHENIX Event Builder6PHENIX The Pioneering High-Energy Nuclear Interaction Experiment Two central arms for measuring hadrons, photons and electrons Two forward arms for measuring muons Event characterization detectors in center

28 Sept 2004David Winter - The PHENIX Event Builder7 Data Collection: The Challenge High rates –(Au-Au) Collisions occur at about 10kHz, while the beam crossing rate occurs at 9.6MHz. Interest in rare physics processes –J/  (  and e channels), high p t Spectra, others Large event sizes (Run-4: >200 kb/event) How do we address these problems? –Triggering Most crossings don’t interact: need to select those that do Selection based on “interesting” physics: Event has muon tracks, high energy or momentum charged or neutral particles. –Buffering & pipelining: Need a “deadtime-less” DAQ –High Bandwidth (Run-4: up to 400 MB/s archiving) –Fast processing (Level-2 triggering, for example) –Granules and Partitions A Granule is a subsystem unit A Partition consists of one or more granules. All granules in the partition receive the same triggers Multiple partitions can run in parallel, independent of each other

28 Sept 2004David Winter - The PHENIX Event Builder8 The PHENIX DAQ FEM GTMGTM LL1 GL1 DCM SEB GBit S W I T C H ATP EVENT TEMP STORAGE FEM SEB ATP Event Builder EBC Some systems are used to make triggers Some aren’t

28 Sept 2004David Winter - The PHENIX Event Builder9

28 Sept 2004David Winter - The PHENIX Event Builder10 The Event Builder at-a-glance Run Control Front-End Electronics Data Collection Modules Event Builder Controller Sub-Event Buffer Assembly Trigger Processor Event Storage

28 Sept 2004David Winter - The PHENIX Event Builder11 Event Builder Overview Multithreaded & Distributed Three functionally distinct programs derived from the same basic object SubEvent Buffer (SEB) –One per granule –Physically connected to DCMs –Collects data for a single subsystem – a “subevent” Event Builder Controller (EBC) –Receives event notification from GL1 SEB –Assigns events to ATPs –Flushes system when event is archived Assembly Trigger Processor (ATP) –Assigned task of assembling events –Requests subevent data from each SEBs –Writes assembled events to short-term storage –Notifies EBC of event completion

28 Sept 2004David Winter - The PHENIX Event Builder12CPUs 105 1U rack-mounted dual CPU servers 28 Dell PowerEdge 1000 –1.0 GHz Dual PIII –256 MB ECC RAM –8 GB Seagate U160 SCSI HDD 77 Microway machines –2.4 GHz Dual P4 Xeon –1 GB ECC RAM –37 GB Seagate ATA100 HDD –45 Tyan MB with 400 MHz FSB –32 Supermicro MB with 533 MHz FSB

28 Sept 2004David Winter - The PHENIX Event Builder13 Network Interfaces Dell PowerEdge servers –On-board dual Intel Fast Ethernet (10/100 Mbps) controllers –17 retrofitted with Gigabit adapter add-on cards 16 Intel PRO/1000 MT Server Adapters 1 Syskonnect SK-98xx Gigabit Adapter Tyan MB based Microways –On-board combination Intel Fast Ethernet and Gigabit network controllers Supermicro MB based Microways –On-board dual Intel PRO/1000 MT Server Adapter controllers

28 Sept 2004David Winter - The PHENIX Event Builder14 Gigabit Switch Foundry FastIron 1500 Gigabit Switch Total switching capacity = 480 Gbps Jumbo Frame capable (used 9014 MTU in Run-4, Run-5 configuration under study) 15 Slots (10 in use) –2 control modules - each includes Mbps ports (module spans two slots) –6 16-port Gigabit modules (95 ports in use) 39 SEBs 52 ATPs 1 EBC, phnxbox2, phnxbox3, corba

28 Sept 2004David Winter - The PHENIX Event Builder15 JSEB Interface Card PLX 9080 PCI controller Input for JSEB cable from Partitioner Module PCI 2.1 interface: 32-bit 33 MHz with 3.3v or 5v signaling 2 banks of 1 MB static RAM Altera FLEX10k FPGA JTAG port for programming FPGA

28 Sept 2004David Winter - The PHENIX Event Builder16 JSEB Cont’d Interfaces the Partitioner Module (PM) to the SEB Data from DCMs transmitted via 50-pair 32-bit parallel cable Firmware contained in FPGA –Versions 0x11 and 0x13 used in Run 4 –Version 0x27 written for Run 5 (Pseudo) Driver provided by Jungo DMA burst mode not available in Run 4 –Version 0x27 fixes this problem DMA Burst Mode off DMA Burst (4-word) DMA Burst (continuous)

28 Sept 2004David Winter - The PHENIX Event Builder17Software Run 4 Windows NT/2000 Visual C Iona Orbix ASP 6.0 Java 1.[34].x –Run Control –Monitoring Boost 1.30.x –Threads, Mutexes, etc. Code developed on dedicated machine, distributed to cluster via FTP Run 5 Linux (SL 3.0.2) GCC/G Iona Orbix ASP 6.1 Java 1.[34].x Boost 1.30.x Code developed on any machine with proper versions of CORBA and GCC –Common file systems NFS mounted, use make install

28 Sept 2004David Winter - The PHENIX Event Builder18CORBA Common Object Request Broker Architecture The networking protocol by which the run control software components talk to each other. Based on a client/server architecture through a heterogeneous computing environment. –VxWorks, Linux, Windows NT/2000 Servers: implement “CORBA Objects” that execute the functionality described in the member functions of the object Clients: invoke local CORBA Object’s methods. This causes the server to execute the code of its corresponding member function

28 Sept 2004David Winter - The PHENIX Event Builder19 Basic Component Design Booted Configured Initialized Configuration Data from Run Control State Machine whose transitions are controlled by CORBA methods Running Monitor Data to client(s) Specific EvB components are derived from this model Event-driven Multithreaded

28 Sept 2004David Winter - The PHENIX Event Builder20Level-2 Logical place to run Level-2: ATP (contains whole event) Primitives: perform calculations, record self, available to all algorithms Algorithms: high level decisions based on primitive calculations Primitives/Algorithms saved in ATP data stream Database primitives can read from db or ASCII files Can be run during event building or as later pass over data stream ~3-400 MB/sGB/s Storage Physics Events Level-1 Throttle: Single Detector Info, Scaledowns Level-2 Throttle: Make decisions based on multiple- detector info

28 Sept 2004David Winter - The PHENIX Event Builder21 Performance Monitoring Each component keeps track of various statistics Data served via CORBA calls Java client displays stats in “real time” Strip charts display data as function of time Histograms display data as function of component

28 Sept 2004David Winter - The PHENIX Event Builder22 Run Control

28 Sept 2004David Winter - The PHENIX Event Builder23 Where does the future lie? The most important improvement we can make: Port to Linux system Linux is a “sane” platform (saner than Win32) –Win32 WAS the right platform when using ATM (ATM-free in ’03!) –Each piece performed well, but when put together suffered degraded performance –At the limit of what Win32 can provide us (Peak stable performance: ~2.2kHz, DAQ is spec’d to at least 8kHz) –Exploit the vast body of Linux knowledge that already exists in PHENIX –Common development platform eases development and maintenance –Re-evaluate architecture and cut away the “fat” Integrate the improvements made in Run-4 but never used –ATPs requesting multiple events at a time –SEB Load balancing algorithm

28 Sept 2004David Winter - The PHENIX Event Builder24 Transition to Linux Architecture Issues –Fundamentally the architecture is sound, the primary issue being excessive overhead in the OS. Therefore, we are taking a direct-porting approach, wherever possible –Use boost template library to enhance portability and performance –Use of ACE is under study, though not planned for the start of Run-5 Chosen platform is Fermilab’s Scientific Linux (currently using version 3.0.1, moving to 3.0.2) –Based on Redhat’s Enterprise Linux –GCC –Linux’s new Native POSIX Thread Library (NPTL) Linux and asynchronous I/O –Win32 implementation relied heavily on overlapped I/O –Asynch (socket) I/O basically not usable in Linux (Pseudo-)Real-time event timers not available

28 Sept 2004David Winter - The PHENIX Event Builder25 The Impact of a Linux Port Win32 Asynch Socket Win32 Synch Socket Linux Synchronous Socket Linux beats Win32 hands-down in simple socket tests JSEB read test Without network writing (top curves) With network writing (bottom curves) Win32 Completion Socket A picture is worth a thousand words

28 Sept 2004David Winter - The PHENIX Event Builder26 Implementing Under Linux All I/O is now synchronous –At first this was a major concern, BUT It greatly simplifies the architecture Linux synchronous performance much better than anticipated Timeouts (Event Timers) –In Win32 version, all sockets were UDP –Unreliability of UDP requires some form of timeout –Level-1 messages from GL1 to EBC now use TCP –Communication between ATPs and EBC now uses TCP –TCP now greatly simplifies code –Data messages are still UDP, therefore a preliminary timeout mechanism has been developed

28 Sept 2004David Winter - The PHENIX Event Builder27 The Payoff To date, all components have been ported Approximately 35% of the servers converted –Distcc is a god-send: 90k lines of code in <3 min The timeout mechanism is in the works Preliminary tests with 1 & 2 granule partitions have been performed

28 Sept 2004David Winter - The PHENIX Event Builder28Summary