1. Martin L Purschke The PHENIX Online Computing System 1 The PHENIX Online Computing System Martin L. Purschke, Edmond Desmond, Lars Ewell, John Haggerty, Hyon-Joo Kehayias, Ryan Roth, Chris Witzig Physics Department, Brookhaven National Laboratory Stephen S Pate Department of Physics, New Mexico State University Outline: Phenix at a glance ONCS’ job description The PHENIX Command center Corba-based tools error and other message reporting data flow and data access EPICS

2. Martin L Purschke The PHENIX Online Computing System 2 PHENIX at a glance Some Numerology 400,000 readout channels in detector’s final state 11 detector subsystems Minimum bias rate is ~ 1KHz (AU-AU) AU-AU central collision data size ~350Kb p-p collision data size 2-4 Kb Data delivered to mass storage at 20MB/s Phenix Schedule: “engineering run” on now, June 1999 (experiment shakedown) start data taking at 1% RHIC luminosity in Nov 1999 go to 10% luminosity in year 2000 ramp up to 100% in 2001

3. Martin L Purschke The PHENIX Online Computing System 3 Front-end block diagram Data Collection Module (DCM) Data Collection Module (DCM) Sub-Event Buffer (SEB) Partition Module (PM) Data Collection Module (DCM) Front End Module (FEM) Front End Module (FEM) Front End Module (FEM) Front End Module (FEM) Assembly &Trigger Processor (ATP) Assembly &Trigger Processor (ATP) Assembly &Trigger Processor (ATP) Assembly &Trgger Processor (ATP) Assembly &Trigger Processor (ATP) Assembly &Trigger Processor (ATP) Assembly &Trigger Processor (ATP) Assembly &Trigger Processor (ATP) Assembly &Trigger Processor (ATP) ATM Switch Busy Global Level 1 Local Level 1 Master Timing Module (MTM) Granule Timing Module (GTM) BusyAccept Timing Fiber Serial Control ARCNET Sub-Event Buffer (SEB)... DAQ front end Event Builder Timing/ Trigger System Data Logging ONCS is responsible for configuring these 4 general areas. Talk #167, Thu 1:00pm, Peter Steinberg

4. Martin L Purschke The PHENIX Online Computing System 4 PHLOPS - The PHENIX Command Center Maintains the proper state of all components of the system Manages the operation of the system PHENIX Online Operating System PHLOPS DCM FEM Crate Controller Crate Controller Crate Controller Crate Controller Crate Controller SEB ATP Timing System Timing System Timing System Timing System Abcdefgghgh GUI xterm Phlops> info

5. Martin L Purschke The PHENIX Online Computing System 5 Online Computer Infrastructure VxWorks Linux CPU-intensive Online monitoring Detector control, Online monitoring Event Builder and Trigger processors Front-end Processors X-terms, Routers, OC-3 connection to the computing center XXX

6. Martin L Purschke The PHENIX Online Computing System 6 Software tools in use The main tool for communications between tasks, processors and services. Pretty much the backbone of the distributed PHENIX online environment. Orbix via JAVA -- used for GUI’s, control applets, info applets, also brings ORBIX to the Linux platforms Objectivity is used to store all sorts of “meta” data - HV information, temperature readings, configuration settings, geometry info (also big in offline computing) ROOT is used for analysis and monitoring, and for most user-developed GUI’s High Voltage control, and some miscellaneous things

7. Martin L Purschke The PHENIX Online Computing System 7 Tools based on CORBA (Orbix) Rather than specifying “hardware addresses” (IP, memory, port number, …), we let the configuration tools connect to “services”, provided by named CORBA servers. In this way, we achieve a great deal of flexibility. We need that because the things that need to come together always involve more than one crate and processor. Configuration files don’t normally contain parameters such as processor names, which can change. In the commissioning phase of the experiment, that has turned out to simplify things dramatically. Most services and components receive names which are unique and can be accessed in a flexible fashion. The most important services are the Event notifier, the Name server, and several object managers. All of them are CORBA servers which can be accessed remotely. In general, each physical hardware component is represented by a (CORBA) object which controls the actual hardware. We show a poster on this, poster 85 (Ed Desmond)

8. Martin L Purschke The PHENIX Online Computing System 8 Insulating Hardware with the Event Notifier PHLOPS DCM PHLOPS wishes to communicate to a DCM (control object): “Please download code xyz” Name Server Event Notifier A) Name Server Object Manager B) Object Manager DCM Event Notifier C) At startup/configuration time, the Event Notifier registers itself with the Name Server (A) the Object Manager (there can be more than one) registers itself with the Name Server and gets hold of the Event Notifier service (B) The Object Manager instantiates the DCM control object and registers its interest in certain messages with the Event Notifier (C). At run time, PHLOPS sends the request to the Event Notifier (1) which forwards it to the DCM object with the proper name. If an acknowledgement is required, it will take the inverse route. Event Notifier 1)2) PHLOPS Event Notifier Object Manager DCM

9. Martin L Purschke The PHENIX Online Computing System 9 Excerpt from a configuration file // #define SWPU_START_AT_INIT 0 #define SWPU_START_AT_DOWNLOAD 1 #define SWPU_START_AT_STARTRUN 2 //------------------------------------------------ // timing system //------------------------------------------------ ONCS_GTM, GTM.EMC.W, TMserver10, $ONLINE_CONFIGURATION/GTM/GTM.EMCAL25.gtm //------------------------------------------------ ONCS_ROOBJECT, ro1, daq1bobjmanager, 1, ONCS_DC, $DCM_SRC/config1b_emc32 ONCS_DCB, dcb.dc.4, daq1bobjmanager, 4 ONCS_DCB, dcb.dc.5, daq1bobjmanager, 5 ONCS_DCM_FE1, dcm.dc.41, daq1bobjmanager, dcb.dc.4, $DCM_DATA/dsp_prog_dc_ldr.bin,... ONCS_DCM_FE1, dcm.dc.42, daq1bobjmanager, dcb.dc.4, $DCM_DATA/dsp_prog_dc_ldr.bin,... ONCS_DCM_FE1, dcm.dc.51, daq1bobjmanager, dcb.dc.5, $DCM_DATA/dsp_prog_dc_ldr.bin,... ONCS_DCM_FE1, dcm.dc.52, daq1bobjmanager, dcb.dc.5, $DCM_DATA/dsp_prog_dc_ldr.bin,... (edited for readability) 1234567812345678 1 -- Ask the TMserver10 service to create a “Global Timing Module” for us, name it “GTM.EMC.W” 2 -- Ask the daqobj1bmanager service to create a “Readout Object” for us, name it “ro1” 3,4 -- Create “Data Collection Boards” called dcb.dc.4 and dcb.dc.5, respectively 5,6 -- Create “Data collection Modules” which are components on dcb.dc.4 7,8 -- Create “Data collection Modules” which are components on dcb.dc.5 1) is a service provided by a server called “TMserver10”. 2) through 8) are services provided by a server called “daqobj1bmanager”. Nowhere in here is any mention of a particular processor or IP address, or similar.

10. Martin L Purschke The PHENIX Online Computing System 10 Data access and online monitoring The main tools of distributing Event data in the Online system is the Data Distributor Pool (DD). The DD system is a very versatile tool to make event data available for online monitoring. from Experiment Online Monitoring Process Data Logger Debug Utility Virtually unlimited number of “consumers” Each consumer defines its access policy consumers can dynamically attach and detach

11. Martin L Purschke The PHENIX Online Computing System 11 Data access policies guaranteed delivery of all events in the pool hold off the input if it cannot keep up with the data flow. Data Logger give me events as fast as possible, but not necessarily all of them must allow to “skip” events Online Monitoring split a stream over more than one consumer each event is deliverd to only one consumer (load balancing) Shared Stream even in a shared stream, certain administrative-type events can still be sent to all consumers (begin-run, etc) Broadcast Events DD Pool Consumer 1Consumer 2Consumer 3 DD Pool Consumer 1Consumer 2Consumer 3 DD Pool Consumer 1Consumer 2Consumer 3

12. Martin L Purschke The PHENIX Online Computing System 12 Event Iterators testEventiteratorddEventiteratorfileEventiteratorEventiterator DD Pool An Event object manages the data of one particular event; provides easy access to the data; provides easy access to “envelope” information (type, length, date/time, structure,…); All data input streams are read by an Eventiterator, which in turn manages and provides access to the Events read in from a stream. It returns Event objects to the consumer.

13. Martin L Purschke The PHENIX Online Computing System 13 ROOT is the main analysis/monitoring tool Phenix has selected ROOT as the main analysis tool for the analysis and the online monitoring. We show here a scenario like this: { gSystem->Load("$EVT_LIB/libEvent.so"); h1f = new TH1F("h1f","Test random numbers",100,-49.5,50.5); Eventiterator *it = new ddEventiterator("online"); int i; for (i=0; i<3000; i++) { Event *e = it->getNextEvent(); Packet *p = e->getPacket(1003); h1f->Fill(p->iValue(0)); delete p; delete e; } delete it; h1f->Draw(); } ddEventiterator DD Pool

14. Martin L Purschke The PHENIX Online Computing System 14 A messaging system (for error messages) The problem set: provide a messaging system in the online environment which is versatile enough yet can be used easily in an offline setting (share object libraries between offline and online) allow a “message profile” such as type, source, severity level allow different destinations of messages with different profiles, and allow multiple destinations allow different system responses based on the profile allow dynamic “re-routing” of alarm messages (“call me at home”) avoid special calls to your code such as msg_print_debug(__LINE__,__FILE__, "Division by 0"); Force output to go through the system to prevent it from going unnoticed into log files Allow legacy code using printf to use the system without modification.

15. Martin L Purschke The PHENIX Online Computing System 15 How it’s done Continue to use cout<< in your code in an online environment, we globally modify the behavior of the cout object to achieve the desired functionality cout (and in general, all objects of type "ostream") are only the formatting engine. They enlist the services of another object (of type "streambuf") to handle their formatted output. We can provide a new "streambuf" object to be used by cout which has the functionality which we want. This is a standard procedure in C++, not some "dirty trick". { if ( file_is_open) return -1; sprintf( ctrlbuf->current_filename, ctrlbuf->filerule, run_number); if ( (the_file = fopen(ctrlbuf->current_filename, "w") )==NULL ) { cerr << "file not opened " << strerror(errno) << endl; *fp = 0; return -1; } *fp = the_file; file_is_open = 1; // cout current_filename << endl; return 0; } int ndd_control::close_file() { if ( !file_is_open) return -1; fclose(the_file); { if ( file_is_open) return -1; sprintf( ctrlbuf->current_filename, ctrlbuf->filerule, run_number); if ( (the_file = fopen(ctrlbuf->current_filename, "w") )==NULL ) { cerr << "file not opened " << strerror(errno) << endl; *fp = 0; return -1; } *fp = the_file; file_is_open = 1; // cout current_filename << endl; return 0; } int ndd_control::close_file() { if ( !file_is_open) return -1; fclose(the_file); } User Code cout streambuf { if ( file_is_open) return -1; sprintf( ctrlbuf->current_filename, ctrlbuf->filerule, run_number); if ( (the_file = fopen(ctrlbuf->current_filename, "w") )==NULL ) { cerr << "file not opened " << strerror(errno) << endl; *fp = 0; return -1; } *fp = the_file; file_is_open = 1; // cout current_filename << endl; return 0; } int ndd_control::close_file() { if ( !file_is_open) return -1; fclose(the_file); { if ( file_is_open) return -1; sprintf( ctrlbuf->current_filename, ctrlbuf->filerule, run_number); if ( (the_file = fopen(ctrlbuf->current_filename, "w") )==NULL ) { cerr << "file not opened " << strerror(errno) << endl; *fp = 0; return -1; } *fp = the_file; file_is_open = 1; // cout current_filename << endl; return 0; } int ndd_control::close_file() { if ( !file_is_open) return -1; fclose(the_file); } User Code cout custom streambuf ASADMS * * ASADMS means “All Singing, All Dancing Messaging System”

16. Martin L Purschke The PHENIX Online Computing System 16 “Message System-aware” code #include void detectorcode() { msg_control *mvd_info = new msg_control( MSG_TYPE_CODEDEBUG, MSG_SOURCE_MVD, MSG_SEV_INFORMATIONAL ); cout << *mvd_info << " this is a info message from MVD" << endl; } Message “profile” Profile information gets added to the message. If there is no ASADMS, this is inactive. cout << " this is some message from MVD" << endl; In the absence of explicit profiling information (legacy/unaware code), the system supplies a default profile. It is guaranteed that all messages generated by a program are funneled through the ASADMS, if one is present.

17. Martin L Purschke The PHENIX Online Computing System 17 Code snippet to give you the idea filter_msg_buffer::filter_msg_buffer(const int msglen) : msg_buffer(msglen) { //... stuff deleted... original_streambuf = cout.rdbuf ( (streambuf *) this); } // in the destructor, we restore the original streambuf for cout. filter_msg_buffer::~filter_msg_buffer() { cout.rdbuf ( original_streambuf ); } The streambuf constructor retains the pointer to cout’s old streambuf and registers itself to be the new streambuf. In the destuctor we reverse the process and restore the original streambuf.

18. Martin L Purschke The PHENIX Online Computing System 18 A Publish-Subscribe scheme ASADMS Subscribe “Give me all messages of severity warning { if ( file_is_open) return -1; sprintf( ctrlbuf->current_filename, ctrlbuf->filerule, run_number); if ( (the_file = fopen(ctrlbuf->current_filename, "w") )==NULL ) { cerr << "file not opened " << strerror(errno) << endl; *fp = 0; return -1; } *fp = the_file; file_is_open = 1; // cout current_filename << endl; return 0; } int ndd_control::close_file() { if ( !file_is_open) return -1; fclose(the_file); { if ( file_is_open) return -1; fclose(the_file); } Different Processes { if ( file_is_open) return -1; sprintf( ctrlbuf->current_filename, ctrlbuf->filerule, run_number); if ( (the_file = fopen(ctrlbuf->current_filename, "w") )==NULL ) { cerr << "file not opened " << strerror(errno) << endl; *fp = 0; return -1; } *fp = the_file; file_is_open = 1; // cout current_filename << endl; return 0; } int ndd_control::close_file() { if ( !file_is_open) return -1; fclose(the_file); { if ( file_is fclose(the_file); } Different Processes { if ( file_is_open) return -1; sprintf( ctrlbuf->current_filename, ctrlbuf->filerule, run_number); if ( (the_file = fopen(ctrlbuf->current_filename, "w") )==NULL ) { cerr << "file not opened " << strerror(errno) << endl; *fp = 0; return -1; } *fp = the_file; file_is_open = 1; // cout current_filename << endl; return 0; } int ndd_control::close_file() { if ( !file_is_open) return -1; fclose(the_file); Different Processes { if ( file_is_open) return -1; sprintf( ctrlbuf->current_filename, ctrlbuf->filerule, run_number); if ( (the_file = fopen(ctrlbuf->current_filename, "w") )==NULL ) { cerr << "file not opened " << strerror(errno) << endl; *fp = 0; return -1; } *fp = the_file; file_is_open = 1; // cout current_filename << endl; return 0; } int ndd_control::close_file() { if ( !file_is_open) return -1; fclose(the_file); { if ( file_is_open) return -1; sprintf( ctrlbuf->current_filename, ctrlbuf->filerule, Different Processes Subscribe “Give me all messages from MVD” We insert all messages in a pool where different users or processes can subscribe to receive messages which matches certain profiles (much like a list server). It supports an unlimited number of subscribers.

19. Martin L Purschke The PHENIX Online Computing System 19 EPICS with ROOT and Objectivity We use EPICS primarily for High Voltage control. We use ROOT as a front-end to display HV properties (strip charts, deviations, channel overviews). Right now we have several LeCroy HV modules in use, some other custom-built hardware (most notably the VME-based HiVoC card) will be interfaced with EPICS soon. The HV data (history) gets stored in an Objectivity database.

20. Martin L Purschke The PHENIX Online Computing System 20 Summary We have a very flexible system to configure the online system (DAQ, Timing/Trigger system, Event Builder, and Data logging); We insulate our hardware from logical concepts by name servers, named servers, and named objects; We use CORBA extensively (please look at Poster 85); We use the data distributor (DD) pool to access data; ROOT is the main analysis and monitoring tool; Objectivity is used to store all sorts of data (configuration, HV, anc. readings) We have a versatile messaging system to handle alarms, error messages, and so on We use EPICS for HV control, with ROOT as a GUI and display tool.