1. Thomas Jefferson National Accelerator Facility (JLab) 6/16/09Multi-threaded event processing with JANA -- David Lawrence 1 6 GeV electron accelerator user facility funded by the US Dept. of Energy Located in Newport News on the east coast of Virginia, USA 1 of the 2 major nuclear physics research labs in the U.S. CHL 2 for basic research into the quark structure of nuclear matter 12 GeV 11 GeV (CD-3 approval came in Sept. 2008 with data planned in 2014)

2. The GlueX Experiment in Hall-D 6/16/09Multi-threaded event processing with JANA -- David Lawrence2 real  beam 2 Tesla solenoid magnet 30 cm LH 2 target Forward EM calorimeter and forward TOF wall downstream Cylindrical and planar drift chambers inside magnet Barrel EM calorimeter inside magnet Conventional meson has quantum numbers determined only by constituent quarks Hybrid meson has some quantum properties due to contributions from the “glue” Mapping the spectrum of light-quark, exotic, hybrid mesons Mapping the spectrum of light-quark, exotic, hybrid mesons

3. Data Rates in 12GeV era Front End DAQ Rate Event Size L1 Trigger Rate Bandwidth to mass Storage GlueX3 GB/s15 kB200 kHz300 MB/s CLAS12100 MB/s20 kB10 kHz100 MB/s ALICE500 GB/s2.5 MB200 kHz200 MB/s ATLAS113 GB/s1.5MB75 kHz300 MB/s CMS200 GB/s1 MB100kHz100 MB/s LHCb40 GB/s40 kB1 MHz100 MB/s STAR8 GB/s80 MB100 Hz30 MB/s PHENIX900 MB/s~60 kB~ 15 kHz450 MB/s 6/16/09Multi-threaded event processing with JANA -- David Lawrence3 LHC JLab BNL * CHEP2007 talk Sylvain Chapelin private comm. * NIM A499 Mar. 2003 ppg 762-765 ** CHEP2006 talk MartinL. Purschke **

4. 4/27 Categories of Developers A: Framework B: Reconstruction C: End users (Analysis) 6/16/09Multi-threaded event processing with JANA -- David Lawrence

5. 5/27 What is the Job of the Reconstruction Framework? Allow users to implement code in a consistent way Provide access to calibrations, run conditions and geometry databases Modular enough to allow simple programs (calibration) Extensible enough to allow complex programs (full reconstruction) 6/16/09Multi-threaded event processing with JANA -- David Lawrence motherhood statements

6. CPU development in the coming years 6/16/09Multi-threaded event processing with JANA -- David Lawrence6 From “Platform 2015: Intel Platform Evolution for the Next Decade” expect more than 100 cores in a box by 2014! CPU development has shifted from increased clock speed to multiple cores Dual quad core CPUs are common today (8cores + 8hyperthreads) Some type of parallelization must be done to use all of the power in a next generation CPU

7. The JANA Factory Model 6/16/09Multi-threaded event processing with JANA -- David Lawrence7 Algorithms are represented as “factory” classes Only const pointers are passed out of the factory (ownership stays with the factory) Passing out only const pointers guarantees that only the factory may modify the objects Subsequent requests get the same const pointers vector tracks; loop->Get(tracks); Templated Get() method helps ensure type safety Data-on-demand structure allows dynamic determination of which algorithms are activated for a given event

8. Threads in JANA 6/16/09Multi-threaded event processing with JANA -- David Lawrence8 Each thread in JANA is composed of its own event processing loop and a complete set of factories Reconstruction of a given event is done entirely inside of a single thread No mutex locking is required by authors of reconstruction code Threads work asynchronously to maximize rates at the expense of not maintaining the event order on output raw data read in reconstructed values written out (e.g. ROOT tree)

9. The GlueX Reconstruction Tree (factory call graph created using janadot plugin) 6/16/09Multi-threaded event processing with JANA -- David Lawrence9 Number of calls and amount of time spent satisfying each is reported Objects at bottom of graph are (mostly) supplied by event source arrows indicate calling sequence data flow is in opposite direction

10. Multi-threading when CPU limited 6/16/09Multi-threaded event processing with JANA -- David Lawrence10 CPU intensive jobs are the ideal application for multi-threading Blue circles are reconstruction of data from a Monte Carlo simulation Red triangles are from a CPU-hungry speed testing plugin Both show very good scaling of the event processing rate with the number of threads Reconstruction of MC data, CPU bound jobs only Overall event processing rate scales linearly with the number of threads

11. Multi-threading when I/O limited 6/16/09Multi-threaded event processing with JANA -- David Lawrence11 Multiple processes trying to access different locations on the same disk leads to competition causing the read head to physically move back and forth from one location on the disk to another A multi-threaded application will access a single file in sequence reducing the number of moves the read head must make blue circles : one multi-threaded process reading from a single file red triangles : multiple single-thread processes reading different files from the same disk No processing of event data, I/O bound jobs only

12. 12/27 JANA is implemented in C++ JANA relies heavily on templates – Provides better type safety – Easily extensible External services need not be in C++ – Calibration server – Geometry description (XML) – Distributed deployment layer 6/16/09Multi-threaded event processing with JANA -- David Lawrence

13. Targeted Customers for JCalibration “B”-coders = reconstruction code authors (“A”-coders = framework authors; “C”-coders = physics analysis) 6/16/09Multi-threaded event processing with JANA -- David Lawrence13 JCalibration URL run_number context (string) … Reconstruction Code namepath (string) STL container ref. Database Reconstruction Program B-codersA-coders The minimal information needed from the reconstruction code is the name of the constants and a container to put them in. (The container implicitly contains a data type.)

14. Calibration Web Service The JCalibrationWS class provides calibration constants through a web service – Implemented as a plugin so remote access can be added to an existing executable – Allows read-only access to calibration constants from anywhere in the world over HTTP (http://www.jlab.org/Hall-D/cgi-bin/calib) – Uses gSOAP, a C++ SOAP implementation – Currently works like a proxy for JCalibrationFile on server side, but could trivially be made to use another type of backend Calibration constants will need to be accessible from remote computers via the internet Direct access to a database is problematic due to cybersecurity concerns Web services work over HTTP and so are the appropriate mechanism for remote access 6/16/0914Multi-threaded event processing with JANA -- David Lawrence

15. Saving a (semi-)complete set of calibration constants to the local disk All JANA programs have the command line option: --dumpcalibrations Records which namepaths are requested during a job and writes the constants into ASCII files compatible with JCalibrationFile Avoids copying and running entire database or even copying a “complete” set of calibration constants (which could include obsolete ones or ones not applicable to the current run/code version) 6/16/0915Multi-threaded event processing with JANA -- David Lawrence

16. BCAL Beam test 2006 The JANA framework was used during a beam test in 2006 to test a 4m long calorimeter module in photon beam. Reconstruction built around objects that could be constructed from multiple event sources. Online/offline systems built around simulated data in ROOT files. Later fed from DAQ system in online environment and finally from raw data files in offline. 6/16/0916Multi-threaded event processing with JANA -- David Lawrence

17. The janaroot plugin (for automatic creation of ROOT TTrees) 6/16/09Multi-threaded event processing with JANA -- David Lawrence17 Each data object implements a toStrings() method which provides an expression of the data object that may not be a full representation of the object The toStrings() mechanism was developed for allowing a simple, low-level dump of objects from single events to the screen This mechanism is leveraged by janaroot to provide a similar expression as TTree s An empty event tree is also created with all other trees Each leaf is an array of size “N” to represent the N objects of this type in the event A leaf named “N” is automatically added to each tree listed as friends so that a leaves from multiple objects can be used together in expressions Limitations make this unsuitable for all applications, but it does provide a quick, easy way to make plots of some reconstructed values for less experienced users

18. JANA is a multi-threaded event processing framework designed to build full reconstruction packages for a multi-core environment Numerous features like: plugins, automatic ROOT-tree generation, and calibrations/conditions DB API with working Web Services implementation Data-on-demand design makes it suitable for L3 trigger algorithm. 6/16/09Multi-threaded event processing with JANA -- David Lawrence18

19. Backup slides 6/16/0919Multi-threaded event processing with JANA -- David Lawrence

20. CEBAF The Continuous Electron Beam Accelerating Factility Electron beam accelerator continuous-wave (1497MHz, 2ns bunch structure in halls) Polarized electron beam Upgrading to 12GeV (from 6GeV) 70  A max @ 12GeV (200  A max @ 6GeV) Existing experimental halls A, B, C Future Hall-D site 6/16/0920Multi-threaded event processing with JANA -- David Lawrence

21. 21/27 Plugins void InitPlugin(JApplication *app) A plugin defines one external routine: Event Processors Event Source Generators Factory Generators Plugins can be used to add: 6/16/09Multi-threaded event processing with JANA -- David Lawrence

22. 22 API For Accessing Constants Constants can be stored in either arrays (1D) or tables (2D) and can be indexed either by name (key-value) or by position. // Get 1-D array of values indexed by name bool Get(string namepath, map &vals) // Get 1-D array of values indexed by row bool Get(string namepath, vector &vals) // Get 2-D table of values indexed by row and name bool Get(string namepath, vector > &vals) // Get 2-D table of values indexed by row and column bool Get(string namepath, vector > &vals) // Get 1-D array of values indexed by name bool Get(string namepath, map &vals) // Get 1-D array of values indexed by row bool Get(string namepath, vector &vals) // Get 2-D table of values indexed by row and name bool Get(string namepath, vector > &vals) // Get 2-D table of values indexed by row and column bool Get(string namepath, vector > &vals) arrays tables 6/16/09Multi-threaded event processing with JANA -- David Lawrence // Get list of available namepaths from backend void GetListOfNamepaths(vector &namepaths) // Get list of available namepaths from backend void GetListOfNamepaths(vector &namepaths) discovery

23. map twpars; loop->Get("FDC/driftvelocity/timewalk_parameters", twpars); slope = twpars["slope"]; offset = twpars["offset"]; exponent = twpars["exponent"]; map twpars; loop->Get("FDC/driftvelocity/timewalk_parameters", twpars); slope = twpars["slope"]; offset = twpars["offset"]; exponent = twpars["exponent"]; 23 Example of Accessing Calibration Constants as key-value pairs... in factory class definition … Template method converts values to doubles using stringstream class For a few parameters like this, it makes sense to copy them into local data members of the factory class 6/16/09Multi-threaded event processing with JANA -- David Lawrence double slope, offset, exponent; double slope, offset, exponent;... in brun() method...

24. Backend Requirements 6/16/09Multi-threaded event processing with JANA -- David Lawrence24 virtual bool GetCalib(string namepath, map &svals) virtual bool GetCalib(string namepath, vector > &svals) virtual void GetListOfNamepaths(vector &namepaths) virtual bool GetCalib(string namepath, map &svals) virtual bool GetCalib(string namepath, vector > &svals) virtual void GetListOfNamepaths(vector &namepaths) To implement a JCalibration interface to a new backend, only 3 virtual methods need to be implemented! OK, actually, it’s closer to 6, but these 3 are for the generator class and are trivial. virtual const char* Description(void) virtual double CheckOpenable(string url, int run, string context) virtual JCalibration* MakeJCalibration(string url, int run, string context) virtual const char* Description(void) virtual double CheckOpenable(string url, int run, string context) virtual JCalibration* MakeJCalibration(string url, int run, string context) The generator mechanism allows access to multiple types of databases in the same executable. Access to a new type can even be brought in through a plugin.

25. 25/27 Configuration Parameters 6/16/09Multi-threaded event processing with JANA -- David Lawrence

26. 26/27 Implementing an Event Source To implement an event source, 2 classes must be provided: JEventSourceGenerator const char* Description(void); double CheckOpenable(string source); JEventSource* MakeJEventSource(string source); JEventSource jerror_t GetEvent(JEvent &event); void FreeEvent(JEvent &event); jerror_t GetObjects(JEvent &event, JFactory_base *factory); 6/16/09Multi-threaded event processing with JANA -- David Lawrence

27. 27/27 JANA’s GUI API JANA’s has publicly accessible, lower level methods that allow the main “event” loop to be external to JEventLoop 6/16/09Multi-threaded event processing with JANA -- David Lawrence