1. Design, Development and Evolution of the ROOT system Rene Brun February 21 2010 ACAT2010 Jaipur René Brun, ACAT2010, Jaipur1ROOT: design,development, evolution

2. plan ROOT in 2010 packagesusers How did we reach this situation? after-PAW fights Objectivity era and big confusion about object persistency LCG project and problems with the committee approach the ramping and consolidation years Directions René Brun, ACAT2010, Jaipur2ROOT: design,development, evolution

3. René Brun, ACAT2010, JaipurROOT: design,development, evolution3 OS & compilers Networking Frameworks like ROOT, Geant4 Experiment Software End user Analysis software Hardware 20 MLOC 2 MLOC 0.1 MLOC Hardware Software hierarchy

4. ROOT today a quick review René Brun, ACAT2010, JaipurROOT: design,development, evolution4

5. 15 years of ROOT René Brun, ACAT2010, JaipurROOT: design,development, evolution5 1995200020052010 after PAW GEANT 3 PAW-like ROOT Objectivity era LCG projects start PI, SEAL, POOL ROOT mainstream

6. Modular structure? Nice and modular building. But can we use only what we need? René Brun, ACAT2010, JaipurROOT: design,development, evolution6

7. Project Structure CORE I/O GRAF3 D GRAF2D MATH HIST NET Tree s TGEO GUI PROOF xroot d Roofit TMVA EVE reflex CINT VMC g3root Tfluka g4roo t RooStats pyroot 7ROOT: design,development, evolutionRené Brun, ACAT2010, Jaipur

8. framework Like in the Albert Hall Museum, we created base classes (the CORE), doors and windows (IO) and utility libraries on top. Can be seen as a mess or a very functional building. René Brun, ACAT2010, JaipurROOT: design,development, evolution8

9. Interpreters In this talk, I have no time to discuss issues about interpreters like CINT and Pyroot, or the internals of the reflexion system (see my slide about LLVM later). CINT and its reflexion system is the cornerstone of ROOT for the IO system and scripts. CINT was not designed to be a full C++ interpreter. We came several years ago with ACLIC that automatizes the interface with the compiler. We strongly recommend the use of ACLIC for medium/large size scripts. LLVM will simplify even more the problem once the system will be operational. René Brun, ACAT2010, JaipurROOT: design,development, evolution9

10. René Brun, ACAT2010, JaipurROOT: design,development, evolution10 root >.x script.C root > DoSomething(…); root >.x script.C++ root >.x script.C+ gROOT->ProcessLine(“.L script.C+”); gROOT->ProcessLine(“DoSomething(…)”); execute file script.C execute function DoSomething compile file script.C and execute it compile file script.C if file has been modified. execute it same from compiled or interpreted code Interpreter & Compiler integration

11. 11 ROOT Math/Stat Libraries ROOT Math/Stat LibrariesTMinuit TFumili Minuit2 Fitting and Minimization Linear Fitter Quadp Extra algorithms Extra Math functions GSL MathMor e Random Numbers MLP Statistical Libraries Limit classes Y Math functions Basic algorithms Math Interfaces MathCore TComplex TRandom Fitting Classes FFTW Extra Libraries Unuran FoamSpectrum SMatrix Linear Algebra TMatrix GenVector Physics Vectors TVector3/TLV RooFit TMVA High Level Analysis Libraries RooStats ROOT: design,development, evolution The new CERNLIB/MAT HLIB With better algorithms René Brun, ACAT2010, Jaipur

12. ROOT 2-D graphics based on 10 years experience with PAW and 15 years of operation with ROOT itself. zillions of tiny new features always requested by users works in batch (canvas.root, canvas.pdf, canvas.gif,..) object editors C++ script generated from canvas (TObject::SavePrimitive) René Brun, ACAT2010, JaipurROOT: design,development, evolution12

13. 2-D Graphics 13 New functions added at each new release. Always requests for new styles and new ways to display data. ps,pdf,svg,gif, jpg,png,c,root, etc jpg,png,c,root, etc Move to GL ? ROOT: design,development, evolutionRené Brun, ACAT2010, Jaipur

14. ROOT: design,development, evolution14René Brun, ACAT2010, Jaipur

15. Image processing René Brun, ACAT2010, JaipurROOT: design,development, evolution15

16. GUI Portable GUI with all widgets found in modern UI toolkits. Many enhancements in the GUI classes: browser, html browser, TABs, EVE widgets, FitPanel. GUI builder with C++ code generator. Session recorder tool (class TRecorder) can store and replay a GUI session: All mouse events Keyboard input, including macro execution QT interfaces: a big pain, difficult to maintain with the successive versions of Qt. 16ROOT: design,development, evolutionRené Brun, ACAT2010, Jaipur

17. GUI ROOT: design,development, evolution17

18. ROOT: design,development, evolution18 GUI One of the most demanding areas Must follow technology and not standard systems (gks -> phigs - >Motif ->Qt ->?) GUI based on OpenGL too ! Plug-ins must work on the web (but 2-d and 3-d graphics too!) GUI builder extensions (code generators) GUI must be scriptable and theme-able HELP + language internalisation Context menus help access to documentation René Brun, ACAT2010, Jaipur

19. ROOT 3-D graphics OpenGL based: graphics accelerators detector geometry visualization event displays (EVE) interaction between GL view and original objects (picking/editing) René Brun, ACAT2010, JaipurROOT: design,development, evolution19

20. 3-D Graphics Highly optimized GL views in TPad TPad the GL viewer 20 ROOT: design,development, evolution Mathematical objects Detector hits René Brun, ACAT2010, Jaipur

21. ROOT: design,development, evolution21 Graphics (2D-3D) “SURF” “LEGO” TF3 TH3 TGLParametric René Brun, ACAT2010, Jaipur

22. The Geometry Package TGEO The TGeo classes are now stable (thanks ALICE). Can work with different simulation engines (G3,G4,Fluka) (See Virtual Monte Carlo) G3->TGeo, G4->TGeo, TGeo  >GDML Used in online systems and reconstruction programs Built-in facilities for alignment Impressive gallery of experiments (35 detectors in $ROOTSYS/test/stressGeometry) 22ROOT: design,development, evolutionRené Brun, ACAT2010, Jaipur

23. ROOT: design,development, evolution23 OpenGL see $ROOTSYS/tutorials/geom René Brun, ACAT2010, Jaipur

24. Event Display : EVE a GL-based package for event displays. Developed in collaboration with ALICE (AliEve) and CMS (FireWorks). Provides all the GUI widgets, browsers, GL infrastructure (far better than the old OpenInventor). Used now by many experiments (see eg FAIRROOT, ILCROOT) to display raw data, MC events or detector oriented visualization. 24ROOT: design,development, evolutionRené Brun, ACAT2010, Jaipur

25. PROOF Massive data analysis with parallel systems. Now in production in many places. Growing success. A lot of developments for a better integration with dataset management systems and to support priority and accounting management. PROOF-LITE is a mini version to run on a multi-core. ROOT: design,development, evolution25 ROOT: one core only with PROOF with PROOF-LITE René Brun, ACAT2010, Jaipur

26. IO and Trees They are the most important components of the system because they have a deep impact on the experiment frameworks and their data handling strategy. Shaping these 2 systems has been a long process. René Brun, ACAT2010, JaipurROOT: design,development, evolution26

27. René Brun, ACAT2010, JaipurROOT: design,development, evolution27 Memory Tree Each Node is a branch in the Tree tr 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 T.Fill() T.GetEntry(6) T Memory

28. Automatic branch creation from object model float a; int b; double c[5]; int N; float* x; //[N] float* y; //[N] Class1 c1; Class2 c2; //! Class3 *c3; std::vector ; TClonesArray *tc; René Brun, ACAT2010, Jaipur28ROOT: design,development, evolution branch buffers

29. branches created from a dynamic list a b c d e a a a a a Each named member of a TList Is a top level branch René Brun, ACAT2010, Jaipur29ROOT: design,development, evolution

30. Top level branches created by user a b c d e Top level branches Are created manually René Brun, ACAT2010, Jaipur30ROOT: design,development, evolution

31. Branches created from STL collections std::vector Use split=199 to reduce The number of branches In case of polymorphic collections std::vector TClonesArray std::vector split = 0 split >0 split <100 split =199 split >0 René Brun, ACAT2010, Jaipur31ROOT: design,development, evolution

32. Trends in Tree structure Maximum split level, ie more branches is better eg, Atlas 9785, CMS 1436, LHCb 1167, Alice >1000 Many branches is good for the future because branch reading can be executed in parallel. many branches is better because the access granularity is better for fast selections. René Brun, ACAT2010, JaipurROOT: design,development, evolution32

33. ObjectWise/MemberWise Streaming abcdabcd a1b1c1d1a2b2c2d2…anbncndn a1a2..anb1b2..bnc1c2..cnd1d2..dn a1a2…an b1b2…bn c1c2…cn d1d2…dn 3 modes to stream an object object-wise to a buffer member-wise to a buffer each member to a buffer member-wise gives better compression René Brun, ACAT2010, Jaipur33ROOT: design,development, evolution

34. Supported Platforms Linux (RH, SLC4, SLC5, Suse, Debian, Ubuntu) gcc3.4,…, gcc4.4 (32 and 64 bits) icc10.1, icc11.1 MacOS X (Intel/PPC, 10.4, 10.5, 10.6) gcc4.0.1, gcc4.2.1,gcc4.4.1, llvm-g++ icc10.1, icc11.1 Windows (XP, Vista) VC++7.1, VC++9 Cygwin gcc3.4, 4.3 Solaris + OpenSolaris CC5.2 gcc3.4, gcc4.4 34ROOT: design,development, evolution Support for many compilers adds robustness at a minor development cost. It facilitates the migration to new compiler versions. René Brun, ACAT2010, Jaipur

35. Distribution Stats 35ROOT: design,development, evolution More than 20000 users 35000 FTP-only distributions of binaries per month. Many more from SVN source or AFS access René Brun, ACAT2010, Jaipur

36. How did we reach the current situation Rien ne sert de courir, il faut partir a point Jean de la Fontaine René Brun, ACAT2010, JaipurROOT: design,development, evolution36

37. one cannot forget the past Systems like ROOT and G4 have been strongly influenced par their predecessors, by politics, by sociology. Like Jumbo jets, large systems that required a lot of manpower evolve with time with the pros and cons of their inertia. Once a system is massively used, people expect backward and forward compatibility. However ROOT in 2010 is very much different of ROOT in 1995 or 2000 or 2005 René Brun, ACAT2010, JaipurROOT: design,development, evolution37

38. Software in HEP René Brun, ACAT2010, JaipurROOT: design,development, evolution38 1970198019902010 cernlib ===============-------------------------………………....... paw---====================-----------------….... zebra---=================------------------------…….. geant1,2 ==-- hbook===============================------------------------…………... geant4----=================== piaf==== geant3 ======================------------------………... ROOT---=================== JAS--===----…... most systems used over 2,3 or 4 decades

39. Main parameters who shaped ROOT Long experience with GEANT and PAW Reuse of components Concept of packages within a framework Optimal for batch, optimal for interactivity User support quasi online René Brun, ACAT2010, JaipurROOT: design,development, evolution39

40. CERNLIB 1970: main.f links with cernlib.a where cernlib is a collection of independent subroutines. 1975: subroutines are grouped into packages identified by the letters of the alphabet, eg Y250 1980: cernlib accepts large packages, eg hbook, minuit,gd3 1985: cernlib is a collection of large packages, mathlib, kernlib,packlib,graflib,pawlib,geant3 René Brun, ACAT2010, JaipurROOT: design,development, evolution40

41. hbook : 1973  20xy first large package in cernlib histogram-oriented system including visualization (from line-printer to graphics) and persistency. interface with Minuit and Fumili interface with zbook in 1975, then zebra in 1983 extended in 1985 to support row-wise ntuples and column- wise ntuples in 1990. the heart of PAW René Brun, ACAT2010, JaipurROOT: design,development, evolution41

42. geant3 : 1980-  20xy successor of geant1 (1975), geant2(1978) includes a powerful geometry package used to model all existing HEP detectors. interfaces with external hadronic shower packages like tatina,gheisha,fluka. includes visualisation: detector cross-sections, event displays. includes IO for geometry and events (zebra) René Brun, ACAT2010, JaipurROOT: design,development, evolution42

43. PAW : 1984  20xy started by a committee (design for the past) quickly reshaped in 1985 with KUIP, HIGZ, ntuples query processor: ntuple/plot 10.px analysis patterns indicates row-wise  column-wise storage and access. developments using zebra RZ IO system more and more graphics options René Brun, ACAT2010, JaipurROOT: design,development, evolution43

44. PIAF : 1992-1994 PAW Parallel Interactive Analysis Facility very important step to understand the issues with parallelism with the IO sub system and the implications on the users analysis code. very important to understand the balance between CPU, disk IO, network IO. René Brun, ACAT2010, JaipurROOT: design,development, evolution44

45. software crisis 1992-1994 It was clear in 1992 that our Fortran-based systems will be impacted by new languages and new techniques. Many people thought that Fortran was the future. MOOSE project investigating languages (C++, Eiffel) and techniques (UML, Rose, OO-design) commercial software or home-grown dilemma OODBMS at the horizon René Brun, ACAT2010, JaipurROOT: design,development, evolution45

46. software crisis 1992-1994 -more many computer scientists in the discussion most of them with no experience existing experience with large software neglected as well as feedback coming from experienced physicists west coast syndrome Babar as guinea pig (C++ and OODBMS) Revolution, not Evolution René Brun, ACAT2010, JaipurROOT: design,development, evolution46

47. Towards Objectivity 1994-2000 Gurus predict that OODBMS will replace RDBMS systems in 2000. Post-mortem influence of defunct SSC dream of a central data base accessed world-wide at the bit level for writing and reading. Don’t worry about containers structure, CPU and network performance; the technology will solve your problems. Most physicists trust gurus for reaching Nirvana René Brun, ACAT2010, JaipurROOT: design,development, evolution47

48. 1994-2000 Lost in the clouds before the grid and the clouds Experienced physicists busy with LHC exp design. It took many years (of frustration) to move them from good old Fortran to the magic of OO and C++ catastrophic LCB (LHC Computing Board) committee with always the same referees with no experience in the new technology. As a consequence the LCB has no choice but trusting the gurus. Despite early signs of problems, gurus and referees persist in their recommendations. René Brun, ACAT2010, JaipurROOT: design,development, evolution48

49. 1995-1998 ROOT is born in the meadows While many people were actively working in committees and expecting magic from Objectivity, a small group of anarchists is exiled in the CERN 2 meadows. free of committees and not spending time only with UML diagrams, but concentrating their work on key issues and delivering frequent releases of prototypes tested in real life experiments (eg NA49). While the first ROOT prototype was a C++ version of PAW, it was rapidly realized that the key problem was with a general object persistency solution much different than the Objectivity solution. René Brun, ACAT2010, JaipurROOT: design,development, evolution49

50. Original Objectives in 95 Continue the tradition started with CERNLIB to develop packages targeted to the needs of our experiments and with a long life time. Eg hbook (36 years), geant3 (28), paw(25): still in use! Focus on data analysis (PAW in the oo world) Open Source model to get contributions and feedback as soon as possible. First ROOT prototype for NA49 in 1995, rapidly followed by many more adopters. Bottom-up (user-driven) instead of top-down (committee) approach. Users workshops. ROOT: design,development, evolution50René Brun, ACAT2010, Jaipur

51. ROOT in 1995 During the first 3 months of 1995 we prototyped several versions of the histograming system, including a version based on templates. In October 1995 we presented a first prototype with histograms, functions, graphics (a la PAW). This prototype included a command line interface supporting very primitive calls to ROOT classes member functions. This prototype generated a lot of attention but also fears. René Brun, ACAT2010, JaipurROOT: design,development, evolution51

52. ROOT in 1996 Instead of our very basic interpreter, we decided to use CINT developed since 1991 by Masa Goto from HP Japan. We used CINT not only as an interpreter, but also as a parser for the ROOT header files to generate automatically a dictionary such that: one could call class member functions from C++ interpreted scripts one could stream objects to a buffer with explicit streamers. ROOT version 1 included also many more classes and better graphics. René Brun, ACAT2010, JaipurROOT: design,development, evolution52

53. ROOT in 1997 The IO Streamer functions generated by rootcint were quite simple to understand and were efficient. However, we realized soon that classes were evolving. We had to introduce class versions and the possibility for users to write their own custom streamers where they could take into account the class schema evolution. At the same time, we made rootcint more general such that it could parse many more classes, in particular non- ROOT classes developed by our experiments guinea-pigs. René Brun, ACAT2010, JaipurROOT: design,development, evolution53

54. ROOT in 1998 We released version 2 with more features (IO, math, graphics and more sophistication in Trees, our main container for event data. In fall 1998 ROOT was selected by FNAL & RHIC as: the persistency solution the data analysis solution This decision generated a big chock-wave in the establishment. It also generated more work for us. René Brun, ACAT2010, JaipurROOT: design,development, evolution54

55. ROOT in 1999 The ROOT Streamers were very efficient. However the library containing the Streamers was required when reading the data sets. While this requirement was not a problem when reading complex object collections and the classes used to write were necessary in the read phase, it was a serious drawback for files with simple Trees like PAW ntuples. At this point we implemented an automatic system to stream data using the information in the dictionary instead of using the generated C++ Streamers René Brun, ACAT2010, JaipurROOT: design,development, evolution55

56. StreamerInfo With the automatic streaming, we also saved to the ROOT file the meta data information describing all classes such that ROOT files are now self-describing. With time and in the following years we continued to optimize this system: to support the full C++ language in general. to support the full C++ language in general. to support more sophisticated cases of class schema evolution. René Brun, ACAT2010, JaipurROOT: design,development, evolution56

57. ROOT vs Objectivity In 99.99% cases people write a file once and read it many times. No need for a locking process when reading. ROOT files are self-described and independent of the location where they are processed. ROOT files are compressed. Typical compression factors for LHC experiments is between 3 and 5. ROOT files are machine independent René Brun, ACAT2010, JaipurROOT: design,development, evolution57

58. ROOT vs Objectivity 2 ROOT does not stream transient data members ROOT can stream collections member-wise, optimizing both speed and disk space. ROOT can read files containing different class versions: automatic class schema evolution. ROOT has a complete query/visualization system. ROOT files can be processed in parallel with PROOF ROOT was processing files orders of magnitude faster ROOT has a powerful caching mechanism allowing to read files very efficiently in wide area networks René Brun, ACAT2010, JaipurROOT: design,development, evolution58

59. Directions René Brun, ACAT2010, JaipurROOT: design,development, evolution59

60. Our Challenges Keep a stable core, improving robustness At the same time follow the technology changes Distributed and parallel computing Multi-core systems, GPU processing (OpenCL) New version of C++, etc CINT to LLVM Improve algorithms (quality and speed) Improve doc, user interface, user support Add support for new communities, LC,etc And, of course, provide a robust and efficient data analysis tool for the LHC experiments. ROOT: design,development, evolution60René Brun, ACAT2010, Jaipur

61. ROOT: design,development, evolution61 Will have to deal with many shared libs Only a small fraction of code used Problem decomposition

62. René Brun, ACAT2010, JaipurROOT: design,development, evolution62 ROOTin1995 ROOTin2010 10 shared libs 200 classes 100 shared libs 2000 classes PAW model Plug-in manager Some Facts

63. René Brun, ACAT2010, JaipurROOT: design,development, evolution63 many_x.f -> many_x.o many_x.o + some_libs.a + many_libs.so -> x.exe x.exe Input.dat (free format) Output.log Objectivity? ROOT ? Zebra file RFIO Executable module in 2000

64. René Brun, ACAT2010, JaipurROOT: design,development, evolution64 x.exe Config.C (interpreter) Output.log ROOT files Oracle Mysql Dcache castor xrootd a.so b.so u.so Shared libs dynamically loaded/unloaded by the plug-in manager LAN Executable module in 2010

65. René Brun, ACAT2010, JaipurROOT: design,development, evolution65 x.exe Config.C (interpreter) Output.log ROOT files Oracle Mysql Cache Proxy manager http a.cxx http b.cxx http u.cxx Local shared libs dynamically Compiled/loaded/unloaded from a source URL WAN x.exe ROOT buffers local cache Multi-threaded Core executor Executable module in 2020 ?

66. The LLVM system We are having big hopes with LLVM (open source project with people from Apple, Adobe, Google, etc). LLVM is a gcc compatible compiler, parser and JIT (Just In Time) compiler. Our idea is that the interpreter will be the compiler itself (C++0x compliant) and offering many additional possibilities like quality and speed for important classes like TFormula, TTreeFormula, etc We have a working prototype today, but a fully operational use of LLVM in ROOT is probably 2 years away. René Brun, ACAT2010, JaipurROOT: design,development, evolution66

67. René Brun, ACAT2010, JaipurROOT: design,development, evolution67 The GRID is a parallel engine. However it is unlikely that you will use the GRID software on your 32-core laptop. There is a lot of work to be done within ROOT to use parallelism internally or making ROOT-based applications more easily parallelizable. Think Top->Down and Bottom->Up Coarse grain: job, event, track Fine grain vectorization Design for Parallelism

68. René Brun, ACAT2010, JaipurROOT: design,development, evolution68 100,000 computers in 1000 locations 5,000 physicists in 1000 locations WAN LAN Towards a distributed system

69. René Brun, ACAT2010, JaipurROOT: design,development, evolution69 Few big users submitting many long jobs (Monte Carlo, reconstruction) They want to run many jobs in one week Many users submitting many short jobs (physics analysis) They want to run many jobs in one hour or less GRID: Users profile

70. Difficult to fit all sizes Like the Taj Mahal, ROOT has many different facets and the visitor may have different impressions depending on his viewpoint and time of day. We try to do our best to satisfy very heterogeneous users. Some use ROOT as a framework, others as a simple tool, others as a set of libraries. René Brun, ACAT2010, JaipurROOT: design,development, evolution70

71. Taj Mahal & ROOT The Taj Mahal combines elements from Persian, Indian, Islamic architectural styles, an integrated complex of structures. Building began around 1632 and was completed around 1653, and employed thousands of artisans and craftsmen. ROOT combines ideas and techniques that proved to be successful in the geant3, zebra and PAW eras. Building began in 1995 and in 2010 is still under construction by many motivated people. René Brun, ACAT2010, JaipurROOT: design,development, evolution71

72. Many Thanks I have been pleased to follow the ACAT meetings since the beginning in Lyon (1989 ?) Many thanks to my colleagues organizing this event. This is my last talk at ACAT as ROOT project leader. I take this opportunity to thank all my colleagues who are working hard in this project and I wish them a healthy future. René Brun, ACAT2010, JaipurROOT: design,development, evolution72

73. Input/Output: Major Steps 73 ROOT: design,development, evolution parallel merge generalized schema evolution member-wise streaming for STL collections member-wise streaming for TClonesArray automatic streamers from dictionary with StreamerInfos in self-describing files streamers generated by rootcint User written streamers filling TBuffer René Brun, ACAT2010, Jaipur

74. CMZ…………………………………CVS………………………………………………………….SVN…………… Major Technical Steps 74 P R O O F Reflex From SEAL to ROOT ROOT: design,development, evolution LLVM CINT7 GL/EVE RooFit TMVA MathLibs Reflex From SEAL to ROOT Trees automatic split GUI based on signal/slot s I/O based on dictionaries in memory Graphics based on TVirtualX CINT René Brun, ACAT2010, Jaipur

75. Testing, Nightly builds Several thousand tests run every night on many platforms. Code checking with a coding convention rule checker Code coverage to be improved, but already much better than any other package reviewed. Frequent valgrinding and profiling ROOT: design,development, evolution75René Brun, ACAT2010, Jaipur

76. PROOF-LITE PROOF-LITE 76  PROOF optimized for single many-core machines  Zero configuration setup. No config files and no daemons  Workers are processes and not threads for robustness  Like PROOF it can exploit fast disks, SSD’s, lots of RAM, fast networks and fast CPU’s  Once your analysis runs on PROOF Lite it will also run on PROOF (with exactly the same user code) ROOT: design,development, evolution ROOT: one core onlywith PROOF-LITE René Brun, ACAT2010, Jaipur

77. ROOT: design,development, evolution77 many_x.f -> many_x.o many_x.o + many_libs.a -> x.exe x.exe Input.dat (free format ) Output.log portable Zebra file Executable module in 1990