1. 1 Xavier Prudent - LAPP BaBar Collaboration Meeting June 2006 - Montreal Andreas Höcker (ATLAS), Kai Voss (ATLAS), Helge Voss (LHCb), Jörg Stelzer (BaBar), Peter Speckmayer (CERN)

2. 2 Need of a global tool that would : provide with the most common MV methods do both the training and evaluation of these methods enable easy computation of systematics Multi-variable analysis widely used in HEP (LEP, BaBar, Belle, D0, MiniBooNE, …) Common Reproaches to Multi-Variable Methods : “Black box methods” In case of correlations cuts are not transparent anymore Training sample may not describe correctly the data Creates no bias, only bad performance Systematics ? Need a control sample Independent & uneasy implementations …

3. 3 “TMVA” means Toolkit for Multivariate Analysis MV Methods available so far : Rectangular cut optimization Correlated Likelihood estimator (PDE) Multi-dimensional likelihood estimator (PDE) Fisher & Mahalanobis discriminant H-Matrix ( χ 2 estimator) Neural Network (2 different implementations) Boosted decision tree Root package written by Andreas Höcker, Kai Voss, Helge Voss, Jörg Stelzer, Peter Speckmayer for the evaluation of MV methods in parallel of an analysis TMVA provide training, testing & evaluation of these methods A dedicated class enables to plug the training results in your favorite analysis

4. 4 Cut Optimization  Scan in signal efficiency for highest background rejection Correlated & de-correlated likelihood  PDE approach, generalization to multi dimension likelihood, Output transformed by an inverse Fermi function (  less peaked) De-correlation possible with the square root of the covariance matrix Fisher discriminant and H-matrix  Classical definitions Neural Network  2 NNs, both multi-layer perceptrons with stochastic learning Clermont Ferrand ANN (used for ALEPH Higgs analysis) TMultiLayerPerceptron (ANN from ROOT) (Boosted) Decision trees  Inspired from MiniBooNE Sequential applications of cuts

5. 5 Each event has a weight W i (= 1 to start) Training Split until minimal #event reached or limit of purity reached Final node = “leaf” : if P>P min “signal leaf” If P<P min “background leaf” Boosting : if a signal event is on a bkg leaf or if a bkg event is on a signal leaf  weight modified Then training re-performed with the new weights (x1000 trees) S/B 52/48 Var1 > x1 Var1 ≤ x1 Var2 > x2Var2 ≤ x2 S/B 4/37 S/B 48/11 S/B 2/10 S/B 46/1 Testing Start from root node, events go through the 1000 boosted trees Each time an event ends on leaf signal or background signal its weight is modified (↔ Neural Net output) (  smoother output than classical discrete output) Cut on the variable that optimizes the separation, based on the purity P (=“node”) Optimization by scanning, Genetic Algorithm soon What is a Boosted Decision Tree ?

6. 6 How to get and use TMVA ?

7. 7 How to download TMVA ? Get a tgz file : from TMVA website http://tmva.sourceforge.net then click onhttp://tmva.sourceforge.net Via cvs : cvs –z3 –d:pserver:anonymous@cvs.sourceforge.net:/cvsroot/tmva co –P TMVA  Automatic creation of 6 directories src/ Source for TMVA library example/ example of how to use TMVA lib/ TMVA library once compiled reader/ all functionalities to apply MV weights macros/ root macro to display results development/ working & testing directories For your own analysis : > cp example myTMVA  Modify “makefile” for compilation in /myTMVA Download

8. 8 /home cd TMVA /home/TMVA source setup.csh /home/TMVA cd src/ /home/TMVA/src make Include TMVA/lib in PATH Compile the librairy  libTMVA.so How to compile TMVA ? Detailed Steps for the Example

9. 9 How to choose the MV method I want ? … Bool_t Use_Cuts = 1; Bool_t Use_Likelihood = 0; Bool_t Use_LikelihoodD = 0; Bool_t Use_PDERS = 0; Bool_t Use_HMatrix = 0; Bool_t Use_Fisher = 1; Bool_t Use_CFMlpANN = 1; Bool_t Use_TMlpANN = 0; Bool_t Use_BDT_GiniIndex = 0; Bool_t Use_BDT_CrossEntro = 0; Bool_t Use_BDT_SdivStSpB = 0; Bool_t Use_BDT_MisClass = 0; … You just have to “switch” on or off the Booleans ! Here for instance I will compare Cuts, Fisher, Neural Net CFM /home/TMVA/src cd../examples Go to exemple/ directory and open You will find a list of available methods (Booleans) Switch to 1/0 the method you want/don’t want TMVAnalysis.cpp

10. 10 How to point to TMVA the training samples & variables ? In /examples/data : toy_sig.dat bkg_toy.dat How to point the variables (example with 4 variables) : How to point the input ascii files : In TMVAnalysis.cpp Both ascii or root files can be used as input Creation of the factory object

11. 11 How to change training options ? In TMVAnalysis.cpp Training cycles, #hidden layer, #neurons per layer  Description of every options in class BookMethod factory->PreparTrainigAndTestTree( mycut, 2000, 4000 ); #events used :trainingtesting

12. 12 How to I run TMVA ? /home/TMVA/src cd../examples /home/TMVA/examples make /home/TMVA/examples TMVAnalysis “myOutput.root” Name of output root file What does it create ? Some weights files for each trained MV methods in weight/ A root file in main directory with the MV outputs and efficiencies How to look at the results ? Use the nice ROOT macros in directory macros/ /home/TMVA/examples root –l /home/TMVA/examples.L../macros/efficiencies.C /home/TMVA/examples efficiencies("MyOutput.root") Plots created in directory plots/

13. 13 Which ROOT macros are available ? (1) variables.C  Distributions of input variables

14. 14 Which ROOT macros are available ? (2) correlations.C  Colored correlation matrix of input variables Numeric values displayed during TMVA running

15. 15 Which ROOT macros are available ? (3) mvas.C  Outputs of MV methods

16. 16 Which ROOT macros are available ? (4) efficiencies.C  Background rejection vs. Signal efficiency Direct comparison of all MV methods !

17. 17 I have trained the MV method I want … I have the weight files … How to use this MV method in my analysis ? Detailed example is Dedicated class The next slide shows what must be included in your analysis program … Work in progress (being implemented in ROOT) thus possible differences with later version … TMVA/reader/TMVApplication.cpp reader/TMVA_reader.hh

18. 18 #include “TMVA_reader.h” using TMVApp::TMVA_Reader; Void MyAnalysis() { vector inputVars; inputVars.push_back( "var1" ); inputVars.push_back( "var2" ); inputVars.push_back( "var3" ); inputVars.push_back( "var4" ); TMVA_Reader *tmva = new TMVA_Reader( inputVars ); tmva->BookMVA(TMVA_Reader::Fisher, “TMVAnalysis_Fisher.weights"); vector varValues; varValues.push_back( var1 ); varValues.push_back( var2 ); varValues.push_back( var3 ); varValues.push_back( var4 ); double mvaFi = tmva->EvaluateMVA( varValues, TMVA_Reader::Fisher ); delete tmva;} [1] Include the reader class [2] Create an array of the input variables names (here 4var) [3] Create the reader class [4] Read the weights and build the MV tool [5] Create an array with the input variables values [6] Compute the value of the MV, it is the value you will cut on

19. 19 TMVA is already used by several AWG in BaBar Group Dalitz Charmless UK : TMVA Fisher for continuum rejection in the Dalitz-plot analyzes of K s    - and K +    + ( BADs: 1376 and 1512 ). Use of 11 input variables, pictures taken from BAD 1376

20. 20 Group D0h0 : TMVA ClmF NN for continuum rejection in the measurement of the BFs of the color suppressed modes B 0  D 0 h 0 (h 0 = ω, η, η’, ρ, π 0 ) and in the measurement of CKM β angle Use of 4 input variables

21. 21 Measurement of sin(2α) with B  ρπ  Uses Clermont-Ferrand NN to get rid of combinatory background Measurement of CKM angle γ with GLW method (Emmanuel Latour – LLR)  Uses Fisher to get rid of combinatory background Signal = MC signal B  D*K, D*  D0p0, D0  Kp Background = MCs udsc

22. 22 A similar tool has been developed by Ilya Narsky ( StatPatternRecognition ) What to keep in mind about TMVA ? A powerful Multivariate toolkit with 12 different methods (more are coming) User friendly package from training to plots ! Already used in BaBar Comparison possible & easy between the different MV methods C++ & Root functionalities, announced in ROOT version V5-11-06 http://root.cern.ch/ Have a look at http://tmva.sourceforge.net/ !!http://tmva.sourceforge.net/ Talk by Kai Voss at CERN http://agenda.cern.ch/askArchive.php?base=agenda&categ=a057207&id=a057207s27t6/transparencies TMVA Tutorial https://twiki.cern.ch/twiki/bin/view/Atlas/AnalysisTutorial1105#TMVA_Multi_Variate_Data_Analysis Physics analysis HN advertisement http://babar-hn.slac.stanford.edu:5090/HyperNews/get/physAnal/2989.html

23. 23 Back Up Slides

24. 24 Available Options For Every Methods in TMVAnalysis.cpp Rectangular cut optimization Correlated Likelihood estimator (PDE) Multi-dimensional likelihood estimator (PDE) Fisher & Mahalanobis discriminant H-Matrix ( χ 2 estimator) Neural Network (2 different implementations) Boosted decision tree

25. 25 Rectangular cuts Option for each variables - "FMax" : ForceMax (the max cut is fixed to maximum of variable i) - "FMin" : ForceMin (the min cut is fixed to minimum of variable i) - "FSmart": ForceSmart (the min or max cut is fixed to min/max, based on mean value) - Adding "All" to "option_vari", eg, "AllFSmart" will use this option for all variables - if "option_vari" is empty (== ""), no assumptions on cut min/max are made TMVA method Method of cut - "MC" : Monte Carlo optimization (recommended) - "FitSel": Minuit Fit: "Fit_Migrad" or "Fit_Simplex" - "FitPDF": PDF-based: only useful for uncorrelated input variables factory->BookMethod( "MethodCuts", “Method : nBin : OptionVar1: … :OptionVarn" ); # bins in the hist of efficiency S/B

26. 26 Likelihood factory->BookMethod( "MethodLikelihood", “TypeOfSpline : NbSmooth : NbBin : Decorr"); TMVA method Option for decorrelation or not - “NoDecorr” – do not use square-root-matrix to decorrelate variable space - “Decorr” – decorrelate variable space Which spline is used for smoothing the pdfs “Splinei” [i=1,2,3,5] How often the input histos are smoothed Average num of events per PDF bin to trigger warning

27. 27 Fisher Discriminant and H matrix factory->BookMethod( "MethodFisher", "Fisher" ); TMVA method Which method -“Fisher” - “Mahalanobis” (another definition of distance) factory->BookMethod( "MethodHMatrix" ); TMVA method

28. 28 Artificial Neural Network factory->BookMethod( “WhichANN”, “NbCycles:NeuronsL1:NeuronsL2:…:NeuronsLn" ); Which type of NN - “MethodCFMlpANN” Clermond Ferrand NN, used for Higgs search in ALEPH - “MethodTMlpANN” ROOT NN Number of training cycles Number of neurons in each layer The 1st layer has necessarily as many neurons as input variables

29. 29 Boosted Decision Trees factory->BookMethod( "MethodBDT", “nTree : BoostType : SeparationType : nEvtMin : MaxNodePurity : nCuts”); TMVA method Method for evaluating the misclassification - GiniIndex - CrossEntropy - SdivSqrtSplusB - MisClassificationError Number or trees Method of boosting - AdaBoost - EpsilonBoost Minimum Number of events in a node (leaf criteria) Higher bound for leave or intermediate node Number of steps in the optimization of the cut for a node