CHEP ' 2003David Chamont (CMS - LLR)1 OVAL A testing tool for CMS software

CHEP ' 2003David Chamont (CMS - LLR)2 General Description Original goal : help to detect automatically the changes in the physics performances of the CMS software used for data production. Current features of the tool : compile the provided test programs, run them in specified runtime conditions (shell variables + auxiliary files), compare the output with references. User interface is CVS-like, configuration files are XML-like. It is written in PERL and can be configured for any build system (scram, make).

CHEP ' 2003David Chamont (CMS - LLR)3 Outline General description. Configuration : OvalFile. Commands : build/run/diff. Results : log files. Other features. Summary. Plans.

CHEP ' 2003David Chamont (CMS - LLR)4 Configuration : OvalFile MaxEvents = 500 Random:Seeds = 0 3

CHEP ' 2003David Chamont (CMS - LLR)5 Commands : build/run/diff… cms038:ElectronPhoton> ls Makefile CVS Clusters.cpp Clusters.ref Electrons.cpp Electrons.ref EnergyFlow.cpp OvalFile

CHEP ' 2003David Chamont (CMS - LLR)6 Commands : …build/run/diff… cms038:ElectronPhoton> oval prod Clusters: build, run, diff. Electrons: build, run, diff (DIFFS).

CHEP ' 2003David Chamont (CMS - LLR)7 Commands : …build/run/diff cms038:ElectronPhoton> ls Makefile CVS Clusters.cpp Clusters.log Clusters.ref Electrons.cpp Electrons.log Electrons.ref EnergyFlow.cpp OvalFile electrons.hbook

CHEP ' 2003David Chamont (CMS - LLR)8 Results : log files… [oval build] ======================================== [oval build] make Electrons [oval build] ======================================== /usr/local/gcc-alt /bin/c++ -O2 -pthread -rdynamic -L/afs/cern.ch/user/c/chamont/ORCA_5/lib/Linux__2.2 -L/afs/cern.ch/cms/Releases/ORCA_5/lib/Linux__2.2 -L/afs/cern.ch/sw/lhcxx/specific/redhat61/gcc /3.6.2/lib -L/afs/cern.ch/cms/cmsim/cms122/lib/i386_linux22 -L/cern/2001/lib Electrons.o -o Electrons -lElectronFacilities -lEgammaH4Support -lEgammaL1Tools … -lmathlib -lblas -lkernlib -lpythia6152 -lz -lnsl -ldl -lg2c -lm

CHEP ' 2003David Chamont (CMS - LLR)9 Results : …log files… [oval run] ========================================== [oval run] LD_LIBRARY_PATH = [oval run] /afs/cern.ch/user/c/chamont/ORCA_5/lib/Linux__2.2 [oval run] :/afs/cern.ch/cms/Releases/ORCA_5/lib/Linux__2.2 [oval run] :/afs/cern.ch/sw/lhcxx/i386_linux22/Objectivity/5.2.1/lib [oval run] FEDERATION = cmsuf01::/cms/jet0900/jet0900.boot [oval run] DATASET = eg_ele_pt15... [oval run].orcarc: [oval run] MaxEvents = 500 [oval run] Random:Seeds = 0 3 [oval run] ========================================= Welcome to COBRA...

CHEP ' 2003David Chamont (CMS - LLR)10 Results : …log files [oval diff] ======================================== [oval diff] diff line : /^OVAL:/ [oval diff] diff number : /^OVAL NUM:.* is (.*)$/ ~5% [oval diff] ======================================== ref#1452 != log#2053 < OVAL: 12 electrons --- > OVAL: 11 electrons ref#1972 !~ log#2592 (>5%) < OVAL NUM: energy mean is > OVAL NUM: energy mean is

CHEP ' 2003David Chamont (CMS - LLR)11 Other Features Oval is not restrained to programs. For complex testing tasks, especially when databases are involved, one can also handle some scripts (no build step needed). The same program or script can be executed several times with different arguments, and/or within different environments. If there are many tests, one can organize them in a hierarchy of Unix directories, and create OvalFiles at any level. The results of an oval command can be mailed to watchers as defined in the configuration.

CHEP ' 2003David Chamont (CMS - LLR)12 Summary Scan the output, whatever it is. Can be combine with any build system. Very easy to use, except the tuning of regular expressions (more examples available soon). No helper code provided. The users still have to write their test programs. CMS use it for regression tests ; some developers run it daily on their code (kind of extreme programming !) LCG plans to use it together with CppUnit.

CHEP ' 2003David Chamont (CMS - LLR)13 Plans Improve infrastructure & support : –improve internal documentation, –add a tutorial, –progressively move to LCG savannah server. React to users’ feedback –enable the users to provide their own diff command, –add a reuse mechanism for the environments, –automatic generation of trivial test programs for a given set of header files.