Work report Xianghu Zhao Nov 11, 2014

Status of User Jobs 8000 jobs total 4732 jobs successful 1.2 TB data transferred to IHEPD-USER

Status of User Jobs Running Jobs Input Sandbox Size

User Requirements Use custom generator for simulation Do simulation + reconstruction + analysis in one job Common point: They need some custom packages which does not exist in the official BOSS release

Solution User compile the packages locally before submission Add corresponding .so lib files to the input sandbox in the ganga script Add “pwd” to the environment “LD_LIBRARY_PATH” before Boss jobs running Add other user files needed by the job to the input sandbox and change the paths to relative ones

Generator Custom Files All user *.so files could be found as soft link under the directory $USERWORKAREA/InstallArea/x86_64-slc5-gcc43-opt/lib/ These two files from package BesEvtGen are put in the input sandbox libBesEvtGen.so libBesEvtGenLib.so

Old Version Problems libcurl.so.3 not found for SL6 nodes This happens in the reconstruction on SL6 nodes Boss version from 664p01 has included this file sqlite memory leak (“by event” jobs will fail after a period of time) Boss version from 664p01 has solved this problem MdcTunningSvc can only use mysql (ignore the settings in the jobOption) Boss version from 664 can use sqlite BesEventMixer can not set custom random trigger directory Boss version from 664 can support custom directory

Patchs with Version There too many different versions and it is not good to hard code all the situations in the GangaBoss code Use patches which are automatically included in the input sandbox and executed before Boss job running /afs/.ihep.ac.cn/bes3/offline/ExternalLib/gangadist/scripts/*.patch For each Boss version, configure the patches needed in a configuration file (python dict format) /afs/.ihep.ac.cn/bes3/offline/ExternalLib/gangadist/scripts/Boss.conf

Adding Analysis Support Add a new option “anaoptsfile” for the Boss application app = Boss(version=bossVersion, optsfile=optionsFile, recoptsfile=recOptionsFile, anaoptsfile=anaOptionsFile) New data type “root” for analysis output file Add new files to the input sandbox anaoptions.pkl anadata.py, anadata.opts Change the splitter to generate the “anadata” files Add analysis to workflow Save analysis logs to the output sandbox anabosslog, anabosserr

Analysis Job Custom Files File of user analysis package DCDCAlg libDCDCAlg.so There are two more packages used by An Fenfen DTagTruthMatchSvc libDTagTruthMatchSvcLib.so libDTagTruthMatchSvc.so McDecayModeSvc libMcDecayModeSvcLib.so libMcDecayModeSvc.so Also two files needed by the analysis deltaE.txt, Mbc.txt User need to change the path in the job option file to relative path

Some Problems User could only submit about 1100 jobs and the submission will fail Process “dirac-server” occupies too much CPU time (about 75%) and it is killed after one hour Submission time is longer than before Need to upload 8.6 M files to input sandbox The size of input sandbox storage increased very fast 8.6 M for one job 67 G total now

Future Plan: Upload Big Files to SE Upload all user release .so files to SE before submitting jobs to DIRAC Tell the boss script which files to download from SE On the node, the boss script download the files before executing boss jobs Upload files to /bes/user/x/xxx/Upload Change the filename to md5 hex format (7086c05f3cead1809dc8d1ac0f5c467c) Calculate and check md5 before uploading to avoid uploading the same file repeatedly Tell the script the real filename and md5 for download

DIRAC Server Upgrade Separate the mysql database to a standalone server badger01 <--> besdirac02 dirac-code -> mysql server Web server: Also on dirac-code? A virtual machine? Another server (with CVMFS, badger02)?