1. Work report
Xianghu Zhao
Nov 11, 2014

2. Status of User Jobs 8000 jobs total 4732 jobs successful
1.2 TB data transferred to IHEPD-USER

3. Status of User Jobs
Running Jobs
Input Sandbox Size

4. User Requirements Use custom generator for simulation
Do simulation + reconstruction + analysis in one job
Common point:
They need some custom packages which does not exist in the official BOSS release

5. Solution User compile the packages locally before submission
Add corresponding .so lib files to the input sandbox in the ganga script
Add “pwd” to the environment “LD_LIBRARY_PATH” before Boss jobs running
Add other user files needed by the job to the input sandbox and change the paths to relative ones

6. Generator Custom Files
All user *.so files could be found as soft link under the directory
$USERWORKAREA/InstallArea/x86_64-slc5-gcc43-opt/lib/
These two files from package BesEvtGen are put in the input sandbox
libBesEvtGen.so
libBesEvtGenLib.so

7. Old Version Problems libcurl.so.3 not found for SL6 nodes
This happens in the reconstruction on SL6 nodes
Boss version from 664p01 has included this file
sqlite memory leak (“by event” jobs will fail after a period of time)
Boss version from 664p01 has solved this problem
MdcTunningSvc can only use mysql (ignore the settings in the jobOption)
Boss version from 664 can use sqlite
BesEventMixer can not set custom random trigger directory
Boss version from 664 can support custom directory

8. Patchs with Version
There too many different versions and it is not good to hard code all the situations in the GangaBoss code
Use patches which are automatically included in the input sandbox and executed before Boss job running
/afs/.ihep.ac.cn/bes3/offline/ExternalLib/gangadist/scripts/*.patch
For each Boss version, configure the patches needed in a configuration file (python dict format)
/afs/.ihep.ac.cn/bes3/offline/ExternalLib/gangadist/scripts/Boss.conf

9. Adding Analysis Support
Add a new option “anaoptsfile” for the Boss application
app = Boss(version=bossVersion, optsfile=optionsFile, recoptsfile=recOptionsFile, anaoptsfile=anaOptionsFile)
New data type “root” for analysis output file
Add new files to the input sandbox
anaoptions.pkl
anadata.py, anadata.opts
Change the splitter to generate the “anadata” files
Add analysis to workflow
Save analysis logs to the output sandbox
anabosslog, anabosserr

10. Analysis Job Custom Files
File of user analysis package DCDCAlg
libDCDCAlg.so
There are two more packages used by An Fenfen
DTagTruthMatchSvc
libDTagTruthMatchSvcLib.so
libDTagTruthMatchSvc.so
McDecayModeSvc
libMcDecayModeSvcLib.so
libMcDecayModeSvc.so
Also two files needed by the analysis
deltaE.txt, Mbc.txt
User need to change the path in the job option file to relative path

11. Some Problems
User could only submit about 1100 jobs and the submission will fail
Process “dirac-server” occupies too much CPU time (about 75%) and it is killed after one hour
Submission time is longer than before
Need to upload 8.6 M files to input sandbox
The size of input sandbox storage increased very fast
8.6 M for one job
67 G total now

12. Future Plan: Upload Big Files to SE
Upload all user release .so files to SE before submitting jobs to DIRAC
Tell the boss script which files to download from SE
On the node, the boss script download the files before executing boss jobs
Upload files to /bes/user/x/xxx/Upload
Change the filename to md5 hex format (7086c05f3cead1809dc8d1ac0f5c467c)
Calculate and check md5 before uploading to avoid uploading the same file repeatedly
Tell the script the real filename and md5 for download

13. DIRAC Server Upgrade
Separate the mysql database to a standalone server
badger01 <--> besdirac02
dirac-code -> mysql server
Web server:
Also on dirac-code?
A virtual machine?
Another server (with CVMFS, badger02)?