1. CRAB and local batch submission
Xiaomei Zhang
CMS IHEP Group Meeting  
Sep

2. Outline Job submission on Grid and PC farm CRAB functions
CRAB architecture
Embed local farm part in CRAB
Local farm job submission usage

3. Grid and PC farm CRAB(CMS Remote Analysis Builder) Grid and farm
A python user interface to grid, to simplify the process of creation, submission and monitoring CMS analysis jobs into grid
User interface to farm?
Grid and farm
Farm is local, Grid is a remote “farm system”
Process is similiar:
Creat (UI)->submit(UI)->schedule(CE, pbs)->running(WN)->return result to UI
Details such as commands, parameters, log files are different:
Edg-job-submit, edg-job-status….
Qsub, qstat……
Other cases:
Atlas , argo…

4. Crab functions(1) Locate dataset Create jobs
Information of the dataset by DBS, DLS.eg. blocks, event number
Especially, LFN and PFN of the dataset
Create jobs
proxy renewal
Job configuration by cfg files(crab.cfg) and command option
Job splitting by user configuration and data distribution
Create run scripts(cmssw.sh) in work nodes
Set up running environment, copy or access the data files, prepare pset.cfg files, start data processing……
由于不是一个简单的job,而是CMSSW紧密结合的

5. Crab functions(2) Submit jobs Monitor job Retrieve output
Prepare submission scripts(Jdl files, shell files)
Wrap submission tools into crab commands(eg.crab –submit)
Monitor job
Get status of running jobs by contacting with job scheduler
Retrieve necessary information from log files
Report it to the CMS dashboard
Retrieve output
Copy data files locally or SE
Retrieve the log files

6. Crab architecture UI parser create retrieve submit CRAB core Locate
pbs
edg
pbs
create
retrieve
cmssw
submit
Scheduler
API
Application
API
CRAB core
pbs
glite
…..
Locate
monitor
osg
Address
Parser
Job Monitoring
API
应用程序, 调度, 作业监控, 数据集定位, 命令解释
API中间层,扩展性,Application Programming Interface
pbs
DBS
DLS
mysql
sqlite
TFC

7. CRAB Core Main class – crab
Set up and initialize global instances and parameters
Start up action instances
Creator, submitter, status, killer, getoutput, checker, resubmitter, cleaner, postMortem, Jdlwritter, publisher, copyLocal, proxyrenewer
Common parameters- shared by all the instances
Workspace – work_space
Jobdir, Resdir, Sharedir, Logdir
Topdir:cwd + common.prog_name + '_' +’0’ + '_' + current_time (cwd/crab_0_080826_161045)
Configuration parameters – cfg_params
Parameters by parser from user options. Eg. cfg_params.get("CRAB.scheduler")
Job database Interface -- JobDB
An instance connecting with JobDB

8. Parser Command option Cfg file option Shared Variable in code
Eg. Crab –create –submit <number>
Cfg file option
<section>.<option>
<section> : [CRAB],[CMSSW], [USER],[EDG], [PBS]
[PBS]:
queue = cmsq
cmssw_path=/opt/exp_soft/cms
input_mode=local, dcap
input_path=/home/cms/zhangxm/test_crab/CMSSW_2_0_7/test/test_pbs/
output_path=/cms/cmsd01/zhangxm/test_pbs
Shared Variable in code
Cfg-param

9. Application API and cmssw(1)
Set up job_type instance(job_type:cmssw)
Functions of cmssw class :
Split job and register in job DB
[CMSSW].total_number_of_events, [CMSSW]. events_per_job, [CMSSW]. number_of_jobs
Set up cmssw.sh
Running scripts in work nodes

10. Application API and cmssw(2)
Cmssw.sh
Set up cmssw running environment
$VO_CMS_SW_DIR
User specified
Fill in the name of data files in pset.cfg
Dataset located Locally or in SE
cmsRun pset.cfg
Copy outfiles to the specified outpath
Copy locally or in SE
Clean temp files

11. Scheduler API BOSS(Batch Object Submission System) schedulerPBS
Developed by INFN, simplified to BOSSLite since CRAB_2_3_0
Provide submission and real-time job monitoring to various schedulers
Good extensibility
schedulerPBS
An extension to support pbs submission, status retrieving…..
Edg-job-submit…..
Jdl files
Sche-edg
submit
BOSSAPI
glite-job-submit…..
Jdl files
Sche-glite
Qsub,Qstat…..
Shell files
Sche-pbs

12. Job monitoring and Job DB
Store Job information in DB for sharing, monitoring and checking
To assure good extensibility and readability, use API
Database can be changed, eg. Mysql, sqlite
API also origin from BOSSAPI
job: ‘name’ , ‘executable‘, ’arguments‘,’logFile’ : ‘inputFiles’ ,‘outputFiles’,
’SubmissionNumber’
LoadJob
saveJob
updateJob
Job DB
mysql
mysqlSession
Main code
DbObject
BOSSAPI
Job DB
sqlite
SqliteSession

13. Address parser Goal: Functions:
Translation: dataset name ->LFN ->PFN
Access blocks and files information of dataset
Functions:
Communicate with DBS
Blocks and files name and list in the dataset
/HCalIsoTracks30/CSA08_CSA08_S43_v1/GEN-SIM-RECO->/store/mc/CSA08/HCalIsoTracks30/GEN-SIM-RECO/CSA08_S43_v1/0046/C09A DD11-BB5C B55FF.root
Communicate with DLS and TFC
Where Blocks and files are located, DLS for site name, TFC for real directory in the site
PFN: srm://srm.ihep.ac.cn:8443/pnfs/ihep.ac.cn/data/cms/store/…./.root

14. Local batch submission usage
Set PBS parameters in crab.cfg
queue = cmsq
cmssw_path=/opt/exp_soft/cms
input_mode=local, dcap
input_path=/home/cms/zhangxm/test_crab/CMSSW_2_0_7/test/test_pbs/
output_path=/cms/cmsd01/zhangxm/test_pbs
Define dataset name in crab.cfg
Get files list from DBS
Local DBS instead?
Crab –create –submit
Crab –status
Crab –getoutput
Logs files already in shared directory, eg. ../log/cmssw_x.stdout, cmssw_x.stderr

15. Summary and Further steps
The basic functions of Local batch submission have been realized
Create, submit, status has been complete
Based on CRAB_2_3_0
Feature:
unified job submission in grid and farm
able to do farm batch submission base on dataset and blocks
Future plan:
Need more tests, More functions added in?
Any test users?
Add local DBS instance?
Currently use global DBS
Can not register local Datasets…..