Framework of Job Managing for MDC Reconstruction and Data Production Li Teng Zhang Yao Huang Xingtao SDU

Outline Motivation: why developing Arbiter Concept: using workflow to manage jobs Profile: introduction to the general situation of Arbiter Principle: introduction to the core of Arbiter Using: introduction to the interface of Arbiter Conclusion: next to do

Motivation Why do we develop the system What function to provide

Motion : to solve trouble The trouble during managing MDC reconstruction and data production jobs: Need to generate job option files “by hands” It’s difficult to monitor status of every job Thus, it’s difficult to find which job is failed and to deal with them Also some other difficulties like how to handle large amount of input and output files, when to check the disk space… In a word, there’s too much repetitive work to do by hand. So we developed a system to do these work. Also, the system may be popularized to manage other production or analysis jobs in the future.

Desired function of Arbiter So we build a framework, Arbiter to serve as a production manager which plays a major role in executing organised data production or analysis activities: Designing the processing sequences Creating hundreds of sub-jobs Submitting and monitoring jobs Verifying output files

Concept and General Situation

Concept: workflow Workflow The workflow is an abstract description of the sequence of operations, like simulation, reconstruction, calibration… to be performed Step Each workflow can be decomposed into a series of steps connected to each other via input-output files. Each step consists a unbreakable operation. To brand these operations as reusable components, for each step, we define a type for it, like boss type, root type… Boss step ROOT step

Profile of Arbiter Arbiter is implemented in python Some python external packages are used: MySQLdb pylons and its dependencies paramiko Database used is mysql Write and access permission of database “arbiterDB” should be passed to user “arbuser” Use git as the version controll tool

Working Principle Structure of Arbiter Workflow in Arbiter How do Arbiter generate and submit workflow How do Arbiter monitor and manage workflow

Introduction to Arbiter Arbiter is consisted of a core for managing workflow, 3 kinds of interfaces and a database

Workflow in Arbiter workflow Workflow defined in Arbiter is described by the class workflow() which contains workflow base information and methods, and a stepPool(), a list for steps. step Each step is described by the class step(). It contains the step base information, a parameterCollection() as a container for user defined parameters, a splitter for splitting jobs and a generator for generating job option files parameter The parameter() class is for storing certain information of a step, like input, output, option name, working directory… generator Generator is a tool for generating job option files. When the system are about to generating job option files, the step() or subStep() defined its generator based on the type and parameters of the step and then the generator will generate scripts according to the job option template and other parameters.

Workflow in Arbiter Splitter The splitter is a useful tool for splitting steps into sub-steps(each sub-step will stand for a job that can run on the farm). We’ve defined two types of splitter so far, inputDataSplitter for splitting step according to input files(number and size) and numberSplitter for just duplicating step into sub- steps of a certain number.

Generating Workflow User can define worklfow through Arbiter API, and then Arbiter will generate scripts for all steps. If they're all OK, Arbiter will record the workflow. Then user can submit the workflow through command. if not testing workflowID

Monitoring Workflow When user want's to know the status of one or more jobs in the workflow, user can execute "arb listworkflow" command to get the list of workflow in Arbiter. Then user can get the ID of the workflow he's interested in and does further management with it.

Arbiter Interface API of Arbiter Command line interface web portal

API of Arbiter Application Programming Interface(API) API of Arbiter is consisted of two python classed, Job() and System(), one for workflow generating and one for managing. Through API, most of the function of Arbiter is exposed. User can generate workflow using API. This is a typical API script for generating a workflow After submitting the script, user will get the ID of the workflow from arbiter.Interface.API.Job import * job = Job(‘jobname’) #job.debug = True job.addStep(‘boss/step1’) job.setStepParameter(‘step1’, ‘input’, ‘/root/test/datalist.txt’) job.setStepParameter(‘step1’, ‘output’, ‘/root/test/output/') job.setStepParameter(‘step1’, ‘optionTemplet’, ‘/root/test/rec.txt’) job.setStepParameter(‘step1’, ‘optionFileDirectory’, ‘/root/test/) job.setStepSplitter(‘step1’, {‘name’: ‘dataSplitter’, ‘maxFileNumber’:1}) job.generate()

Command Line Interface In order to provide an easy way to manage the workflow generated by the system, we provide several commands. All commands follows sub-command style like: "arb checkstatus..." Most commands need user to provide the ID of the workflow. User can get list of all commands by executing "arb -h" User can get commands help message by executing "arb command -h"

Web Portal of Arbiter Being proposed to find a most easy and visual way to generate and manage workflow, we developed the web portal for Arbiter. We used pylons to build the web page. The server host must have access to the DB host and the host for running jobs. So we placed it on offline.ihep.ac.cn:5000

Web Portal of Arbiter We have added the pages for monitoring workflow and for defining new worklfows. User can see all sub-job status conveniently. We have also provided function for re-submitting jobs.

Conlusion Next to do

Conclusion and next to do Conclusion We have deveoped the system and tested most of the function implemented. The skeleton of the system has been established and most of the basic function is now working correctly. But wether the functions are working smoothly and stably is unknown add we need more test. We've still got many functions to modify or to add. Next to do Do more test Add other step types(new generator, new method for monitoring) Think more of the web portal...

Your suggestion is my power ！ 谢谢！