1. Job Application Monitoring (JAM)
Giacinto Donvito – Marcello Maggi – Giorgio Maggi
INFN Bari

2. Giacinto Donvito IV INFN-Workshop
Goals of the package
To provide a “BOSS-LIKE” tool for any application running over the Grid being
User driven
Scalable
Easy to be adapted to different applications
To use user jobs to diagnosticate problems related to:
the Grid
the environment setup
the application
To provide a tool to be used in a “analysis environment” on the Grid (keeping pace with code changes)
The idea comes from the “gridification” of tools used for MonteCarlo production of ALEPH experiment at LEP
Giacinto Donvito IV INFN-Workshop

3. Giacinto Donvito IV INFN-Workshop
Goals of the package (2)
To check the state of a job and of their relevant parameters (starting and ending date, destination site , Worker Node, ecc)
To Monitor each step of the job (copy of input and output files, check of the environment, ecc)
To Monitor whatever parameter internal to the executable
To provide the information needed for a “Post-Mortem” analysis of jobs
To provide an infrastructure to include whatever parameter that the user would monitor
To retrieve any output file in real-time on the UI (on a SE)
To track the information of jobs that could be “Aborted” for the grid.
The user can create a Database to collect the information in whatever machine
Giacinto Donvito IV INFN-Workshop

4. Giacinto Donvito IV INFN-Workshop
Characteristics
The submission actually is performed using a wrapper (any other possibility is open). The arguments of the wrapper are:
jdl file
the “ClassID” of the job (needed to indentify all the job from the same application in the DB, or distinguish different versions of the same application) …
The job sends the information to the DB using a “proxy” installed on the CE, via SOAP protocol, and fowards them to the DB (to avoid outbound connectivity problems of WN and to garantee the scalability of the system)
The control of the status is on-demand and it is needed only the GridJobID
the informations that job sends to DB are used to give a more detailed view of the status
Only the owner of the job can retrieve it’s output
Giacinto Donvito IV INFN-Workshop

5. GUI Prototype Standard Parameters User Parameters JOBID classid Status
Events
Higgs found
Higg_search_v2
Running
550 3
….. …. ……
Giacinto Donvito IV INFN-Workshop

6. Work flow (1) SubmissionWN
Local User
Path
Job State
Class
Grid JobID
Job Class
Giacinto Donvito IV INFN-Workshop

7. Work flow (2) Status Check
Grid JobID
Job Status
Custom Information
Giacinto Donvito IV INFN-Workshop

8. Work flow (3) Retrieve of Output Files (STDOUT… )
Cache File
Location Information
File Cache Request
Cache Ack
File Trasfer
Grid JobID
Job Status
Location Information
Giacinto Donvito IV INFN-Workshop

9. Giacinto Donvito IV INFN-Workshop
Experience of Real Use
To test the pakage we had used it to monitor a real analisys of a CMS Bari user (H->ZZ->2e 2mu):
~ jobs
> entries in DB
MONITORIZED PARAMETERS
Input/Output Files
Start/End executables
Root File
Global Statistics of ClassID: tt
Detailed information about jobs
Giacinto Donvito IV INFN-Workshop

10. Giacinto Donvito IV INFN-Workshop
Status of the work
A C++ API was developed to write internal information of the executable and to read information from the DB
A C++ CLI to write information into DB (for example in a bash script) was developed
A set of perl script was developed to retrieve all the information and the output file
A set of C++ API to read the information from the DB was developed
Giacinto Donvito IV INFN-Workshop

11. Giacinto Donvito IV INFN-Workshop
TO DO List
To test C/C++ library to improve this system
To improve the C++ read API (needed some use cases)
To develop a GUI to provide a more friendly use of the information stored in the database
To replace the wrapper used to submit the jobs with a more complete program (using various API)
To develop an “intelligent” system of analysis to flag a job as “bad” or “good” in an automatic way.
Giacinto Donvito IV INFN-Workshop

12. To have a first Alpha Relese in few days User feedbacks are needed
Future
To have a first Alpha Relese in few days
(~ 2 weeks)
User feedbacks are needed
To analyse new use case
(CDF, Bioinformatics…)
Giacinto Donvito IV INFN-Workshop