03/27/2003CHEP20031 Remote Operation of a Monte Carlo Production Farm Using Globus Dirk Hufnagel, Teela Pulliam, Thomas Allmendinger, Klaus Honscheid (Ohio State University)

03/27/2003CHEP20032 High luminosity experiments need large MC sample (Belle,BaBar require hundreds of millions of MC events) Massive computing power needed (farms of Linux machines) Farms are typically geographically distributed CLEOtwo sites DELPHIfive sites BaBartwo dozen sites (US and Europe) Belleeight sites The Problem:

03/27/2003CHEP20033 Hardware alone is not sufficient: Hardware, system level software maintenance Experiment specific MC software setup MC production Job submission Job monitoring (rerun failed jobs) Data transfer Coordination

03/27/2003CHEP20034 Is there another way? Reduced manpower requirements More efficient coordination Our approach Select one of the steps in the MC production chain MC Production Centralize operations Remote submission and monitoring Evaluate GRID tools. Can they help with MC production? Globus toolkit

03/27/2003CHEP20035 OSU MC Production Farm 27 dual Athlon nodes 1U 1 dual Athlon server 4U 840GB disk in RAID OpenPBS batch system File/batch queue server k MC events/day

03/27/2003CHEP20036 Globus Toolkit Globus Secure access Certificates for user and server Remote command execution system We observed significant overhead few seconds for single command Integrated tools e.g. GRIDftp Installation at Ohio State Globus on dedicated server Separate batch queue system for testing No Resource Broker Farm configuration details hidden Loss of dynamic configurability but much simpler

03/27/2003CHEP20037 MC production I : Job submission Typical input information : (MC software release), run range, #events … To do : build MC jobs and submit them Choose on option: One Globus command starts whole run range production many (thousands) of local jobs still need local script One Globus command starts a single MC production job Too slow Submit all production runs at once Only submit enough runs to fill queue Re-submitted jobs proceed faster

03/27/2003CHEP20038 MC production II : Job monitoring Job Status (“qstat”) Use local script to monitor log files Resubmit crashed jobs locally Monitor through Globus (remotely) Speed? Data Quality Monitoring check physics histograms not always done during production

03/27/2003CHEP20039 MC production III : Data transfer Easy if MC output is in file format GridFTP … Can be complicated otherwise Example would be writing MC into a database Limited disk space -> delete generated MC Log files

03/27/2003CHEP Conclusion MC production for a high luminosity experiment requires significant hardware and manpower resources. GRID tools can help to centralize this effort. Simple test show that remote operation of MC farms is possible Relatively easy to setup Globus framework (secure access, remote command execution) Local scripts for job submission, monitoring Still, significant software infrastructure (“local scripts” required. Other parts of the MC production chain need to be addressed before this becomes a realistic option. Remote MC software installation and version management