McFarm Improvements and Re-processing Integration D. Meyer for The UTA Team DØ SAR Workshop Oklahoma University 9/26 - 9/27/2003

Sept. 26, 2003McFarm Improvements; D. Meyer, UTA DØ SAR Workshop, OU 2 Reasons for Using McFarm McFarm is a DØ MC Control Software developed at UTA and used in six farms Simplifies Monte Carlo Production Manages the Cluster with Minimum Labor Manages the Cluster Efficiently Minimizes Impact of Changes to SAM, mc_runjob, other DØ software User-Oriented

Sept. 26, 2003McFarm Improvements; D. Meyer, UTA DØ SAR Workshop, OU 3 McFarm Software Integration DØ Binaries - minitars or full release SAM - declaration, storage, retrieval mc_runjob - job and metadata construction NFS - access to binaries, minbias database NIS - account management ssh - intra-cluster monitoring and control Batch Queues - PBS and Condor

Sept. 26, 2003McFarm Improvements; D. Meyer, UTA DØ SAR Workshop, OU 4 Improvements: Procedural Changes Request monitoring now provided by McFarm monitor to close requests “check_sam” now obsolete, replaced by archive daemon & store-verification Mechanism to handle too-large reco tasks: do just the pythia/d0g/sim (PDS jobs) and let requestor do reco on CAB

Sept. 26, 2003McFarm Improvements; D. Meyer, UTA DØ SAR Workshop, OU 5 Bug Fixes event-count now correct when not all events done available-space correct on NFS-mounted disks (df command) No longer attempting to patch metadata for bad key-words. Others

Sept. 26, 2003McFarm Improvements; D. Meyer, UTA DØ SAR Workshop, OU 6 Enhancements Two-day grace period for final tmb merge now configurable: FARM_MERGE_GRACE_HOURS Also FARM_MERGE_MAX_EVENTS and FARM_MERGE_MAX_FILES Monitor reassurance can be turned off: FARM_MONITOR_REASSURE=‘NO’

Sept. 26, 2003McFarm Improvements; D. Meyer, UTA DØ SAR Workshop, OU 7 Enhancements - 2 FARM_SAM_VERIFY_STORE_HOURS makes storing and purging separate events in McFarm. “Archive” daemon. SAM store retry improved - will undeclare, cancel-store as necessary bin/onetime/re-store full-job-dir-name SAM gather will get to merger files periodically even if busy with regular stores

Sept. 26, 2003McFarm Improvements; D. Meyer, UTA DØ SAR Workshop, OU 8 Enhancements - 3 Request life-cycle monitoring handled by McFarm monitor to improve turn-around. Number of events now in gather.log execute daemon will detect reco stall due to over-swapping and will kill job. Execute daemon retains job hist even when stopped/restarted (job.hist file)

Sept. 26, 2003McFarm Improvements; D. Meyer, UTA DØ SAR Workshop, OU 9 Enhancements - 4 launch_request accepts PDS (and S) jobs to handle unwieldy reco requests - it stores sim files. purge_job accepts “--d0phase=mcpNN” argument to purge archives by D0 phase, including merger archives.

Sept. 26, 2003McFarm Improvements; D. Meyer, UTA DØ SAR Workshop, OU 10 Re-Processing Real Data Basic approach is to do d0reco binary only, using reprocessing Framework rcp, running on raw or reco file as input. Joel Snow traced the rcp usage in UMICH job Mark Sosebee has done sample by hand and analyzed histograms - so far so good Dave Evans has included re-processing support in version of mc_runjob

Sept. 26, 2003McFarm Improvements; D. Meyer, UTA DØ SAR Workshop, OU 11 Re-Processing Real Data - 2 I have used mc_runjob v06 to manually re- process both MC reco files and raw files. Testing is continuing - presently some problems with metadata declaration. The bad news: mc_runjob v06 contains substantial changes to job structure and execution that will require days of work to integrate into McFarm and test all code

Sept. 26, 2003McFarm Improvements; D. Meyer, UTA DØ SAR Workshop, OU 12 Re-Processing Real Data - 3 Our approach is to have Request_NNNN.py file include a sam dataset definition of files to be re- processed, feed into McFarm just like a Monte Carlo request Some of McFarm is ready (SAM acquire), some is not (launch_request RT, v06 adaptation, switch from events to files) Dave Evans is leaving

Sept. 26, 2003McFarm Improvements; D. Meyer, UTA DØ SAR Workshop, OU 13 Re-Processing Real Data - 4 Key contents of Request_REPROC02.py: 'Reconstructed':{'datasetdefname':’ reco_ _raw_2_files’, 'frameworkrcpname':'runD0recoSAM_data_reprocess _p13dst.rcp',}, launch_request REPROC02 /home/mctest 0 RT job UTATEST-RT-ReqREPROC It runs under mc_runjob v05 / McFarm v10.04, but no proper metadata yet.

Sept. 26, 2003McFarm Improvements; D. Meyer, UTA DØ SAR Workshop, OU 14 Re-Processing Real Data To Be Done mc_runjob v06 must be debugged and released McFarm must be adapted to v06 and debugged Metadata must be stabilized and accepted by SAM Re-processing authority should use MC-like Request_NNNN.py to invoke re-reco

Sept. 26, 2003McFarm Improvements; D. Meyer, UTA DØ SAR Workshop, OU 15 Conclusions McFarm has morphed significantly since its creation to accommodate –Enhanced error handling –Enhanced monitoring –Other improvements Re-processing capability in the works, despite some worries on schedule and support IAC’s use and comments prompted McFarm improvements (Thank you everyone!!)  Comments always appreciated