gridmake for GlueX software Richard Jones University of Connecticut GlueX offline computing working group, June 1, 2011

Richard Jones, Gluex offline computing working group, June 1, motivation run GlueX jobs in a grid environment  verify a minimal working environment (perl, wget, …)  specify minimal setup needed to accomplish a task  ability to discover steps already done and not redo them  target-based syntax (make this output file) preferred over procedure-based scripting (run this executable)  discover required components already installed on a site, and do a fresh install of any missing components  coordination across sites, integration with SRM automate a GlueX workflow for anyone  wget  wget ./gridmake dana_events_0.hddm

Richard Jones, Gluex offline computing working group, June 1, what gridmake does 1.starts the perl interpreter – no perl, no go 2.looks for a bunch of perl modules it needs (eg. XML::Xerces) if already installed, loads them into the runtime if not installed, downloads, builds, installs, and loads them 3.opens the gridmake.xml file (overridden by –f option) 4.validates it against the xml schema for gridmake 5.finds the target “dana_events_0.hddm” as something’s output 6.scans the dependency tree for this target (make heuristics) 7.determines if the target is stale (can be forced in config file) 8.remakes anything needed to freshen the target 9.cleans up intermediate files (config “keep” option saves them) example:./gridmake dana_events_0.hddm

Richard Jones, Gluex offline computing working group, June 1, what gridmake needs a basic unix environment (bash, gnu compilers, …) perl (tested with 5.8) wget (can be a symlink to curl) live connection to the internet, every time live connection to the internet, every time disk quota for any installs you might need, or for data files – current policy is to make it fit within a 10GB box for all jobs standard environment variables pointing to any preinstalled packages that you prefer not to see rebuilt for this job your xml config file (default:./gridmake.xml) make target for this job example:./gridmake dana_events_0.hddm

Richard Jones, Gluex offline computing working group, June 1, where gridmake does it./packages   setup  setup.env  installation area for arch_1  … any other architectures  … various bits leftover from the installer (eg. installer_log) example:./gridmake dana_events_0.hddm a complicated bash script able to discover, verify, install, and setup the environment for this package a simple list of environmental variables for the package where it was last found this part only present if a local install of this package was required

Richard Jones, Gluex offline computing working group, June 1, what packages gridmake supports xerces-c evio clhep cernlib geant4 jana sim-recon (includes hdds and calib) threePi hello example:./gridmake dana_events_0.hddm a simple package for testing gridmake, a c program that writes a file “hello_world”, example:./gridmake hello_world

Richard Jones, Gluex offline computing working group, June 1, how to add a package to gridmake edit your gridmake.xml and add the package add a processor to gridmake.xml that uses it setup not found on server try it out:./gridmake foo_bar_output – setup not found on server send package details, setup script to … any other require tags foo_bar_output #!/bin/bash $FOO_BAR_HOME/bin/foo_bar >foo_bar_output

Richard Jones, Gluex offline computing working group, June 1, status of gridmake this is an alpha release successfully tested on UConn, IU, ifarm machines currently being tested on other OSG sites groups can set up their own package distribution servers, independent of the UConn server for the future – is this scalable to 10,000 jobs? for the future – is this scalable to 10,000 jobs? ./gridmake three_pion_0.hddm currently generates 12 HTTP GETs to my web server, once installation is done.  120,000 HTTP GET connections in 24 hours for ~5kB files is not too bad, but most of these could be avoided. ongoing development expected, suggestions welcome