RunJob in CMS Greg Graham Discussion Slides

RunJob in CMS RunJob is an Application Configuration and Job Creation Tool –RunJob uses metadata to abstract application steps and bolt them together into a workflow (Configurators) –RunJob uses metadata to abstract service invocations and iterators to parallelize job creation (other Configurators) –RunJob has an abstract model of jobs. These can be translated into concrete jobs for a variety of environments (local or grid) later depending on what modules are plugged in

RunJob in CMS RunJob does job creation in scripts (parallelized or not) –RunJob scripts support external service invocations, looping over data things, inline job submission, metadata constraint modeling among applications and services, persistence, and conventional provenance. –Scripts can be factored into simpler scripts and contexts. Contexts contain site level details, administrative details, logical details about constraints on the simpler scripts. Saving the context scripts can enable “meta-provenance” in the sense that recording constraints provides clues as to why certain parameters are set the way they are instead of just their value.

RunJob in CMS Specific Uses –Official Monte Carlo Generation Over 30 sites, over 75 million GEANT events, controlled centrally by central RefDB (pull model) –Data Challenge 2004 Used at CERN to govern online reconstruction of simulated data at 25 Hz. Used at FNAL to create physics plots from reconstructed data transferred there.

RunJob in CMS Extensions to RunJob –XMLP: An XML persistency mechanism –shREEK: Runtime extensions to RunJob, monitoring, and late job building. –logger: An event logger –scriptObjects: Abstract descriptions of jobs that serve as input to code generators producing actual jobs for specific environments –CAST: A GUI for construction Configurators and workflows.

Databases & Catalogs Local/Grid Resources Linker: A container for Configurators that enables constraint propagation, workflow modeling, provenance, scripting, and context resolution of scripts Configurator: An Object Oriented, Metadata based description of services or applications with APIs for extending behaviors to many environments Applications Execution Managers RunJob ScriptsRunJob Contexts UserAdmin ScriptObjects

Relationship to Physics Physics Parameters Specification –By physics group: in a context file written by a special user designated by the physics group Members working within a physics group will inherit these parameters by using a common context Constrains multiple applications according to the same set of constants –By knowledgeable user: in a personal MCRunjob script file or in a personal context file. –By novice user: on the command line of a pre-existing MCRunjob script written by a knowledgeable user. In this case, the novice is restricted to specifying parameters foreseen by the author of the MCRunjob script in question.

Linker: A container for Configurators that enables constraint propagation, workflow modeling, provenance, scripting, and context resolution of scripts RunJob ScriptsRunJob Contexts User Admin -User’s Job Description eg- attach Pythia attach BatchManager -Physics Parameters of User Interest eg- TopMass=275 GeV/c2 -Specification of Which Context To Use -Physics Parameters set by Physics Group eg- HiggsMass=115 GeV/c2 TopMass=273 GeV/c2 Decay Parameters from a Control DB -Physical Description of Site Resources eg- BatchManager=FBS -Configuration of Site Resources eg- FBSQueueName=CMS; Submit Jobs Inline -Combine all specified contexts -Fully Specified Workflow Description eg- attach ControlDB attach Pythia HiggsMass=115 GeV/c2 TopMass=275 GeV/c2 Decay Parameters from a Control DB attach FBS FBS QueueName=CMS; Submit Jobs Inline -Read User Script-Read Context(s) specified by UserResult Text Results from Configurators

Control DB Linker: A container for Configurators that enables constraint propagation, workflow modeling, provenance, scripting, and context resolution of scripts Configurators: An Object Oriented, Metadata based description of services or applications with APIs for extending behaviors to many environments PythiaFBS -Fully Specified Workflow Description eg- attach ControlDB attach Pythia HiggsMass=115 GeV/c2 TopMass=275 GeV/c2 Decay Parameters from a Control DB attach FBS FBS QueueName=CMS; Submit Job MySQL attach ControlDB Get decay parameters return decay parameters: {x} attach Pythia HiggsMass=115 TopMass=275 DecayParameters {x} return Pythia scriptObject attach FBS FBSQueueName=CMS Submit Job Now FBS Submit Results FBS Submit Results Queue CMS

Relationship to Physics Provenance –By specification of fully reduced workflow specification (all contexts resolved, job level) –By specification of contexts actually used in modifying a workflow (why does a constant have value x?) –By specification of