1. LHCb Software Week 12-14 November 2003 Gennady Kuznetsov Production Manager Tools (New Architecture)

2. Data Challenges 2001 Gennady Kuznetsov History: Controlling Monte-Carlo simulation. In 2001, control system was based on a single Java Servlet which in fact was a single file with Java code. The sequence of execution was fixed and all site dependencies had to be programmed inside. Code had to be redistributed for each location with the Servlet runner. Increased complexity of the tasks and demands for flexibility (requirement to run more then one type of job) required architectural changes. As a result, the “Data Management” group produced a document with a new flexible architecture.

3. The Grand Design Gennady Kuznetsov Job Configuration, Data Production, Bookkeeping - Design Document, M. Frank (Editor), 04/12/2001

4. Data Challenges 2003 Gennady Kuznetsov Agent Agents interpret XML file with Workflow description Advantages: This very flexible architecture gives us freedom to adjust any parameters in the workflow and also allows us to manipulate Steps known to the “Agent”. Disadvantages: Agent had to exist on the site or had to be submitted. For any change in execution, we had to update the agents everywhere.

5. Components - New System Module Step Workflow Job Production Levels of usage: 1.Module – Programmer 2.Step – Production manager 3.Workflow – User/Production manager We add Module as additional component of the architecture Each step generates job as a Python program. This module structure allow Production Manager to construct any algorithm as a combination of modules. Gennady Kuznetsov

6. Architecture - New System Presenter Name: Variable VariableCollection ModuleDefinitionModuleInstance n 1 StepDefinition n 1 n StepInstance WorkflowDefinition 1 1 nn 1 Module Editor Step Editor Workflow Editor edit

7. Module/Step example Execution Sequence Gennady Kuznetsov Distribution Software check and installation Example of the Workflow 370000-Sv260r2-Bv17r4 Brunel Reconstruction Gauss Simulation Mon. Service Monitoring Data quality “rawh” validation File Service File transfer Data quality “oodst” validation File Service File transfer Bookkeeping. Service Bookkeeping Gauss Simulation Mon. Service Monitoring Data quality “rawh” validation File Service File transfer Bookkeeping. Service Bookkeeping Gauss Simulation Mon. Service Monitoring Data quality “rawh” validation File Service File transfer Bookkeeping. Service Bookkeeping Gauss Simulation Mon. Service Monitoring Data quality “rawh” validation File Service File transfer Bookkeeping. Service Bookkeeping Another three lines (not enough space) Module

8. Module/Step example Data propagation Gennady Kuznetsov Distribution Software check and installation Example of the Workflow 370000-Sv260r2-Bv17r4 Gauss Simulation Mon. Service Monitoring Data quality “rawh” validation File Service File transfer Bookkeeping. Service Bookkeeping Module

9. Module Editor Python code of single module. Can be many classes. Module variables. Description Module Name Stored as XML file Gennady Kuznetsov

10. Step Editor Step Name Description Definitions of Modules Instances of Modules Variables of currently selected instance Selected instance Stored as XML file, where all modules are embedded Gennady Kuznetsov Step variables.

11. Workflow Editor Gennady Kuznetsov Workflow Name Description Step Definitions Step Instances Variables of currently selected Step Instance Selected Step Instance Workflow Variables. Stored as XML file

12. Job Splitting Gennady Kuznetsov Step Workflow Definition Job Production Python List The input value for the job splitting is a Python list object. Every single (top level) element of this list applies to the Workflow Definition and propagates through the code and generates single element of the production (one or several jobs).

13. Current Status Gennady Kuznetsov Done: All three editors (Module, Step, Workflow) ready. XML schema seems stable. To do for the Data Challenge 2004: Define variable types. Extend Python code generator to cover Workflow. Create appropriate module library. Implement “list input” for the Workflow Editor (relates to the job splitting). Future development: Add graphical editing panel to Step and Workflow editors. Finish embedded Python editor with highlighting feature.