SEAL: Core Libraries and Services Project CERN/IT After-C5 Meeting 6 June 2003 P. Mato / CERN

6 June 2003SEAL - Core Libraries and Services ProjectP. Mato/CERN2 Contents  History: The blueprint RTAG  Project Aims  Current Work Areas: –Foundation Libraries –Math Libraries –Plugin Management –LCG Dictionary –Scripting services  Software Process  Project Status and Summary

6 June 2003SEAL - Core Libraries and Services ProjectP. Mato/CERN3 The Blueprint RTAG  The mandate of the RTAG was to define the architectural ‘blueprint’ for LCG applications: –Define the main architectural domains (‘collaborating frameworks’) of LHC experiments and identify their principal components. –Define the architectural relationships between these ‘frameworks’ and components, including Grid aspects, identify the main requirements for their inter-communication, and suggest possible first implementations. –Identify the high-level milestones for each domain and provide a first estimate of the effort needed. –Derive a set of requirements for the LCG  Any piece of software developed by any LCG common project must conform to a coherent overall architectural vision  The end goal is the integration of LCG and non-LCG software to build coherent applications  The blueprint will be established in terms of a set of requirements, suggested approaches and guidelines, and recommendations

6 June 2003SEAL - Core Libraries and Services ProjectP. Mato/CERN4 Architecture requirements  Long lifetime: technology evolution  Languages: C++ today; allow multi-language and evolution  Distributed applications  TGV and airplane work  Modularity of components  Component communication via public interfaces  Interchangeability of implementations  Integration into coherent framework  Design for end-user’s convenience more than the developer’s  Re-use existing implementations  Software quality at least as good as any LHC experiment  Meet performance, quality requirements of trigger/DAQ software  Platforms  Trigger/DAQ environment

6 June 2003SEAL - Core Libraries and Services ProjectP. Mato/CERN5 Architectural Elements  Interface model –Abstract interfaces, versioning, guidelines,…  Component model –Communication via public interfaces (no hidden channels) –Plug-ins (run-time loading) –Life-time management (reference counting) –Configuration  Design guidelines –Dependencies –Exception handling –Interface to external components Basic Framework Foundation Libraries Simulation Framework Reconstruction Framework Visualization Framework Applications... Optional Libraries Other Frameworks Software Structure

6 June 2003SEAL - Core Libraries and Services ProjectP. Mato/CERN6 Architectural Elements (2)  Object Dictionary –The ability to query a class about its internal structure (Introspection) –Essential for data browsing, rapid prototyping, persistency, etc.  Object Whiteboard –Uniform access to application-defined objects (equivalent to the Gaudi transient stores)  Component Bus –Integration of components providing a wide variety of functionality GUI Python math shell PyGaudi Database EDG API GUI XML Gaudi Frame work PyROOT Root Classes PVSS JPE Java Classes

6 June 2003SEAL - Core Libraries and Services ProjectP. Mato/CERN7 The Blueprint RTAG Recommendations  RTAG establishes a user/provider relationship between LCG software an ROOT –LGC software will not be based on ROOT, it will use ROOT where adequate  Start common project on core tools and services  SEAL Project  Start common project on physics interfaces  Start RTAG on analysis, including distributed aspects  Tool/technology recommendations –CLHEP, CINT, Python, Qt, AIDA, …  Develop a clear process for adopting third party software

6 June 2003SEAL - Core Libraries and Services ProjectP. Mato/CERN8 SEAL Overview  SEAL aims to –Provide the software infrastructure, basic frameworks, libraries and tools that are common among the LHC experiments –Select, integrate, develop and support foundation and utility class libraries –Develop a coherent set of basic framework services to facilitate the integration of LCG and non - LCG software  Scope –Foundation Class Libraries »Basic types (STL, Boost, CLHEP, …), utility libraries, system isolation libraries, domain specific foundation libraries –Basic Framework Services »Component model, reflection, plugin management, incident (event) management, distributed computing, grid, scripting

6 June 2003SEAL - Core Libraries and Services ProjectP. Mato/CERN9 Domain Coverage

6 June 2003SEAL - Core Libraries and Services ProjectP. Mato/CERN10 Assumptions, constraints, risks  Do not re-invent the wheel –Most of the core software to be delivered by SEAL exists - more or less - in experiments’ core software »We will re-use as much as possible existing software »Most of the work will be in re- packaging existing pieces of software  If wheel squeaks… –Develop / adapt / generalize in order to achieve the necessary level of coherency and conformance to the architectural vision already established  Adopt a Seal –In order to use SEAL, projects will need to replace their own software elements with SEAL functionally equivalent ones. This will certainly imply some period of instability for the experiment applications

6 June 2003SEAL - Core Libraries and Services ProjectP. Mato/CERN11 Project Work Packages Foundation and Utility Libraries Math Libraries Component Model and Plug-in Manager LCG Object Dictionary Basic Framework Services Scripting Services Grid Services Education and Documentation New work package added to the project by incorporating the MathLib project (F. James et al.) This work package will probably be split between “Foundation and Utility Libraries” and “Basic Framework services” later Not yet started working in this area

6 June 2003SEAL - Core Libraries and Services ProjectP. Mato/CERN12 SEAL Versions Road Map ReleaseDateStatusDescription (goals) V /02/03internal  Establish dependency between POOL and SEAL  Dictionary generation from header files V /03/03public  Essential functionality sufficient for the other existing LCG projects (POOL)  Foundation library, system abstraction, etc.  Plugin management V /05/03internal  Improve functionality required by POOL  Basic framework base classes V /06/03public  Essential functionality sufficient to be adopted by experiments  Collection of basic framework services  Scripting support Released 04/04/03 Released 14&26/02/03 Released 23/05/03

6 June 2003SEAL - Core Libraries and Services ProjectP. Mato/CERN13 1. Foundation and Utility Libraries  Inventory of existing utility classes  Support for Boost library –Open source utility library (parts of which are destined to be included in the following C++ standard)  Participation to CLHEP project  Adopted CMS classlib –A set of foundation and utility classes mainly used in Iguana (CMS)  Development of SealUtil and SealKernel packages –The idea is to develop SEAL utility and system isolation library complementary to Boost, CLHEP and STL  Guidelines for selecting external libraries –Set of criteria to define

6 June 2003SEAL - Core Libraries and Services ProjectP. Mato/CERN14 2. Math Libraries  GSL (Gnu Scientific Library) evaluation –How it compares with NagC –Automatic testing of GSL (with cppUnit) as part of every SEAL release  Re-implementation of MINUIT in C++ –Prototype available in current SEAL release –Migrad and Minos available –The numerical results of the two prototypes compared to the Fortran version. Compatible within the errors.  Other libraries –MathLibs/GSLAlgebra. Prototype wrapper around GSL. Not complete but restricted to Minuit needs.  Provide to experiments with math and statistics libraries to be used in analysis, reconstruction, simulation. –GSL support, …

6 June 2003SEAL - Core Libraries and Services ProjectP. Mato/CERN15 3. Component Model and Plug-in Manager  Plug-in Management –Service in charge of managing, querying, [un]loading plug-ins  Implementation in two levels –Lower level (no framework constraints) »Ability to create “module” libraries that contain “plug-ins” »Ability to instantiate concrete implementations knowing the plug-in category and the concrete type »Caching information about what modules contain what plugins and their categories –Plugin instances management (assuming a given framework model) »Locate other plug-ins and manage their lifetime »Framework base classes to obey plug-in/framework model

6 June 2003SEAL - Core Libraries and Services ProjectP. Mato/CERN16 3. Component Model and Plug-in Manager(2)  Howto’s available –Host modules in your project –Create a new plugin factory –Create a new plugin module library Plug-In Cache Object Factory Object Factory Plug-in Database Plug-In Cache Module Object Factory Attached Unattached // Module Definition DEFINE_SEAL_MODULE(); DEFINE_SEAL_PLUGIN(SomeFactory, MyObject); // Usage by a client BaseObject* p; p = SomeFactory::get()->create(“MyObject”);

6 June 2003SEAL - Core Libraries and Services ProjectP. Mato/CERN17 4. Object Dictionary  Adding “introspection/reflection” capabilities to C++  Reflection packages –The Dictionary/Reflection package provides the capability to introspect and interact with any C++ object at run-time –The Dictionary/ReflectionBuilder package is the “write” interface of the reflection information. Typically generated code uses this interface to built the reflection information at run-time  Dictionary generation –Main goal: Full support of C++ without any class instrumentation –DictionaryGenerator package provides the command to do it  Usages so far –Object persistency (POOL) –Interactivity and scripting (Python)

6 June 2003SEAL - Core Libraries and Services ProjectP. Mato/CERN18  Dictionary generation from header files –Uses gccxml (0.4.0) to parse header-files (extension to gcc- compiler), generates intermediate XML file with dictionary information –XML file parsed by a python script which generates dictionary building C++ code (for “selected classes”) –Compiled to shared library and loaded at run-time to create dictionary in memory 4. Object Dictionary (2).xml.h lcgdict.h gccxml.xml filter + exten gendict par ser _dict.cpp.so make selection file (python script) #include files

6 June 2003SEAL - Core Libraries and Services ProjectP. Mato/CERN19 LCG Object Dictionary: Usage ROOT I/O LCG Dictionary CINT Dict Streamer.h in out ROOTCINT CINT generated code Dict generating code.adl.xml ADL/GOD Other Clients: (python, GUI, etc.) LCG to CINTDict gateway (2) (1) Population Conversion Reflection GCC-XML

6 June 2003SEAL - Core Libraries and Services ProjectP. Mato/CERN20 5. Basic Framework Services  Developing the basic framework –Component model definition » component identification, configuration, lifetime strategy, interface model, etc. –Started to review existing designs (Gaudi, Cobra, Iguana,…)  Basic Component Model –Set of “base classes” and interfaces that developers will use to build their components  Developing Basic Common Services –Started design and implementation for the message reporting service –Other services like component configuration, “event” management, object white board, etc. will continue soon after

6 June 2003SEAL - Core Libraries and Services ProjectP. Mato/CERN21 5. Basic Framework Services (2)  The current ideas –Give to the “users” (framework developers) a number pre-defined service base classes (from simple to more complex) –Developers should “model” their services using one of the proposed base classes –Standard functionality (instantiation, configuration, lifetime management, etc.) should be provided by these bases classes Service MyService Context PropMgr BServLetAServLet ClientNameSvc

6 June 2003SEAL - Core Libraries and Services ProjectP. Mato/CERN22 6. Scripting Services  Scripting needed for Interactivity, Configuration, Rapid prototyping, Integration, etc.  GOAL: Bring Python and C++ (CINT) at the same level of functionality –Interoperability between them. The end-user choice.  Define guidelines for developing Python bindings –Evaluate existing options: SWIG, Boost.Python, SIP, and raw Python C-API –Study how each technology handles a number of predefined cases »Examples: Method overloading, inheritance across boundaries, templated classes and methods, natural mapping of Python constructs, etc. –Study interoperability between binding technologies

6 June 2003SEAL - Core Libraries and Services ProjectP. Mato/CERN23 6. Scripting Services (2)  PyROOT: Python bindings for ROOT (former RootPython) –PyROOT is a Python extension module that allows the user to interact with any ROOT class »Done generically using the ROOT dictionary. No need to generate any Python wrapper code to include new ROOT classes  PyLCGDict: Python extension module –Started to work on the LCG Dictionary bindings –Basically repeating the work of PyROOT using the LCG dictionary instead

6 June 2003SEAL - Core Libraries and Services ProjectP. Mato/CERN24 8. Education/Documentation  Main activities and tasks –Produce documentation –Produce training material (tutorials) –Help incorporating SEAL components into LCG projects and experiment frameworks  Existing documentation –Very limited for the time being –Produced a number of HowTo pages for the released elements »Plugin Manager, Dictionary generation, PyROOT, etc.  Helping POOL to incorporate released components –We have a team member working for both POOL and SEAL  Will be preparing tutorials for June release –Time of possible adoption by experiments

6 June 2003SEAL - Core Libraries and Services ProjectP. Mato/CERN25 Software Process  SPI provided infrastructure –Savannah Web portal, CVS repository, development tools (SCRAM, Doxygen, CppUnit, etc.) and services (external tool repository, etc.) –Slowly getting the good dynamics and practices  Efforts to “standardize” tool usage and conventions –Avoiding divergences between LCG projects –Adaptation of tools to “LCG proper” working models  Release procedures –Developing SEAL proper procedures –Tagging, building, testing, releasing, documentation generation, announcing, etc. –Automation of the process is essential (rotating release manager role)  Platforms –Single release platform for the moment: Linux RedHat 7.3/gcc-3.2

6 June 2003SEAL - Core Libraries and Services ProjectP. Mato/CERN26 Status and Summary  The SEAL project started last November as a result of the “blueprint” RTAG recommendations  SEAL has had three main releases: 0.1.0, 0.2.0, since the beginning of the year at the scheduled times –The main emphasis has been to support POOL (Dictionary, Plugin Management, Foundation classes, etc.)  The functionality available is not yet enormous but its development has help us –To get experience on new tools and procedures –Build up the development team  No technical problem encountered so far  Next scheduled release –Basic functionality completed and ready to be adopted by experiments at end-June

6 June 2003SEAL - Core Libraries and Services ProjectP. Mato/CERN27