Genova 10-Dec-2001 Andreas Pfeiffer, CERN/IT-API, AIDA Abstract Interfaces for Data Analysis Andreas Pfeiffer CERN IT/API

Genova 10-Dec-2001 Andreas Pfeiffer, CERN/IT-API, 2 Outline zIntroduction and Motivation zArchitectural issues yAbstract Interfaces zIterative development yWhy software engineering ? zStatus and future plans zSummary

Genova 10-Dec-2001 Andreas Pfeiffer, CERN/IT-API, 3 The AIDA project zAIDA project (Abstract Interfaces for Data Analysis) was initiated at the HepVis’99 workshop in Orsay zPresently active mainly developers from existing packages yTony Johnson (JAS) yAndreas Pfeiffer (Lizard/Anaphe) yGuy Barrand (OpenScientist ) yMark Dönszelmann (Wired) yDevelopers from LHCb/Gaudi

Genova 10-Dec-2001 Andreas Pfeiffer, CERN/IT-API, 4 Introduction zDesign Interfaces for Data Analysis (in HEP) y“The goals of the AIDA project are to define abstract interfaces for common physics analysis tools, such as histograms. The adoption of these interfaces should make it easier for developers and users to select to use different tools without having to learn new interfaces or change their code. In addition it should be possible to exchange data (objects) between AIDA compliant applications.” ( zOpen for contributions of any kind yquestions, suggestions, code, implementations …

Genova 10-Dec-2001 Andreas Pfeiffer, CERN/IT-API, 5 Motivation zUnify/standardize “look and feel” for various tools ythere is no longer “only one tool” zProvide flexibility to interchange implementations of these interfaces ycan use specific features of specific tools w/o change ! zAllows and try to re-use existing packages yeven across “language boundaries” xe.g., C++ analysis using Java Histograms zMinimize coupling between components zAllow for faster turn-around time

Genova 10-Dec-2001 Andreas Pfeiffer, CERN/IT-API, Architectural Issues

Genova 10-Dec-2001 Andreas Pfeiffer, CERN/IT-API, 7 Use of Components with Abstract Interfaces  User Code uses only Interface classes  IHistogram1D * hist = histoFactory-> create1D(‘track quality’, 100, 0., 10.) zActual implementations are selected at run-time yloading of shared libraries zNo change at all to user code but keep freedom to choose implementation yflexible ycustomizable Histo- Impl. 2 Histo-IFFitter-IF User Code Fitter- Impl. Y Histo- Impl. 1 Fitter- Impl. X

Genova 10-Dec-2001 Andreas Pfeiffer, CERN/IT-API, 8 Architectural issue: Abstract Interfaces zAbstract Interfaces yOnly pure virtual methods, inheritance only from other Abstract Interfaces yComponents use other components only through their Abstract Interface yDefines a kind of a “protocol” for a component yAllow each component to develop independently xreduces maintenance effort significantly yMaximize flexibility and re-use of packages xRe-use of existing packages to implement components reduces start-up time significantly

Genova 10-Dec-2001 Andreas Pfeiffer, CERN/IT-API, 9 Abstract Interfaces zAbstract Interfaces yonly pure virtual methods, inheritance only from other A.I. ycomponents use other components only through their A.I. ydefines a kind of a “protocol” for a component yMaximize flexibility and re-use of packages yallow each component to develop independently yre-use of existing packages to implement components reduces start-up time significantly zDe-couple implementation of a component from its use

Genova 10-Dec-2001 Andreas Pfeiffer, CERN/IT-API, 10 Architectural issue: Components (I) zIdentify components by functionality zDefine “protocol” using Abstract Interfaces zEmphasize separation of different aspects for each component yExample: Histogram xstatistical entity (density distribution of a physics quantity) xview of a “collection of data points” (which can be a density distribution but also a detector efficiency curve) xcommand to manipulate/store/plot/fit/... y“User’s view” is different from “implementor’s (developer’s) view” xseparate Abstract Interfaces for both aspects

Genova 10-Dec-2001 Andreas Pfeiffer, CERN/IT-API, 11 Initial Categories and dependencies

Genova 10-Dec-2001 Andreas Pfeiffer, CERN/IT-API, 12 AIDA Workshop Autumn 2000 (Orsay) zStart working from “users view” zConcentrated discussions on yHistogram x“developer” vs. “user” view yManagement xdirectory-like Tree yStore (developer only, hidden from user) yTuple zprepared next round of iterations

Genova 10-Dec-2001 Andreas Pfeiffer, CERN/IT-API, 13 AIDA Workshop Spring 2001 (Boston) zContinued discussion on Interfaces yconcentrated on open issues from Paris workshop and Infrastructure zNew developments yJAida: C++ access to Java libs yJava reference implementation zInfrastructure yCVS repository, mailing lists, web,... zVery constructive zResulted in V-2.0 (“for discussion & feedback”) z==> Iterate

Genova 10-Dec-2001 Andreas Pfeiffer, CERN/IT-API, Iterative development “Why s/w engineering”

Genova 10-Dec-2001 Andreas Pfeiffer, CERN/IT-API, 15 What is so good about iterative development anyway? zImage the project is not to build software but a bridge… zInitial Requirements: A to B AB

Genova 10-Dec-2001 Andreas Pfeiffer, CERN/IT-API, 16 Iteration I zMeets primary requirement: A to B zBasic architecture is in place zSingle user version zCan only be used in winter zNot very safe

Genova 10-Dec-2001 Andreas Pfeiffer, CERN/IT-API, 17 Iteration II zNew requirements: yWorks in the summer yMulti-user zSame basic architecture but different technology zMulti-user version! zCan be used all year round

Genova 10-Dec-2001 Andreas Pfeiffer, CERN/IT-API, 18 Iteration III zNew requirements ymore stable and safe zSame architecture and technology zMore solid construction zExtra security

Genova 10-Dec-2001 Andreas Pfeiffer, CERN/IT-API, 19 Iteration V zNew requirements yProtected from the rain yTwo-way zSame architecture with improved technology zProtected from environment (at least from above) zBi-directional

Genova 10-Dec-2001 Andreas Pfeiffer, CERN/IT-API, 20 Iteration VI zNew requirements: y “I want to move house to B” zSame basic architecture but advanced technology zCan carry other goods

Genova 10-Dec-2001 Andreas Pfeiffer, CERN/IT-API, 21 Iteration VII zNew requirements: y“I want to be able to use my car and let ships go by” zMulti-purpose

Genova 10-Dec-2001 Andreas Pfeiffer, CERN/IT-API, 22 Successful Development Analogy shows successful iterations: ythe basic product existed from the first iteration and met the primary requirement: A to B yearly emphasis on defining the architecture ybasic architecture remained the same over iterations yextra functionality was added at each iteration yeach iteration required more analysis, design, implementation and testing yuse case (requirements) driven does what the users want - not what the developers think is cool

Genova 10-Dec-2001 Andreas Pfeiffer, CERN/IT-API, 23 How to pick what goes in the next iteration? Choice of additions for an iteration is risk driven yEarly development focuses on components with the highest risk and uncertainty xavoids investing resources in a project that is not feasible

Genova 10-Dec-2001 Andreas Pfeiffer, CERN/IT-API, 24 Bad iterations I zDoes not go from A to B zWent for “full functionality” from the start ybig bang approach yface full complexity at the start zUsers/sponsors got cold feet? yRan out of resources, patience y or enthusiasm zRequirements have long since changed yno feedback from users since never used Sounds like the water-fall lifecycle?

Genova 10-Dec-2001 Andreas Pfeiffer, CERN/IT-API, 25 Bad iterations II zDoes not go from A to B any more zInsufficient testing? zUnstable environment? zLack of routine maintenance? zToo many concurrent users? Went straight to the code?

Genova 10-Dec-2001 Andreas Pfeiffer, CERN/IT-API, 26 Legacy systems zStill goes from A to B zBeen in use for a long time zDifficult to determine the original architecture zThe original development team are no longer around zNo documentation zLots of inconsistencies resulting from later additions made with insufficient analysis and design

Genova 10-Dec-2001 Andreas Pfeiffer, CERN/IT-API, 27 Advantages of Iterative and Incremental Development zComplexity is never overwhelming only tackle small bits at a time avoid analysis paralysis and design decline zEarly feedback from users provides input to the analysis of subsequent iterations zDevelopers skills can grow with the project don’t need to apply latest techniques/technology at the start get use to delivering finished software zRequirements can be modified each iteration is a mini-project (analysis, design….)

Genova 10-Dec-2001 Andreas Pfeiffer, CERN/IT-API, Status and future plans

Genova 10-Dec-2001 Andreas Pfeiffer, CERN/IT-API, 29 Status of Components

Genova 10-Dec-2001 Andreas Pfeiffer, CERN/IT-API, 30 Across the languages zJAida : C++ access to Java libs yusing C++ proxies implementing the C++ Abstract Interfaces to the Java interfaces C++UserCode AIDA-IF C++AIDA-IF Java Java Lib JAida

Genova 10-Dec-2001 Andreas Pfeiffer, CERN/IT-API, 31 “Use-cases” of AIDA zJava reference implementation in FreeHEP repository zJAS, OpenScientist and Lizard/Anaphe plan for implementations of version 2.2 by end 2001 zUsed by Gaudi/Athena (LHCb, Atlas, Harp) yGaudi people involved in design zAdopted and used in Geant-4 examples/testing zNo need to go for “least common denominator” yuse “reasonable” superset and concentrate on design

Genova 10-Dec-2001 Andreas Pfeiffer, CERN/IT-API, 32 AIDA XML standards zDefining DTDs yschema will come later (C++) zStarted with 1D and 2D Histograms yaim: easy transfer between applications zWill extend to other data types yother histos, fits, ntuples, … zComments/contributions welcome !

Genova 10-Dec-2001 Andreas Pfeiffer, CERN/IT-API, 33 Time schedule / release plan zV 2.0 mid May 2001 (“Boston release”) yRelease for discussion and feedback yeven if not (yet) “complete” yC++ and Java version of the Interfaces zV 2.1 Aug (“Genova release”) yupdates from discussions at Geant-4 workshop zV 2.2 Nov 2001 ywork on implementations has started yOpenScientist (v8) ready yAnaphe/Lizard and JAS plan for end 2001 zNext iteration on design after feedback yfrom users and implementers

Genova 10-Dec-2001 Andreas Pfeiffer, CERN/IT-API, 34 Summary zAIs unify/standardize “look and feel” for various tools ythere is no longer “only one tool” ycan use specific features of specific tools w/o change ! zDesign (and use) of Abstract Interfaces for Data Analysis yMaximize flexibility and re-use yAllow for faster turn-around time yAllows for and try to re-use existing packages zSoftware engineering provides useful tools/methodologies ytimes of “just writing code” are over yneed to use them to cope with rapidly changing environment

Genova 10-Dec-2001 Andreas Pfeiffer, CERN/IT-API, 35 More information  yUpdated automatically from repository yOn web page links to implementations xfrom whoever provides one (and informs us) zMailing lists (archived) xproject-aida-dev (open) xproject-aida (open) xproject-aida-announce (posting moderated, subscription open)