February 8, 2000 CHEP2000, Padova M.Kasemann, FNAL1 How to progress fast towards a new OO HEP library? Some ideas on prerequisites and impediments

February 8, 2000 CHEP2000, Padova M.Kasemann, FNAL2 Outline Disclaimer Some thoughts Summary and Outlook

February 8, 2000 CHEP2000, Padova M.Kasemann, FNAL3 Disclaimer This is not a talk about prerequisites: software project management software engineering software architecture data models I have no new ‘silver bullet’ to offer! go and read (one of many good books): “How to run successful projects II” by Fergus O’Connel

February 8, 2000 CHEP2000, Padova M.Kasemann, FNAL4 Disclaimer(2) This is only my biased, imperfect view. The whole talk is meant to trigger discussion.

February 8, 2000 CHEP2000, Padova M.Kasemann, FNAL5 A few observations on software projects Fraunhofer Institute for Software and Systems Engineering (Berlin, Germany): 33% of software projects fail Of those which don’t fail: 80% last substantially longer than planned and cost more than estimated Track record in HENP is of the same order of magnitude!!

February 8, 2000 CHEP2000, Padova M.Kasemann, FNAL6 Software projects: A comparison If you want to build a summer house it is sufficient to employ a bricklayer who will do it using his experience. If you want to build a skyscraper or a bridge you need foremen, engineers, architects and an agreed process. Nobody would do it with a 1000 bricklayers only!!! (except physicists) A bridge lasts 100 years, very good software lasts years at most, much software in HEP experiments survives < 5 years.

February 8, 2000 CHEP2000, Padova M.Kasemann, FNAL7 How to progress fast towards a new OO HEP library? Using OO in HENP for now is a fact, and a requirement. We should careful evaluate which parts are HENP specific. CERNLIB style libraries: Quite some code and algorithms are ‘reinvented’ over and over => incompatible

February 8, 2000 CHEP2000, Padova M.Kasemann, FNAL8 Fergus O’Connel: The silver bullet (1) Planning the [software, mka ] project 1: Visualize the goal; set your eyes on the prize 2: Make a list of jobs to be done 3: There must be one leader 4: Assign people to jobs 5: Manage expectations, allow a margin error, have a fallback position

February 8, 2000 CHEP2000, Padova M.Kasemann, FNAL9 Fergus O’Connel: The silver bullet (2) Implementing the plan/Achieving the goal 6: Use an appropriate leadership style 7: Know what is going on 8: Tell people what is going on 9: Repeat 1-8 until 10 10: The prize (“oh what fun, my job is done”)

February 8, 2000 CHEP2000, Padova M.Kasemann, FNAL10 Planning - 1: Visualize the goal A new OO HEP library?!! Contains all experiment-non-specific but HEP- specific algorithms Elements for common solutions for HEP problems Robust, debugged, available for different OS’s compatible to all or most commonly used non- HEP packages Leverage knowledge from public domain or IT industry allows HEP to concentrate on HEP-specific problems May need adaptation for HEP

February 8, 2000 CHEP2000, Padova M.Kasemann, FNAL11 Example: CMS Software Tasks This talk Why not have a HEP framework?

February 8, 2000 CHEP2000, Padova M.Kasemann, FNAL12 Issues for Core Software Types Non - HEP software Version evolution on multi-platforms Distribution and possible licensing Cost Long term availability of commercial product HEP Software Manpower costs for development and maintenance Technical requirements for “common items” Layered Architecture Standard Application Program Interfaces (APIs) Phasing: early linkage with key experiment-specific architecture decisions

February 8, 2000 CHEP2000, Padova M.Kasemann, FNAL13 Example: LHC ++ LHC++ is a HEP-wide effort to provide a suite of OO software covering roughly the functionality of CERNLIB Use public domain or commercial SW when appropriate Develop own HEP software if necessary Collect and distribute SW; take care of possible licensing Develop a well-engineered toolkit of software components Build interface layers for HEP usage Develop/acquire, maintain and replace components separately Build different applications from components Use standards and standard solutions Use of standards may help in maintaining and developing software over the long lifetime of the experiment LHC++ has ‘one big’ problem: already the name indicates that is it not meant to be a HEP-wide solution ;-)

February 8, 2000 CHEP2000, Padova M.Kasemann, FNAL14 LHC++ components Standard components C++ libraries, STL: Object Space Basic mathematics: NAG C library ODBMG database: Objectivity/DB Standard graphics: OpenGL, OpenInventor, Qt,... HEP components Simulation tools: Pythia7, Geant4 Data analysis, graphics, visualisation: HEPVis, HEPTags,… Mathematical libraries: Gemini, HEPfitting Histogramming: HTL Foundation level class libraries: CLHEP, HepODBMS

February 8, 2000 CHEP2000, Padova M.Kasemann, FNAL15 Planning - 3: There must be one leader A leader is required who can lead who is leading who is identified with the project and wants to be successful One and only one leader is required, not a committee. Bridges are not build by 1000 bricklayers who do what they want.

February 8, 2000 CHEP2000, Padova M.Kasemann, FNAL16 Implementation - 7: Know what is going on Use the plan to guide the project.

February 8, 2000 CHEP2000, Padova M.Kasemann, FNAL17 Implementation - 8: Tell people what is going on Progress reviews, updated ‘work-plans’ or WBS’s. Good and bad surprises should be published instantly in order to adapt.

February 8, 2000 CHEP2000, Padova M.Kasemann, FNAL18 Implementation - 9: Repeat 1-8 until 10 Be prepared to take long term commitment to finish project there is no shortcut to finish the project without cutting back in requirements. need to compromise on short-term goals

February 8, 2000 CHEP2000, Padova M.Kasemann, FNAL19 Implementation - 10: The prize Now the maintenance starts: needs resources and commitment, again: conflict of interest for scientists who want to operate experiment and do HEP science. We have to get serious: Learn to involve professionals from IT disciplines (like engineers in detector building). HEP software designers needs a career path like detector designers and builders. We should assign a higher status to those who release their scientific code with open source licenses.

February 8, 2000 CHEP2000, Padova M.Kasemann, FNAL20 How to progress fast towards a new OO HEP library? My preferred model is: an idea/goal is connected to a person (leader, PI) an approval process happens resources must be allocated at universities, IT-divisions, in experiments progress is tracked feedback and early buy-in by customers operations and maintenance phase must be covered. Use ‘silver bullet’ steps This is how experiments are build, also applied to successful software projects.

February 8, 2000 CHEP2000, Padova M.Kasemann, FNAL21 How to convince experiments to use a new OO HEP library? Experience shows: It is hard to impossible to coordinate or enforce usage of common products. Different time-scales and decisions in experiments make common development difficult. Chance for Success: Persuade with good quality software and functionality. As systems become more complex, calculations that are verifiable in principle are not verifiable in practice without public access to the code. Authors of open scientific code should be recognized in the same way as they are recognized for scientific papers.

February 8, 2000 CHEP2000, Padova M.Kasemann, FNAL22 Summary and Outlook We have to develop and grow the understanding of the benefit of a common library to achieve a common long-term goal. Good ideas and leadership must be acknowledged and supported with resources. We want the best algorithms and modules available to HEP, this needs the best people! The academic career path for HEP software builders must be better developed. Publishing code should be equivalent to publishing a paper!

February 8, 2000 CHEP2000, Padova M.Kasemann, FNAL23 Planning - 2: Make a list of jobs to be done

February 8, 2000 CHEP2000, Padova M.Kasemann, FNAL24 Planning - 4: Assign people to jobs Each job has a name

February 8, 2000 CHEP2000, Padova M.Kasemann, FNAL25 Planning - 5: Manage expectations, allow a margin error, have a fallback position

February 8, 2000 CHEP2000, Padova M.Kasemann, FNAL26 Implementation - 6: Use an appropriate leadership style