1. NERSC High Energy and Nuclear Physics Computing Group
Craig E. Tull
HENPCG/NERSC/LBNL
2006 Director’s Review of Computing
LBNL - September 22, 2006
NERSC High Energy and Nuclear Physics Computing Group
Craig E. Tull
HCG/NERSC/LBNL
2005 Science Colloquium Series
DOE - August 23, 2005

2. HENP Computing Group Group Leader: Craig Tull
1 Senior Scientist, 4 Scientists, 6 Engineers, 1 Postdoc
Embedded software professionals w/ science backgrounds
Provide computing systems & support for science collaborations
Mostly software focus, but with a track record for integrated software and hardware systems
Scientists focus on science and requirements on the software rather than detailed design or implementation.
Leave non-scientific code to computing professionals with expertise and time to apply software process.
Management and Leadership Roles
Software and Technology Development
Software Engineering Best Practices and Support

3. HENP Environment Large, distributed collaborations are the norm
~2000 scientists, from ~150 institutions in ~50 countries
Scientists require equal access to data and resources
Very long time duration of projects & software
Detectors take 5-10 years to design and build
Detectors have an operational lifetime of 5-20 years
10 to 30 year Project lifetimes
Strong focus on robust, maintainable software supporting graceful evolution of components & protocols
Commodity computing (Intel, Linux)
Polite parallelism/Partitioning of calculations
Data Intensive (100's TB => 1,000's TB)
The World is Networked
Scientists are developers and not just users
Many skill levels from Wizard to Neophyte
Issues of scaling are sociological as well as technical

4. HENPC Group Role Within a Project
Software professionals with scientific background as full collaborators
Establishment of a software development process
Adapt CSE methodologies and processes to HENP environment
Object Oriented, Architectural
Aspects of Unified Software Development Process (USDP) and Extreme Programming (XP)
Design and implementation of software development infrastructure
Code repository, release build, test, and distribution systems
Design and implementation of major software components
Control frameworks
Online experiment control
Data persistency frameworks
Physics toolkits
Training and mentoring
Tutorials, code guidelines, requirement/design/code reviews, etc.
Re-engineering of existing designs
Provide expertise to improve robustness, performance, maintainability

5. HENPC Group Role across Projects
Promote a common culture
Best practices, open source, architecture, code reuse
Develop and integrate tools that support these best practices
Explore and integrate new technologies
Object Oriented Database Systems
CORBA based distributed systems
GRID integration
C++, Java, Python
J2EE, JBoss, JMX
Generate an Institutional knowledge base
User Requirements
Architectural decomposition
Components
Leverage coupling between NERSC and Physical Sciences at LBNL

6. HENPC Projects ( )

7. Experiments & Projects (2004-2006)
ATLAS LHC accelerator at CERN, Geneva
Software lead, Chief Architect, Core software & Athena Gaudi framework
BaBar PEP-II collider at SLAC, Stanford
Conditions Database,
IceCube Neutrino detector at South Pole
Chief Architect, Experiment Control, Build Environment, Offline Framework
Majorana Neutrinoless double beta decay, LDRD
GEANT4 build system, GEANT4 Geometry Database
SNAP Supernova satellite
Java Simulation Framework
GUPFS Global Unified Parallel File System
Management and Scientific Liason
SNF Super Nova Factory, Telescope, Hawaii
Lisp-based Observation Scheduling Software
PPDG Particle Physics Data Grid
Replica catalogs technical survey, Security & VO roles

8. The ATLAS Experiment: A large HEP project

9. LHC pp → search for rare processes with small σ (N = Lσ )
• √s = 14 TeV (7 times higher than Tevatron/Fermilab)
→ search for new massive particles up to m ~ 5 TeV
• Ldesign = 1034 cm-2 s (>102 higher than Tevatron/Fermilab)
→ search for rare processes with small σ (N = Lσ )
LHC pp
ATLAS and CMS :
pp, general purpose
27 km ring used for
e+e- LEP machine in
ALICE :
heavy ions
Start : Summer 2007
LHCb :
pp, B-physics

10. ATLAS Tracking (||<2.5, B=2T) : Length : ~ 46 m Radius : ~ 12 m
Weight : ~ 7000 tons
~ 108 electronic channels
~ 3000 km of cables
Tracking (||<2.5, B=2T) :
-- Si pixels and strips
-- Transition Radiation Detector (e/ separation)
Calorimetry (||<5) :
-- EM : Pb-LAr
-- HAD: Fe/scintillator (central), Cu/W-LAr (fwd)
Muon Spectrometer (||<2.7) :
air-core toroids with muon chambers

11. ATLAS Collaboration 34 Countries 151 Institutions
1770 Scientific Authors
Albany, Alberta, NIKHEF Amsterdam, Ankara, LAPP Annecy, Argonne NL, Arizona, UT Arlington, Athens, NTU Athens, Baku, IFAE Barcelona, Belgrade, Bergen, Berkeley LBL and UC, Bern, Birmingham, Bonn, Boston, Brandeis, Bratislava/SAS Kosice, Brookhaven NL, Bucharest, Cambridge, Carleton, Casablanca/Rabat, CERN, Chinese Cluster, Chicago, Clermont-Ferrand, Columbia, NBI Copenhagen, Cosenza, INP Cracow, FPNT Cracow, Dortmund, JINR Dubna, Duke, Frascati, Freiburg, Geneva, Genoa, Glasgow, LPSC Grenoble, Technion Haifa, Hampton, Harvard, Heidelberg, Hiroshima, Hiroshima IT, Indiana, Innsbruck, Iowa SU, Irvine UC, Istanbul Bogazici, KEK, Kobe, Kyoto, Kyoto UE, Lancaster, Lecce, Lisbon LIP, Liverpool, Ljubljana, QMW London, RHBNC London, UC London, Lund, UA Madrid, Mainz, Manchester, Mannheim, CPPM Marseille, Massachusetts, MIT, Melbourne, Michigan, Michigan SU, Milano, Minsk NAS, Minsk NCPHEP, Montreal, FIAN Moscow, ITEP Moscow, MEPhI Moscow, MSU Moscow, Munich LMU, MPI Munich, Nagasaki IAS, Naples, Naruto UE, New Mexico, Nijmegen, BINP Novosibirsk, Ohio SU, Okayama, Oklahoma, LAL Orsay, Oslo, Oxford, Paris VI and VII, Pavia, Pennsylvania, Pisa, Pittsburgh, CAS Prague, CU Prague, TU Prague, IHEP Protvino, Ritsumeikan, UFRJ Rio de Janeiro, Rochester, Rome I, Rome II, Rome III, Rutherford Appleton Laboratory, DAPNIA Saclay, Santa Cruz UC, Sheffield, Shinshu, Siegen, Simon Fraser Burnaby, Southern Methodist Dallas, NPI Petersburg, Stockholm, KTH Stockholm, Stony Brook, Sydney, AS Taipei, Tbilisi, Tel Aviv, Thessaloniki, Tokyo ICEPP, Tokyo MU, Tokyo UAT, Toronto, TRIUMF, Tsukuba, Tufts, Udine, Uppsala, Urbana UI, Valencia, UBC Vancouver, Victoria, Washington, Weizmann Rehovot, Wisconsin, Wuppertal, Yale, Yerevan

12. ATLAS Computing Characteristics
Large, complex detector
~108 channels
Long lifetime
Project started in 1992, first data in 2007, last data 2027?
320 MB/sec raw data rate
~3 PB/year
Large, geographically dispersed collaboration
1770 people, 151 institutions, 34 countries
Most are, or will become, software developers
Programming abilities range from Wizard to Neophyte
Scale and complexity reflected in software
~1000 packages, ~8000 C++ classes, ~5M lines of code
~70% code is algorithmic (written by physicists)
~30% infrastructure, framework (written by sw engineers)
HENPC responsible for large portion of this software
Provide robustness but plan for evolution
Requires enabling technologies
Requires management & coherency

13. Software Methodology Object-Oriented using C++ as programming language
Some wrapped FORTRAN and Java
Python as interactive & configuration language
Heavy use of components behind abstract interfaces
Support multiple implementations
Robustness & evolution
Lightweight development process
Emphasis on automation and feedback rather than very formal process
Previous attempt at developing a software system had failed due to a too rigorous software process decoupled from developers
Make it easy for developers to do the “right thing”
Some requirements/design reviews
Sub-system “functionality” reviews
2 weeks each
Focus on client viewpoint

14. Athena/Gaudi Component Model
Application
Manager
Converter
Converter
Converter
Event
LoopMgr
Event Store
Data
Files
Message
Service
Persistency
Service
StoreGateSvc
JobOptions
Service
Algorithm
Sequencer
Algorithm
Data
Files
Particle Prop.
Service
Detector
Store
StoreGateSvc
Persistency
Service
Athena-Gaudi Object Diagram
The main components of the architecture can be seen in the slide.
The green boxes play the role of the Control component in the blackboard diagram
The Algorithms are the Knowledge sources
The StoreGateSvc provides an API to access Data Store (the blackboard)
The conversion mechanism (in blue) act as bridge between the framework and the specific technology of persistency mechanism.
Other algorithm services are in grey
Other
Services
Data
Files
Histogram
Store
Histogram
Service
Persistency
Service
Auditors

15. Athena Components Algorithms Provide basic per-event processing
Share a common interface (state machine)
Sequencer is type of Algorithm that sequences/filters other Algorithms
Tools
More specialized but more flexible than Algorithms
Services
E.g. Particle Properties, Random Numbers, Histogramming
Data Stores (blackboards)
Data registered by one Algorithm/Tool can be retrieved by another
Multiple stores handle different lifetimes (per event, per job, etc.)
Stores accessed via Services (e.g. StoreGateSvc)
Converters
Transform data from one representation to another
e.g. transient/persistent
Properties
Adjustable parameters of components
Can be modified at run-time to configure job

16. ATLAS Computing Organization

17. HENPC Group within ATLAS
David Quarrie
Software Project Lead
Paolo Calafiura
Chief Architect
Core Services Group Lead
Wim Lavrijsen
Python configuration and support tools
Charles Leggett
Athena/Gaudi framework support
Martin Woudstra
Integration of Athena with production system

18. FY06-07 Major Activities
Computing support for ATLAS Detector Commissioning
Electronics & detector calibrations
Cosmic ray tests
Computing System Commissioning (CSC)
Commissioning of the Computing and Software system itself
High Level Trigger Large Scale Tests
Offline software components used in HLT
Athena framework
Tracking & calorimetry algorithms

19. ATLAS (some highlights)
Management & Leadership Roles
ATLAS Software Project Lead: David Quarrie
ATLAS Chief Architect: Paolo Calafiura
Previously D.Quarrie
US ATLAS Core Software Manager: Paolo Calafiura
Previously D.Quarrie, C.Tull
Software and Technology Development
Athena Control & Analysis Framework
PyROOT: Introspection-driven ROOT Python interface
StoreGate: Object Transient Store
Software Engineering Best Practices and Support
Nightly Build & Release Campaign
Dozens of tutorials and extensive documentation
ASK (Athena Startup Kit): Robust GUI for build system
Hephaestus: Low-overhead Memory Profiler

20. IceCube Management & Leadership Roles Software Architect: Simon Patton
Experiment Control: Simon Patton
Software and Technology Development
Ice Tray: Component-based Analysis Framework
JBoss/JMX Control Architecture
Hierarchical State Machine
Web Portal interface
Software Engineering Best Practices and Support
BFD (Baseline File Development): UML based develop, build, release system.
Tutorials & Developer Support

21. Heirarchical State Machine
IceCube Computing
System Architecture and Development (Simon Patton)
Strong coherent vision for all IceCube software.
Laying out "best practices" to follow to ensure good code.
Development environment, tools supporting best practices.
Experiment Control (Simon Patton, Chris Day, Akbar Mohktarani)
Layered State Machine control of components of data flow
Uses J2EE, JBoss/JMX
Heirarchical State Machine

22. SNAP Management & Leadership Roles
Computing Framework Architect: Gary Kushner
SNAP Collaboration Systems Manager Group
Represent SNAP computing w/ Bill Carithers
Software and Technology Development
Computing Framework & Monte Carlo (Java-based)
Simulate the Universe being observed
Simulate the Instrumentation and Detecton Process
Simulate the Extraction of Cosmological Parameter from Mission Data
Software Engineering Best Practices and Support
Ant-based build environment
Shrink-wrapped deployment package (out-of-box experience)
Redesign/implementation of Physics Codes: up to X15 speedup

23. Supernova Mission Simulator
With our Monte Carlo:
Have simulated SNAP with detector characteristics and observing program
Have simulated other potential experiments including ground-based instruments
Use state-of-the-art SNe models that simulate SNe population drift with redshift
Included systematic effects and calibration errors
Can generate error ellipses for cosmological parameters
Can optimize SNAP

24. High Level CS Architecture

25. BaBar Management & Leadership Roles
Previously Chief Architect: David Quarrie
Previously Database Head: Simon Patton
Software and Technology Development
Conditions DataBase (CDB): Igor Gaponenko & Akbar Mokhtarani
Historically:
Object Oriented Database (Objectivity)
Offline & Online software & General applications
Software Engineering Best Practices and Support
Refactorizing all BaBar database applications to newer persistency technologies (ROOT I/O & MySQL)
Expert-level support for distributed database management & tools
Consultation on Database Software

26. CDB Concepts : Scope & Ownership Diagram
MIR
DATABASE
ORIGIN
provides scope
for
owns
PHYSICAL CONDITION
(2-D SPACE)
P-LAYOUT
(2-D SPACE)
VIEW
REVISION
PARTITION
An finally, here is a simplified diagram showing how the previously mentioned components relate to each other. Each upper level components provides scope for the lower level ones, and it also owns them.
INSTRUCTIONS: Don’t waste the time on this slide!!!
FOLDER
uses
VISIBLE INTERVAL
ORIGINAL INTERVAL
USER DATA OBJECT
CONFIGURATION

27. Majorana Software and Technology Development
Centralized MySQL database for MaGe (GEANT4) material properties and geometry
Schema, API, implementation, and support
Software Engineering Best Practices and Support
Unified software management to ensure code integrity across participating institutions
Incremental build and test system
General application and development support
Geant4, Root, CLHEP, Dawn, etc

28. “MaGe” Simulation Package.
Framework uses powerful object-oriented and abstraction capabilities of C++ and STL for flexibility
Majorana-related
detector geometries
Gerda-related
detector geometries
MaGe
Geant 4/ ROOT
Event Generators
Common geometries
Physics processes
Majorana-related
output
Gerda-related
output

29. MaGe Activities Previous Activities:
Characterization of radioactive backgrounds in conceptual design for NuSAG proposal.
Interpretation of results from highly segmented detector at MSU.
TUNL-FEL Run
Charge Distribution in 0nbb-decay
Current Activities:
Full characterization of Majorana reference design and optimization of segmentation scheme.
Neutron background from muons in rock
Alpha contamination on surfaces.
Pulse-generation.
Gerda
Posters at Neutrino04,
TAUP05

30. Challenges & Opportunities
Many other science disciplines are growing in size, distribution, and data volume
HENP lessons learned should be leveraged
Non-HENP techniques of interest to HENP
New HENP experiments/projects:
SNAP, Majorana, Daya Bay, Gretina
"Old" experiments/projects:
BaBar, ATLAS, IceCube
Lack of any base funding for HENPC:
Problems with long-term stability & predictability
Great difficulty jump-starting new projects
Common use software, libraries & tools a "volunteer" effort

31. Summary of HENPC Group A small group with involvement in many projects
Have had a major impact in the field
Leaders in use of Object Oriented programming and software process
Leaders in use of Object Oriented Databases
Control Frameworks in use by many experiments
Not just those we’ve worked directly on
Demonstrated ability to leverage our expertise and empower large, dispersed, software teams
Demonstrated ability to design and implement large scale scientific computing systems
Keeping abreast of modern computing techniques and tools, but delivering concrete, robust software for production use.