1. ©1998-2003, Michael Aivazis DANSE Software Issues Michael Aivazis California Institute of Technology DANSE Software Workshop September 3-8, 2003

2. ©1998-2003, Michael Aivazis 2 DANSE DANSE is a “public works” project community infrastructure community development paving the way for future users of large scale facilities ties to the TeraGrid and the national cyberinfrastructure DANSE is scientifically interesting a sociological experiment integration of analysis, modeling and simulation large scale distributed scientific computing human computer interactions in a distributed setting

3. ©1998-2003, Michael Aivazis 3 The Caltech ASCI center Constructing a Virtual Test Facility (VTF) Large, multi-disciplinary project five Caltech research groups and a dozen sub-contractors Application highlights: currently: 500,000 lines of code in C, F77, F90 and C++ massively parallel: ~10K processors large data storage requirements: ~10Tb per run relies heavily on existing code, both solvers and infrastructure Python provides the glue extensibility is an essential feature Pyre: the application framework Python scripts: 1300 classes, 70,000 lines of code C++: 30000 lines of bindings and infrastructure

4. ©1998-2003, Michael Aivazis 4 GeoFramework Simulations of multi-scale deformation in solid earth Geophysics (ITR) GeoFramework is a modeling package that will be usable by the entire Earth sciences community address the limitations of what is currently feasible will be engineered with software evolution and growth as design requirements emphasis on validation Pyre is the simulation framework solver integration and coupling uniform access to facilities accessible to non-experts integrated visualization collaboration extended to include VPAC (Australia) More info at www.geoframework.org

5. ©1998-2003, Michael Aivazis 5 Software engineering practices Version control provides a record of the evolution of the software CVS: well supported, open source Configuration management uniform, portable build procedure automatic, regular builds of the entire software base config: a system based on make merlin: a python-based replacement under development Regression testing test cases that exercise expected behavior exercise fixes for known bugs Bug tracking organize the “to do” list, the feature requests … and the known defects gnats: well supported, open source processors 10 2 10 3 10 4 10 1 10 2 10 3 10 4 10 10 2 configurations developers

6. ©1998-2003, Michael Aivazis 6 Design directions Integrate analysis modules using scripting Python Data flow paradigm well understood easy to implement and document restricted set of data types Meta-data in XML fully reproducible description of the data analysis pipeline tag and archive data record the version number of each module used in the analysis Enable distributed computing XMLRPC, SOAP, OGSA File formats: NeXus + XML meta-data reuse, reuse, reuse augment, contribute HDF5!

7. ©1998-2003, Michael Aivazis 7 User stereotypes Visiting scientist occasional user of prepackaged and specialized analysis tools wants UI that is simple to understand and easy to interact with reasonable defaults for most choices complexity must be concealed intelligently software/hardware failures must be well diagnosed and explained software must answer physics questions Instrument specialist author of prepackaged specialized tools wants tools to write specialized packages for visiting scientist robust modules that form standard analysis packages a GUI builder to compose forms and build wizards

8. ©1998-2003, Michael Aivazis 8 User stereotypes II Expert scientist prospective author/reviewer of PRL paper wants flexible UI that enables interactive explorations of the data access to a comprehensive set of data transformations access to modeling and simulation packages tools to compare the outputs of different analyses high-end, discriminating graphics Analysis expert author of analysis, modeling or simulation software wants well documented access to the integration layer easy access to sample data portable building tools robust framework to facilitate debugging validation/verification infrastructure regression test suites

9. ©1998-2003, Michael Aivazis 9 User stereotypes III Software integrator responsible for extending software with new technology wants portable building tools well documented access to the integration layer easy access to sample data robust framework to facilitate debugging validation/verification infrastructure regression test suites Framework maintainer responsible for maintaining and extending the DANSE infrastructure wants sailing vacation to the British Virgin Islands

10. ©1998-2003, Michael Aivazis 10 Why are we here? Status update architecture implementation prototypes Technical briefing design and implementation strategies suggested technologies detailed look at the steps required to be a contributor Assessment of the community IP analysis, modeling, simulation graphics lessons from other efforts, usable (and not!) Feedback!