1. Extending the Neutron Scientist’s Toolkit Michael McKerns Materials Science and Applied Physics Center for Advanced Computing Research California Institute of Technology (Distributed Data Analysis for Neutron Scattering Experiments):

2. Part I:Overview of DANSE Part II:The Scientist’s Toolkit

3. What is DANSE? a 12M$ five-year NSF IMR-MIP software construction project a collaborative effort between software professionals, neutron scattering scientists, and facilities a software engineering effort –open-source development environment –framework for the interoperability of modular components –integration of legacy codes and community-standard software –connectivity to facility databases and software repositories a scientific endeavor –to develop software modules for different subfields of neutron scattering –to enhance neutron scattering research and facilitate new science –to build tools for education, collaboration, and plausibility assessment an integration framework for building data analysis, visualization, modeling, and instrument simulation tools for all areas of neutron scattering

4. Serving a Growing Community With the availability of SNS, OPAL, and J-PARC fast approaching, the neutron community has the potential to undergo a large growth spurt. Software is a vital part of scattering research, and unless the software is both robust and easy to use, that growth may be limited. Mature packages do exist (McStas, ISAW, DAVE, …), and commercial packages are also used (Matlab, IDL, Abaqus, IGOR Pro, …) in the analysis process. However, groups often use cryptic legacy code for at least one step. To grow as a community, we need: –a way to cultivate and maintain the valuable portions of these legacy codes –to make legacy and community-standard codes interoperable define common data structures and interfaces stop duplication of effort –to allow scientists to concentrate on science by lowering barriers to software engineering –to allow programmatic integration of analysis software with facility databases & libraries

5. There is much to do… Software is needed to support the massive quantity of data that will be produced at modern neutron facilities. However, existing software may be incapable of utilizing the full richness of the data that will be produced. Although the barrier to developing new software must be reduced, it is also critical that more complex software technologies (i.e. high-performance and grid-based computing) are enabled. Time is short – we must use the best existing tools to provide a robust solution… yet be flexible enough to allow for the easy substitution of better future solutions. The fundamental commodity for neutron scattering software is found within the cores of time-tested community-standard software. Rather than rewrite or duplicate this software, we can use DANSE to provide an integration path for both new and legacy codes into a common environment.

6. Building a Scientific Toolkit Through DANSE, scientists will have access to many tools –basic data structures, optimization algorithms, numerical libraries –basic data reduction library [obtain I(Q), S(Q), S(E), S(Q,E)] –graphical/plotting environments IDL, Matlab, Matplotlib, Gnuplot, Grace, ParaView, ACIS (AutoCAD), … –instrument simulation McStas, VITESS, sample simulation framework, … –materials simulation ABINIT, VASP, GAMESS, NWChen, NAMD, CHARMM, … –crystallography cctbx, FOX, ObjCryst++, … –molecular viewers and format translators OpenBabel, Molden, PyMol, ViewMol, DRAWxtl, VMD, AtomEye, … –and MORE! ISAW, texture analysis (MAUD), SLD calculator, scattering intensity, … These tools will be available in DANSE as component libraries

7. Component Implementation build core engine (Python, Fortran, C++, Java, Matlab, IDL, …) –legacy or custom code and third-party libraries –provide life-cycle management and exception handling strategy construct Python bindings –select entry points to expose to Python –modularize entry points to monolithic compiled libraries cast as a component –extend and leverage framework services –describe user-configurable parameters –provide meta-data that specify the IO port characteristics test code –satisfy functional requirements with concurrent test development –utilize interactive runtime testing within Python interpreter –demonstrate integration with other components Component CORE

8. Component-based Applications granularity allows reusability of object-oriented components rebinning application modularity provides flexibility and extensibility NeXusReader Selector Bckgrnd Selector Energy NeXusWriter times instrument info raw counts filename time interval energy bins filename

9. One Code, Many Interfaces… DANSE applications are designed to be UI agnostic. A clean separation between UI and application code allows for: –maximum code reusability –the use of automated tools for rapid UI construction automated form and wizard builder widget and view configuration same look-and-feel generated for a web and GUI application DANSE applications can be launched: –from command line executables; or built dynamically in the command shell –from within a dashboard UI; or built dynamically in a Visual Programming Environment

10. Broad Scope of Applications data reduction and experiment simulation –diffraction, engineering diffraction, and inelastic scattering data reduction –SANS/USANS and neutron reflectometry data reduction –instrument and microstructure simulation modeling –full profile modeling in real and reciprocal space (GSAS, FullProf, PDFFIT) –finite element modeling (ABAQUS); self-consistent modeling –constrained fitting by use of data from other experimental techniques –1D/2D model fitting; model independent peak fitting –direct modeling of physical systems; ab-initio modeling –scattering kernel; multiple scattering –neutron weight correction; separation of nuclear and spin scattering –micromagnetic simulations (OOMMF); disordered spin dynamics –chemical spectroscopy dynamics (CLIMAX)

11. Facilitates New & Better Science better data analysis –FEM calculations of strains in microstructures –Monte-Carlo inversions of S(Q,E) to obtain parameters of structure and dynamics models –model refinements with multiple data sets integration of theory –micromechanics using correlations of local strains –phase diagrams from thermodynamic functions –ab-initio calculations of spin interactions –soft matter structure using atomic force fields guided by diffraction experiment planning and execution –single crystals on chopper spectrometers –feedback control and real-time assessment –plausibility testing and contingency planning –assessment of science/data trends from previous data

12. DANSE cycle of life Every subgroup is interested in the same analysis steps, we should be able to reuse much of the analysis… Raw datakernelReal space reduction inversion simulationfitting

13. Part I:Overview of DANSE Part II:The Scientist’s Toolkit

14. Building a Scientific Toolkit… Through DANSE, scientists will have access to many tools –basic data structures, optimization algorithms, numerical libraries –basic data reduction library [obtain I(Q), S(Q), S(E), S(Q,E)] –graphical/plotting environments IDL, Matlab, Matplotlib, Gnuplot, Grace, ParaView, ACIS (AutoCAD), … –instrument simulation McStas, VITESS, sample simulation framework, … –materials simulation ABINIT, VASP, GAMESS, NWChen, NAMD, CHARMM, … –crystallography cctbx, FOX, ObjCryst++, … –molecular viewers and format translators OpenBabel, Molden, PyMol, ViewMol, DRAWxtl, VMD, AtomEye, … –and MORE! ISAW, texture analysis (MAUD), SLD calculator, scattering intensity, … These tools will be available in DANSE as component libraries

15. Integration Strategy for Data Reduction In October 2006, DANSE & SNS teams began to work together to define a strategy for data reduction and analysis SNS to provide basic data reduction for all SNS instruments DANSE will couple reduction to analysis and simulation - post-reduction corrections - analysis tools used in reduction DANSE & SNS must work together to - ensure each other’s requirements are met - standardize data structures & define a clear interface - leverage each other’s effort

16. Several Tools Requested In November 2006, DANSE representatives met with several SNS instrument scientists to collect input on desired analysis tools. GUI Builder –extract user-settable fields and defaults from applications & components –configure view & widgets with simple XML description Peak Fitter –obtain peak information from noisy data –allow selection of fitting methods Detector Viewer –examine raw data –look for bad detectors, create masks Histogram Viewer –provide physical representation at each reduction step –aid visualization of slicing and complex operations

17. Collaboration Through Modular Design API creates standardized interface, and facilitates modular design Transforms from either SNS or DANSE library Standardized data objects (histograms) passed between each transform Histogram Viewer SNS Reduction Transform DANSE Reduction Transform API DANSE Rebinner API histogram

18. Histogram is a fundamental data structure for data reduction Reduction can be represented by mapping an I(pixel,tof) histogram to a S(Q,E) histogram over the instrument information [inelastic] Extensibility by Design Abstraction Raw Data Histogram Mapping

19. Making Reduction Instrument-Agnostic Can we separate instrument-specific parts of data reduction, so that transformations are reusable across instruments? First, we identify & isolate instrument-specific pieces - instrument definition migrated to instrument data object - extracting histograms from instrument run files dissociated to measurement Then, we reconnect with abstract interfaces Reduction & simulation become instrument-independent - supporting a new instrument now may be very simple… - add an instrument description & new class to extract data from run files

20. Standardized Reduction Data Objects

21. Define Reduction & Standardize Interface

22. Extend Reduction to Inelastic Instruments

23. Example Reduction Components detCalibrator –computes calibration constants and calibrates histograms normalizer –normalizes histograms by incident neutron flux timeBG –removes time-independent bg from histogram speReducer –reduces data to S(phi,E) Spe2Sqe –converts S(phi,E) to S(Q,E) e_iSolver –computes neutron incident energy darkAngleFinder –determines dark angle for vanadium run

24. A Closer Look at Instrument Instrument is a reusable instrument definition –is built from available instrument elements and their relative positions –element descriptions are maintained separately from element position –instrument description may be obtained from NeXus file Instrument description is a hierarchy of elements –FermiChopper, T0Chopper, Moderator, Monitor, DetectorArray, DetectorTube, Guide, –SampleEnvironment: Cryomagnet, Cryogoniometer, … Instrument Viewer –provides a 3-D representation of instrument or instrument elements Instrument Builder –drag-and-drop interface for instrument construction (proposed)

25. Building an Instrument moderatormonitor

26. Monte-Carlo Instrument Simulation Simulation framework connects sources, samples, detectors, and other instrument elements - by providing a buffer to handle neutron flow between components - by providing an engine to drive the neutrons Connecting through the neutron component API allows exchange of newly built components with those provided by McStas, Vitess, … moderatormonitor McStas moderator python bindings neutron component API

27. Sample Simulation composite sample assembly required –a sample is usually inside some kind of container –a sample simulation needs to take into account a collection of scatterers including the sample and other objects composite scatterers also required –available sample simulations usually focus on one kind of scattering mechanism –a full simulation should take into account all possible scattering mechanisms with similar scattering strength clean separation of physics properties and geometry –reuse geometer as coordinate environment –shapes & sample description –random number generator –scattering path calculator sample containersample

28. Building a Composite Scatterer Coherent inelastic phonon scattering kernel Sample Assembly Collection of scatterers Aluminum Can Collection of scattering kernels Incoherent inelastic phonon scattering kernel Fcc Ni Sample Shape Collection of scattering kernels Shape possible kernels: - phonons, magnons - elastic, inelastic - coherent, incoherent - … kernels provide scattering probability physics is contained in scattering kernels

29. Define Instrument & Sample Simulation

30. DANSE Project Information milestones for the DANSE software –project start2006 –beta release2008 –release 1.02009 –transition to community/SNS2011 documentation, tutorials, and further information –the DANSE wiki at http://wiki.cacr.caltech.edu/dansehttp://wiki.cacr.caltech.edu/danse –the DANSE trac page at http://danse.us/trac/tickets/wikihttp://danse.us/trac/tickets/wiki –the ARCS software page at http://www.cacr.caltech.edu/projects/ARCS/Software.htmlhttp://www.cacr.caltech.edu/projects/ARCS/Software.html –the Pyre homepage at http://www.cacr.caltech.edu/projects/pyrehttp://www.cacr.caltech.edu/projects/pyre contacts –Brent Fultz btf@caltech.edu; Michael Aivazis, Ian Andersonbtf@caltech.edu –Simon Billinge, Ersan Üstündag, Paul Butler, Paul Kienzle, Tom Swain –Michael McKerns mmckerns@caltech.edummckerns@caltech.edu

31. End Presentation