Neutron Science TeraGrid Gateway Update Vickie Lynch, Meili Chen, John Cobb Oak Ridge National Laboratory AUS Technical Presentation October 8, 2009

2Managed by UT-Battelle for the U.S. Department of Energy Outline Sinergia Startup Project Fitting Service Transition to SNS

3Managed by UT-Battelle for the U.S. Department of Energy Sinergia Startup Project Collaboration between the University of Zürich, ETH Zürich and Oak Ridge National Laboratory (ORNL) Swiss Sinergia grant supports PostDoc at Zürich and graduate student at ORNL

4Managed by UT-Battelle for the U.S. Department of Energy Diffuse Scattering Modeling 3D fitting of large datasets Needed for Single Crystal instruments – Topaz (SNS) – Snap (SNS) – MaNDi (SNS) – Corelli (SNS) – Four-circle Diffractometer (HFIR) – Imagine (HFIR)

5Managed by UT-Battelle for the U.S. Department of Energy Sinergia July 09 Kickoff in Zurich Organized by Hans-Beat Bürgi Michel Chodkiewicz (PostDoc at Zürich) will develop new C++ diffuse scattering code over 3 years Tara Clark (Grad student at ORNL) working with Vickie Lynch will get previous code running on TeraGrid and test Jürg Hauser from University of Bern will provide a reference dataset with less noise

6Managed by UT-Battelle for the U.S. Department of Energy Original diffuse scattering code Used for publications: – Weber T.,Bürgi H.B., Determination and refinement of disordered crystal structures using evolutionary algorithms in combination with Monte Carlo methods, Acta Crystallographica A 58, (2002) – H.B. Bürgi, J. Hauser, T. Weber, R.B. Neder, Supramolecular Architecture in a Disordered Perhydrotriphenylene Inclusion Compound from Diffuse X-ray Diffraction Data. Crystal Growth & Design, (2005) Master/slave code was written mainly in Perl with telnet to communicate between workstations (Calls Fortran and C codes for FFTs) Much file communication between master and slave – Code assumed unique disk space for each slave One run for over 200 generations took ~30 days on ~10 workstations

7Managed by UT-Battelle for the U.S. Department of Energy VOP code by Phil Bentley Vitess Optimisation Program written for optimizing parameters in instrument simulations Genetic algorithm with tournament selection; Swarms Written in C++ using MPI for parallel computing (Allreduce to find min; Bcast to sent best parameters) Command line input for Vitess with parameters to be optimized preceded Limits file read by code Written to maximize flux from Vitess, but easily modified to minimize  2

8Managed by UT-Battelle for the U.S. Department of Energy Code port to TeraGrid Replaced master telnet code with VOP MPI code VOP calls Perl slaves with pipes Unique directories created for each slave Ported and ran successfully on NSTG Needed faster turnaround and 1048 clones Got Kraken Startup allocation Popen for Perl slaves did not work with CNL

9Managed by UT-Battelle for the U.S. Department of Energy Code port to Ranger Tested that popen worked on Ranger – Transferred part of Startup to Ranger Too much file communication in Perl slaves for 16 cores/node Put OpenMP directives in FFT codes Using 4 slaves/node with 4 OpenMP threads/slave

10Managed by UT-Battelle for the U.S. Department of Energy VOP results with Perl slaves

11Managed by UT-Battelle for the U.S. Department of Energy Sinergia accomplishments Add differential evolution option to vop master Compared differential evolution, genetic algorithm and swarm for disordered crystal data. Genetic algorithm converged fasted, but each generation takes twice as long using tournament selection Added hybrid OpenMP directives Have code running on NSTG and Ranger Have Levenberg-Marquardt nonlinear least squares driver for slaves for further optimization Running using reference dataset on Ranger now (160 cores)

12Managed by UT-Battelle for the U.S. Department of Energy Future Sinergia work Install PGPLOT Perl module and helper scripts Testing with reference dataset – Number of individuals – Number of clones – Optimize mutation and crossover constants Scale to thousands of cores Automatically switch between genetic algorithms and least squares using history of chisq Write slave code in C++, not Perl Communicate between modules with memory, not files

13Managed by UT-Battelle for the U.S. Department of Energy Fitting Service Fast data fitting, the major and the most time consuming procedure of the data analysis, is the key of high performance data analysis and real-time data processing. Uses NL2SOL or Dakota to fit experimental data – Instrument scientists writes functions – Read data – Model to fit data –

14Managed by UT-Battelle for the U.S. Department of Energy Fitting Service results Fitting service used for experimental reflectometer data is shown NL2SOL results show 5 times smaller Chisq than previous fitting code used for this data

15Managed by UT-Battelle for the U.S. Department of Energy Fitting Service uses TeraGrid New fitting service – going into production – available from the development portal – will be “monitored” for health in the future Calculation scales linearly on TeraGrid on Mercury at NCSA See TG09 slides by Meili Chen for more details.

16Managed by UT-Battelle for the U.S. Department of Energy Fitting Service GUI in Portal

17Managed by UT-Battelle for the U.S. Department of Energy NSTG Cluster Transition to SNS Cluster: dual processor 3.06 GHz Intel Xeon nodes 14 nodes with 2.5GB of memory are compute nodes 4 nodes with 4GB memory are dedicated to GridFTP Transitioning to Scientific Linux 5.2 Move planned from ORNL to SNS machine room (~ 3 miles) Cluster will link SNS with TeraGrid