1. University of Edinburgh January 2011 DL_POLY_4 Ilian Todorov @ CCG CSED, STFC - Daresbury Laboratory, Daresbury Warrington WA4 1EP, Cheshire, England, UK

2. University of Edinburgh January 2011 Where is Daresbury?

3. University of Edinburgh January 2011 DL_POLY Project Background General purpose parallel (classical) MD simulation software It was conceived to meet the needs of CCP5 - The Computer Simulation of Condensed Phases (academic collaboration community) Written in modularised Fortran90 (NagWare & FORCHECK compliant) with MPI2 (MPI1+MPI-I/O) fully self-contained 1994 – 2011: DL_POLY_2 (RD) by W. Smith & T.R. Forester (funded for 6 years by EPSRC at DL) -> DL_POLY_CLASSIC 2003 – 2011: DL_POLY_3 (DD) by I.T. Todorov & W. Smith (funded for 4 years by NERC at Cambridge) -> DL_POLY_4 Over 13,000 licences taken out since 1994 (~1500 annually) Over 1100 registered FORUM members since 2005 Available free of charge (under licence) to University researchers (provided as code) and at cost to industry

4. University of Edinburgh January 2011 DL_POLY_DD Development Statistics

5. University of Edinburgh January 2011 DL_POLY_DD Licence Statistics

6. University of Edinburgh January 2011 DL_POLY Licence Statistics

7. University of Edinburgh January 2011 DL_POLY Licence Statistics

8. University of Edinburgh January 2011 DL_POLY Licence Statistics

9. University of Edinburgh January 2011 DL_POLY Licence Statistics

10. University of Edinburgh January 2011 DL_POLY Project Current State October 2010: DL_POLY_3 -> DL_POLY_4 still under STFC Licence, over 1600 licences taken out since November 2010 Rigid Body dynamics (ITT) Parallel I/O & netCDF I/O – NAG dCSE (IJB & ITT) CUDA+OpenMP port (as source, ICHEC) & MS Windows ports (as MSI 32- and 64-bit self-installers) SPME processor grid freed from 2^N decomposition – NAG dCSE (IJB) January 2011: DL_POLY_2 -> DL_POLY_CLASSIC on a BSD type Licence (BS retired but supporting GUI and fixes) Load Balancer development (finished 30/03/2011) Continuous Development of the DL_FIELD field builder (pdb to DL_POLY input, Chin Yong), over 400 licences since November 2010 October 2011: new DL_POLY GUI (Bill Smith)

11. University of Edinburgh January 2011 DL_POLY Project Roadmap August 2011 (PRACE-2IP funding): further CUDA porting & support by ICHEC (2 PM) CUDA to OpenCL port by SC@WUT (5 PM) FMM library testing as substitute of SPME electrostatics October 2011 (2 FTE years Software Development EPSRC funding with Kostya Trachenko) Two temperature thermostat methodology (TTM) for high energy events in metals (Michael Seaton) Fragmented I/O and on-the-fly thermodynamic properties calculations for very large systems (Michael Seaton) November 2011 (1 FTE year dCSE funding with David Quigley): OpenMP within the vanilla DD/MPI framework DL_POLY_4 (Ian Bush & Asimina Maniopoulou) Beyond 2.1 billion particles limit – long integers (Ian Bush)

12. University of Edinburgh January 2011 DL_POLY Project Roadmap II December 2011 (MMM@HPC - FP7-eInfrastructure funding, 3 FTE years) Gay-Berne particles (Laurence Ellison) Implementing Gentle Stochastic Thermostat Extension of NsT ensemble to handle liquid bio-chemical systems nfold and bio-chemical verification Modified Tersoff potential for band-gap materials Porting the DL_POLY_Classic Hyper-dynamics

13. University of Edinburgh January 2011 DL_POLY_4 (version 4.02) –Dynamic Decomposition parallelisation, based on domain decomposition but with dynamic load balancing –limits up to ≈2.1×10 9 atoms with inherent parallelisation. –Full force field and molecular description with rigid body description –Free format (flexible) reading with some fail-safe features and basic reporting (but fully fool-proofed) DL_POLY Classic (version 1.8) –Replicated Data parallelisation, limits up to ≈30,000 atoms with good parallelisation up to 64 (system dependent) processors (running on any processor count) –Full force field and molecular description –Hyper-dynamics: Temperature Accelerated Dynamics & Biased Potential Dynamics, Solvation Dynamics – Spectral Shifts, Metadynamics, (Path Integral MD) –Free format reading but somewhat strict Current Versions

14. University of Edinburgh January 2011 Supported Molecular Entities Point ions and atoms Polarisable ions (core+ shell) Flexible molecules Rigid bonds Rigid molecules Flexibly linked rigid molecules Rigid bond linked rigid molecules

15. University of Edinburgh January 2011 Force Field Definitions – I particle: rigid ion or atom (charged or not), a core or a shell of a polarisable ion(with or without associated degrees of freedom), a massless charged site. A particle is a countable object and has a global ID index. site: a particle prototype that serves to defines the chemical & physical nature (topology/connectivity/stoichiometry) of a particle (mass, charge, frozen-ness). Sites are not atoms they are prototypes! Intra-molecular interactions: chemical bonds, bond angles, dihedral angles, improper dihedral angles, inversions. Usually, the members in a unit do not interact via an inter-molecular term. However, this can be overridden for some interactions. These are defined by site. Inter-molecular interactions: van der Waals, metal (EAM, Gupta, Finnis-Sinclair, Sutton-Chen), Tersoff, three-body, four-body. Defined by species.

16. University of Edinburgh January 2011 Force Field Definitions – II Electrostatics: Standard Ewald*, Hautman-Klein (2D) Ewald*, SPM Ewald (3D FFTs), Force-Shifted Coulomb, Reaction Field, Fennell damped FSC+RF, Distance dependent dielectric constant, Fuchs correction for non charge neutral MD cells. Ion polarisation via Dynamic (Adiabatic) or Relaxed shell model. External fields: Electric, Magnetic, Gravitational,Oscillating & Continuous Shear, Containing Sphere, Repulsive Wall. Intra-molecular like interactions: tethers, core shells units, constraint and PMF units, rigid body units. These are also defined by site. Potentials: parameterised analytical forms defining the interactions. These are always spherically symmetric! THE CHEMICAL NATURE OF PARTICLES DOES NOT CHANGE IN SPACE AND TIME!!!

17. University of Edinburgh January 2011 Force Field by Sums

18. University of Edinburgh January 2011 Ensembles and Algorithms Integration: Available as velocity Verlet (VV) or leapfrog Verlet (LFV) generating flavours of the following ensembles NVE NVT (E kin ) Evans NVT Andersen^, Langevin^, Berendsen, Nosé-Hoover NPT Langevin^, Berendsen, Nosé-Hoover, Martyna- Tuckerman-Klein^ N  T/NPnAT/NPn  T Langevin^, Berendsen, Nosé- Hoover, Martyna-Tuckerman-Klein^ Constraints & Rigid Body Solvers: VV dependent – RATTLE, No_Squish, QSHAKE* LFV dependent – SHAKE, Euler-Quaternion, QSHAKE*

19. University of Edinburgh January 2011 19 A B C D Domain Decomposition

20. University of Edinburgh January 2011 Global force field P 0 Local atomicindices P 1 Local atomicindices P 2 Local atomicindices Processor Domains Tricky! Molecular force field definition Bonded Forces within DD

21. University of Edinburgh January 2011 U. Essmann, L. Perera, M.L. Berkowtz, T. Darden, H. Lee, L.G. Pedersen, J. Chem. Phys., 103, 8577 (1995) 1.Calculate self interaction correction 2.Initialise FFT routine (FFT – IJB’s DaFT: 3M 2 1D FFT) 3.Calculate B-spline coefficients 4.Convert atomic coordinates to scaled fractional units 5.Construct B-splines 6.Construct partial charge array Q 7.Calculate FFT of Q array 8.Construct partial array G 9.Calculate FFT of G array 10.Calculate net Coulombic energy 11.Calculate atomic forces I.J. Bush, I.T. Todorov, W. Smith, Comp. Phys. Commun., 175, 323 (2006) DD Scheme for long-ranged part of SPME

22. University of Edinburgh January 2011 Performance Weak Scaling on IBM p575 2005-2011

23. University of Edinburgh January 2011 Rigid Bodies versus Constraints 450,000 particles with DL_POLY_4

24. University of Edinburgh January 2011 I/O Solutions in DL_POLY_4 1. Serial read and write (sorted/unsorted) – where only a single MPI task, the master, handles it all and all the rest communicate in turn to or get broadcasted to while the master completes writing a configuration of the time evolution. 2. Parallel write via direct access or MPI-I/O (sorted/unsorted) – where ALL / SOME MPI tasks print in the same file in some orderly manner so (no overlapping occurs using Fortran direct access printing. However, it should be noted that the behaviour of this method is not defined by the Fortran standard, and in particular we have experienced problems when disk cache is not coherent with the memory). 3. Parallel read via MPI-I/O or Fortran 4. Serial NetCDF read and write using NetCDF libraries for machine-independent data formats of array-based, scientific data (widely used by various scientific communities).

25. University of Edinburgh January 2011 MPI-I/O Write Performance for 216,000 Ions of NaCl on XT5

26. University of Edinburgh January 2011 MPI-I/O Read Performance for 216,000 Ions of NaCl on XT5

27. University of Edinburgh January 2011 xyz, PDB DL_FIELD ‘black box’ FIELD CONFIG DL_FILED AMBER & CHARM to DL_POLY OPLSAA & Drieding to DL_POLY Protonated

28. University of Edinburgh January 2011 MgO, NaCl, simple ionic crystals - easy Force fields for Two vdw descriptions: Mg 2+ with O 2- and O 2- with O 2- Materials Force Filed

29. University of Edinburgh January 2011 Simple covalent molecules - bearable For example: dimethyl sulphoxide atoms 10 HD 1.00797 0.09000 1 CD 12.01115 -0.14800 1 HD 1.00797 0.09000 1 CD 12.01115 -0.14800 1 SD 32.06400 0.31200 1 OD 15.99940 -0.55600 1 bonds 21 harm 4 1 644 1.11000 harm 4 2 644 1.11000 harm 4 3 644 1.11000 harm 8 5 644 1.11000 harm 8 6 644 1.11000 harm 8 7 644 1.11000 harm 4 9 480 1.80000 harm 8 9 480 1.80000 harm 9 10 1080 1.53000 -126 1 8 96195 129.03 -126 1 10 33433 84.716 angles 15 harm 1 4 2 71.00000 108.40000 harm 1 4 3 71.00000 108.40000 harm 1 4 9 92.20000 111.30000 harm 2 4 3 71.00000 108.40000 harm 2 4 9 92.20000 111.30000 harm 3 4 9 92.20000 111.30000 harm 5 8 6 71.00000 108.40000 harm 5 8 7 71.00000 108.40000 harm 5 8 9 92.20000 111.30000 harm 6 8 7 71.00000 108.40000 harm 6 8 9 92.20000 111.30000 harm 7 8 9 92.20000 111.30000 harm 4 9 8 68.00000 95.00000 harm 4 9 10 158.00000 106.75000 harm 8 9 10 158.00000 106.75000 Also 12 dihedrals, and 10 vdw Biological Force Field

30. University of Edinburgh January 2011 How about this? Protein molecules consist of hundreds of amino acids. 4382 atoms 19400 two-body 7993 three-body 13000 four body 730 vdw Clearly, manual entry is not practical! Biological Force Field

31. University of Edinburgh January 2011 Two Temperature Model Superheating? Lattice temperature above the melting temperature with no track formation 1. Continuum models cannot give accurate track radii; require MD-coupling! 1. D.M. Duffy, N. Itoh, A.M. Rutherford and A.M. Stoneham, J. Phys: Condens. Matt. 20 (2008), p. 082201 Continuum neglects lattice straining and emission of shock waves (carry away some energy). C L in continuum is neither identical with C v or C p. Volume change by a phase transition is not included, i.e. Silicon shrinks upon melting TTM+MD 50x50x10 unit Si cells 50keV/nm ~1ps

32. University of Edinburgh January 2011 MD additions: Coarse-grained T-cells Choice of boundary conditions… Periodic for the atomistic lattice Electronic system: z-dir von-Neumann, xy-dir infinite sink Inhomogeneous Langevin thermostat Mechanism for electronic energy transfer to the lattice Depends on the local electronic temperature Molecular dynamics Continuum or Towards Molecular Dynamics

33. University of Edinburgh January 2011 DL_POLY_4 Calling Tree Always Advise with the MANUAL Start up Parse INPUT data Set Limits & Array Bounds Allocate Arrays Read INPUT data, VERIFY & Make SENSE (semi-iterative) Initialise/Read Restart data Set Halo Decomposition Bookkeeping, VERIFY & Report Set Temperature/Motion (CS+CB+RB+CGM) Main MD Loop (Time & Steps Conditioned) Finalise & Report Statistics Shut Down

34. University of Edinburgh January 2011 DL_POLY_4 Calling Tree Main MD Loop (Time & Steps Conditioned) LFV Evaluate Inter-molecular Forces & Collect Spatial Statistics (VNL) Make Adjustments & Evaluate Intra-molecular Forces Leap Frog Verlet RELOCATION & BOOKKEEPING & Set Halo Collect Statistics Print OUTPUT data Check on Loop Condition

35. University of Edinburgh January 2011 DL_POLY_4 Calling Tree Main MD Loop (Time & Steps Conditioned) VV Velocity Verlet stage I Evaluate Inter-molecular Forces & Collect Spatial Statistics (VNL) Make Adjustments & Evaluate Intra-molecular Forces RELOCATION & BOOKKEEPING & Set Halo Velocity Verlet stage II Collect Statistics Print OUTPUT data Check on Loop Condition

36. University of Edinburgh January 2011 DL_POLY on the Web WWW: http://www.ccp5.ac.uk/DL_POLY/ http://www.ccp5.ac.uk/DL_POLY/ FTP: ftp://ftp.dl.ac.uk/ccp5/DL_POLY/ftp://ftp.dl.ac.uk/ccp5/DL_POLY/ DEV: http://ccpforge.cse.rl.ac.uk/gf/project/dl-poly/http://ccpforge.cse.rl.ac.uk/gf/project/dl-poly/ FORUM: http://www.cse.scitech.ac.uk/disco/forums.shtmlhttp://www.cse.scitech.ac.uk/disco/forums.shtml

37. University of Edinburgh January 2011 Acknowledgements Thanks to Bill Smith (retired) Chin Yong (STFC – DL) Michael Seaton (STFC – DL) Szymon Draszewicz (UCL) Ian Bush (NAG Ltd.)