1. Algorithms and Infrastructure for Molecular Dynamics Simulations Ananth Grama Purdue University http://www.cs.purdue.edu/people/ayg ayg@cs.purdue.edu Various parts of this work are in collaboration with Eric Jakobsson, Vipin Kumar, Sagar Pandit, Ahmed Sameh, Vivek Sarin, H. Larry Scott, Shankar Subramaniam, Priya Vashishtha. Supported by NSF, NIH, and DoE.

2. Statistical and continuum methods ps ns ss ms nm mm mm Ab-initio methods Atomistic methods The Regime of Simulations

3. Multiscale Modeling Techniques  Ab-initio methods (few approximations, but slow)  DFT, CPMD  Electron and nuclei treated explicitly.  Classical atomistic methods (more approximations)  Classical molecular dynamics  Monte Carlo  Brownian dynamics  No electronic degrees of freedom. Electrons are approximated through fixed partial charges on atoms.  Continuum methods (No atomistic details)

4. Initialize positions, velocities and potential parameters At each time step Compute energy and forces Non bonded forces, bonded forces, external forces, pressure tensor Update configuration Compute new configuration, apply constraints and correct velocities, scale simulation cell Molecular Dynamics

5. V = V bond + V angle + V dihedral + V LJ + V Elecrostatics Simplified Interactions Used in Classical Simulations

6. Force Parameters Determination of partial charges on the atoms. Get bond and angle parameters from spectroscopic data. Work on smaller fragments of the molecule and determine LJ parameters by fitting the density and heat of vaporization. For example, linear chain alkanes are used to determine the LJ parameters of hydrocarbon chains in lipids. Determination of dihedral parameters. Tune parameters to match known experimental observables. -0.84 0.42

7. Lennard-Jones (LJ) potential short range potential can be truncated with a cutoff r c (errors, dynamic neighbor list) Electrostatics long range potential cutoffs usually produce erroneous results Particle Mesh Ewald (PME) technique with appropriate boundary corrections (poor scalability) Fast Multipole Methods. Non Bonded Forces

8. Handling Long-Range Potential ● Multipole Methods (FMM/Barnes-Hut) ● Error Control in Multipole Methods ● Parallelism in Multipole Methods ● Applications in Dense System Solvers ● Applications in Sparse Solvers ● Application Studies: Lipid Bilayers ● Ongoing Work: Reactive Simulations

9. Multipole Methods ● Represent the domain hierarchically using a spatial tree. ● Accurate estimation of potentials requires O(n 2 ) force computations. ● Hierarchical methods reduce this complexity to O(n) or O(n log n). ● Since particle distributions can be very irregular, the tree can be highly unbalanced.

10. Multipole Methods

11. Error Control in Multipole Methods

12. Grama et al., SIAM SISC.

13. Error Control in Multipole Methods Grama et al., SIAM SISC.

14. Error Control in Multipole Methods Grama et al., SIAM SISC.

15. Error Control in Multipole Methods Grama et al., SIAM SISC.

16. Error Control in Multipole Methods Grama et al., SIAM SISC.

17. Error Control in Multipole Methods Grama et al., SIAM SISC.

18. Error Control in Multipole Methods Grama et al., SIAM SISC.

19. Parallelism in Multipole Methods ● Each particle can be processed independently ● Nearby particles interact with nearly identical set of sub-domains ● Spatially proximate ordering minimizes communication ● Load can be balanced by assigning each task identical number of interactions Grama et al., Parallel Computing.

20. Parallelism in Multipole Methods Grama et al., Parallel Computing.

21. Parallelism in Multipole Methods Grama et al., Parallel Computing.

22. Solving Dense Linear Systems

25. ● Over a discretized boundary, these lead to a dense linear system of the form Ax = b. ● Iterative methods (GMRES) are typically used to solve the system. ● The matrix-vector product can be evaluated as a single iteration of the multipole kernel – near field is replaced using Gaussian quadratures and far-field is evaluated using multipoles.

26. Solving Dense Linear Systems

27. Multipole Based Dense Solvers All times in seconds on a 64 processor Cray T3E (24,192 unknowns)

28. Multipole Based Solver Accuracy

29. Preconditioning the Solver

31. Solving Sparse Linear Systems

32. Generalized Stokes Problem ● Navier-Stokes equation ● Incompressibility Condition

33. Generalized Stokes Problem ● Linear system ● B T is the discrete divergence operator and A collects all the velocity terms. ● Navier-Stokes: ● Generalized Stokes Problem:

34. Solenoidal Basis Methods ● Class of projection based methods ● Basis for divergence-free velocity ● Solenoidal basis matrix P: ● Divergence free velocity: ● Modified system =>

35. Solenoidal Basis Method ● Reduced system: ● Reduced system solved by suitable iterative methods such as CG or GMRES. ● Velocity obtained by : ● Pressure obtained by: ;

36. Preconditioning Observations Vorticity-Velocity function formulation:

37. Preconditioning Reduced system approximation: Preconditioner: Low accuracy preconditioning is sufficient

38. Preconditioning ● Preconditioning can be affected by an approximate Laplace solve followed by a Helmholtz solve. ● The Laplace solve can be performed effectively using a Multipole method.

39. Preconditioning Laplace/Poisson Problems ● Given a domain with known internal charges and boundary potential (Dirichlet conditions), estimate potential across the domain.  Compute boundary potential due to internal charges (single multipole application)  Compute residual boundary potential due to charges on the boundary (vector operation).  Compute boundary charges corresponding to residual boundary potential (multipole-based GMRES solve).  Compute potential through entire domain due to boundary and internal charges (single multipole).

40. Numerical Experiments ● 3D driven cavity problem in a cubic domain. ● Marker-and-cell scheme in which domain is discretized uniformly. ● Pressure unknowns are defined at node points and velocities at mid points of edges. ● x, y, and z components of velocities are defined on edges along respective directions.

41. Sample Problem Sizes 92,25695,23232,76832x32x32 10,80011,5204,09616x16x16 1,1761,3445128x8x8 Solenoidal Functions VelocityPressureMesh

42. Preconditioning Performance

43. Preconditioning Performance – Poisson Problem

46. Parallel Performance

47. Comparison to ILU

48. Application Validation: Simulating Lipid Bilayers.

49. Membrane as a barrier Thanks to David Bostick

50. Fluid Mosaic Model

51. What is a lipid bilayer ? Hydrophilic Hydrophobic lipid bilayer Depending on the cross sectional area and the chain length the molecules can form vesicles, micelles, hexagonal phase Fatty acid Phosphate Glycerophospholipid

52. Key Physical Properties of Lipid Bilayers Thickness. increases with the length of the hydrocarbon chains decreases with number of unsaturated bonds in chains decreases with temperature Area per molecule increases with the length of the hydrocarbon chains increases with the number of unsaturated bond in chains increases with temperature depends on the properties of polar region

53. Key Physical Properties of Lipid Bilayers Order parameter b

54. Key Physical Properties of Lipid Bilayers Phase Transition Effect of phase transition o order parameters o thickness o area per lipid o bending and compressibility modulus. o lateral and transverse diffusion coefficients.

55. How real are the simulations ? Area per lipid (Å 2 ) experimentssimulations Lipid ~ 38 * ~ 27  Cholesterol ~ 53 Sphingomyelin (18:0) ~ 54~ 53.6 Dipalmitoylphosphatidylserine (DPPS with Na + counter ions) ~ 72~ 71 Dioleylphosphatidylcholine (DOPC) ~ 64~ 62 Dipalmitoylphosphatidylcholine (DPPC)

56. How real are the simulations ? Pandit, et al., Biophys. J. Simulation of 128 DPPC bilayer in SPC water

57. Predictability of atomistic simulations Pandit, et al., J. Chem. Phys. Water shells are observed using the surface to point correlation function.

58. Heterogeneous Mixures in bilayers Lipids with other membrane components. Lipid mixtures with different headgroups and polar region. Lipid mixtures with different chain lengths and saturation. Mixture with cholesterol a - faceb - face Hydrophilic Cholesterol increases mechanical strength of the bilayer Ordering of chains and phase behavior.

59. Phase Diagram with Cholesterol Reproduced from Vist and Davis (1990), Biochemistry, 29:451 L o or b phase Liquid ordered phase where chain ordering is like gel phase but the lateral diffusion coefficients are like fluid phase.

60. Rafts Rafts are liquid ordered domains in cell plasma membranes rich in sphingomyelin (or other sphingolipids), cholesterol and certain anchor proteins. Rafts are important membrane structural components in: signal transduction protein transport sorting membrane components process of fusion of bacteria and viruses into host cells How do rafts function and maintain their integrity? How does Cholesterol control the fluidity of lipid membranes?

61. A Model for Rafts

62. Simulation Studies of Rafts Study of liquid ordered phase Study of ternary mixtures that form raft like domains. Mixture of cholesterol and saturated lipid (DPPC and DLPC). Mixture of cholesterol and sphingomyelin. Preformed domain Spontaneous domain formation Ternary mixture of DOPC, SM, Chol with a preformed domain. Ternary mixture of DOPC, SM, Chol with random initial distribution

63. simulated systems: 120 PC molecules (DPPC/DLPC) 80 cholesterol molecules (~40 mol %) 5000 water molecules duration: 35 ns / 18 ns conditions: NPT ensemble (P = 1 atm, T = 323 / 279 => ) Study of L o phase Pandit, et al., Biophys. J. (2004), 86, 1345-1356.

64. DPPC+CholesterolDLPC+Cholesterol Pandit, et al., J. (2004), 86, 1345-1356.

65. Pandit, et al., Biophys. J. Conclusions from Simulations  Cholesterol increases order parameter of the lipids. The order parameters are close to liquid ordered phase order parameters.  Cholesterol forms complexes with two of its neighboring lipids. Life time of such a complex is ~10ns.  Complex formation process is dependent on the chain length of the lipids. Shorter chain lipids do not form large complexes.

66. Simulation of Domain Formation 1424 DOPC, 266 SM, 122 cholesterols and 62561 waters (284,633 atoms) Duration: 12 ns (with electrostatic cutoff) and 8 ns (with PME), 3 fs steps Ensemble: NPT (P = 1 atm (isotropic), T = 293 K) Configuration Biased Monte Carlo (CBMC) is used to thermalise chains. Pandit et al., Biophys. J.

67. Binary mixture: 100 DOPC, 100 18:0 SM, 7000 water (31600 atoms) 0 ns250 ns

68. Ternary mixture: 100 DOPC, 100 18:0 SM, 100 CHOL, 10000 water (43500 atoms) 0 ns250 ns

69. Some Ongoing Work

70. Reactive systems ● Chemical reactions are association and dissociation of chemical bonds ● Classical simulations cannot simulate reactions ● ab-initio methods calculate overlap of electron orbitals to investigate chemical reactions ● ReaX force field postulates a classical bond order interaction to mimic the association and dissociation of chemical bonds 1 1 van Duin et al, J. Phys. Chem. A, 105, 9396 (2001)

71. Bond Order Interaction 1 van Duin et al, J. Phys. Chem. A, 105, 9396 (2001) Bond order for C-C bond Uncorrected bond order: Where  is for  and  bonds  The total uncorrected bond order is sum of three types of bonds Bond order requires correction to account for the correct valency

72. After correction, the bond order between a pair of atoms depends on the uncorrected bond orders of the neighbors of each atoms The bond orders rapidly decay to zero as a function of distance, so one can easily construct a neighbor list for efficient computation of bond orders Bond Order Interaction

73. Neighbor Lists for Bond Order Efficient implementation critical for performance Implementation based on an oct-tree decomposition of the domain For each particle, we traverse down to neighboring octs and collect neighboring atoms Has implications for parallelism (issues identical to parallelizing multipole methods)

74. Bond Order : Choline

75. Bond Order : Benzene

76. Other Local Energy Terms ● Other interaction terms common to classical simulations, e.g., bond energy, valence angle and torsion, are appropriately modified and contribute to non-zero bond order pairs of atoms ● These terms also become many body interactions as bond order itself depends on the neighbors and neighbor’s neighbors ● Due to variable bond structure there are other interaction terms, such as over/under coordination energy, lone pair interaction, 3 and 4 body conjugation, and three body penalty energy

77. Non Bonded van der Waals Interaction van der Waals interactions are modeled using distance corrected Morse potential Where R(r ij ) is the shielded distance given by

78. Electrostatics Shielded electrostatic interaction is used to account for orbital overlap of electrons at closer distances Long range electrostatics interactions are handled using the Fast Multipole Method (FMM).

79. Charge Equilibration (QEq) Method ● The fixed partial charge model used in classical simulations is inadequate for reacting systems. ● One must compute the partial charges on atoms at each time step using an ab-initio method. ● We compute the partial charges on atoms at each time step using a simplified approach call the Qeq method.

80. Charge Equilibration (QEq) Method Expand electrostatic energy as a Taylor series in charge around neutral charge. Identify the term linear in charge as electronegativity of the atom and the quadratic term as electrostatic potential and self energy. Using these, solve for self-term of partial derivative of electrostatic energy.

81. Qeq Method We need to minimize: subject to: where

82. Qeq Method

83. From charge neutrality, we get:

84. Qeq Method Let where or

85. Qeq Method ● Substituting back, we get: We need to solve 2n equations with kernel H for s i and t i.

86. Qeq Method ● Observations: –H is dense.  The diagonal term is J i  The shielding term is short-range  Long range behavior of the kernel is 1/r Hierarchical methods to the rescue! Multipole- accelerated matrix-vector products combined with GMRES and a preconditioner.

87. Qeq: Parallel Implementation ● Key element is the parallel matrix-vector (multipole) operation ● Spatial decomposition using space-filling curves ● Domain is generally regular since domains are typically dense ● Data addressing handled by function shipping ● Preconditioning via truncated kernel ● GMRES never got to restart of 10!

88. Qeq Parallel Performance 27 (26.5)187 (3.8)7189255,391 13 (24.3)87 (3.6)3169108,092 4.95 (21.0)32 (3.3)104743,051 P=32P=4P=1IterationsSize Size corresponds to number of atoms; all times in seconds, speedups in parentheses. All runtimes on a cluster of Pentium Xeons connected over a Gb Ethernet.

89. Qeq Parallel Performance 19.1 (26.6)132 (3.8)508255,391 9.2 (24.8)61 (3.7)228108,092 3.2 (22.8)21 (3.5)7343,051 P=16P=4P=1Size Size corresponds to number of atoms; all times in seconds, speedups in parentheses. All runtimes on an IBM P590.

90. Parallel ReaX Performance ● ReaX potentials are near-field. ● Primary parallel overhead is in multipole operations. ● Excellent performance obtained over rest of the code. ● Comprehensive integration and resulting (integrated) speedups being evaluated.

91. ... and yet to come... ● Comprehensive validation of parallel ReaX ● System validation of code – from simple systems (small hydrocarbons) to complex molecules (larger proteins) ● Parametrization and tuning force fields.

92. Parting Thoughts ● There is a great deal computer scientists can contribute to engineering and sciences. ● In our group, we work on projects ranging from: structural control (model reduction, control, sensing, macroprogramming -- NSF/ITR), materials modeling (multiscale simulations – NSF, DoE), systems biology (interaction networks – NIH), and finite element solvers (sparse solvers – NSF).

93. Parting Thoughts ● There is a great deal we can learn from scientific and engineering disciplines as well. ● Our experiences in these applications motivate our work in core areas of computing, including wide-area storage systems (NSF/STI), speculative execution (Intel), relaxed synchronization (NSF), and scaling to the extreme (DARPA/HPCS).

94. Thank you very much! References and papers at: http://www.cs.purdue.edu/pdsl/