Quantum Simulation Neural Networks Brian Barch Examples of results of molecular dynamics Problem: Molecular Dynamics Want to simulate motion or distributions of atoms Do this by constructing a potential energy surface (PES) Predicts energy from position Can then derive energy to get forces on atoms Impossible analytically due to electron wavefunctions Traditionally use iterative methods, i.e. density functional theory (DFT) or quantum chemistry methods These have time vs accuracy tradeoff Can augment with neural networks, which can predict accurately very quickly once trained Neural network goal: predict energy from atomic position Snapshot of melting water from NN-based simulation Poliovirus, used in 108 atom simulation of infection Dataset: Atomic hydrogen 8 datasets at different temperatures and pressure ~3000 configurations of hydrogen atoms each Each configuration: x, y, z for 54 atoms and total energy As neural network PES go, this is very high dimensional Preprocessing: Calculated interatomic distances and angles between atoms For baseline model: used these to calculate symmetry function values, which were then normalized and saved For new model, preprocessed only distances and angles Symmetry functions Represent atomic environment in a form invariant under changes that shouldn’t affect energy i.e. rotation, total system translation, index swaps, etc Differentiable but highly nonlinear. Different types of parameters and functions used per sym. func. mean can’t represent as a single vector function (which would make things much easier) Cutoff function ensures they represent only local environment – reduces scaling Baseline model (as used in literature): manually picked, kept as static hyperparameter, and used to preprocess data Atomic Neural Network Structure Baseline NN structure Preprocess Sym. funcs. Separate NNs that share weights Sum of energies Project goal: design and implement trainable symmetry functions i.e. turn hyperparameter to normal parameter To my knowledge, there are no papers even suggesting this Sym. func. types: Effects of parameters of G2 Each atom is described by a vector of sym. funcs Baseline model: Take list of sym. func. vectors as input to NN Individual atomic NN trained on each vector NNs share weights Total energy is sum of outputs of atomic NNs Cutoff: used in others, but not a sym. func. Activation of sym. func. G2 for 2 atoms as a function of distance for different parameter values New convolutional Neural Network model: Batch normalization Sum of atomic energies D ѳ Preprocess Atoms NN angles distances Sym. Func. Layer Conv layers Ê + Changes to model Replaced atomic NNs with 1D convolutional layers, with atom index as dimension This part is functionally the same as the baseline, but more efficient Allows GPU optimization, i.e. CuDNN Different symmetry functions in different channels Takes distances and angles as input and applies a sym. func. layer Batch normalization layer instead of using pre-normalized data Final layer is same as baseline model Trainable symmetry functions Sym. func. layer calculates sym func values with parameters stored as theano tensors Allows more generalizability, since no need to preprocess data Allows backpropagation onto parameters the same as is done for weights Caused NN to train slower, but once good sym. func. parameters are found they can be used for preprocessing to avoid this Results: Loss during training Goal was to make trainable sym. funcs, which succeeded New convolutional NN structure greatly increased train and prediction speed of NN through GPU optimization G1 – G3 could be implemented quickly, but G4 and G5 use angles, so require a triple sum over all atoms Causes massive slowdown in feed forward, and even more during backpropagation Baselines results relied heavily on G4 and G5, which are the most complex sym. funcs, so could not beat my best results with the baseline model I did beat the results of the model used by researchers from LBL on this problem though MSE (eV) Future directions Will focus on increasing efficiency of complex sym. func. implementation and training I plan to represent sym. funcs not as a single complex layer, but as a combination of multiple simple add, multiple, exponent, etc layers I will also focus on making sure NNs are learning useful sym. func. parameters by penalizing redundancies and optimizing training with an evolutionary neural network structure I made previously (based on MPANN) Loss during training for a NN with static sym. func. parameters (left) and with trainable sym. func. parameters (right). Training MSE (red) and validation MSE (blue) are measured in electronvolts. Used 8 each of G1, G2, and G3 for this. When compared to the same untrained sym funcs, trained ones always performed better Decreased MSE, faster training and less overfitting ~10% MSE decrease for best case, but more with worse hyperparameters Trained sym. fun. Parameters were unpredictable – sometimes stayed near initialization, other times multiple converged to same value or crossed over. Unsure what to make of this