Parallel Methods for Nano/Materials Science Applications Andrew Canning Computational Research Division LBNL & UC Davis (Electronic Structure Calculations)

Outline Introduction to Nano/Materials science Electronic Structure Calculations (DFT) Code performance on High Performance Parallel Computers New Methods and Applications for Nanoscience

1991 Silicon surface reconstruction (7x7), Phys. Rev. ( Stich, Payne, King-Smith, Lin, Clarke) Meiko I860, 64 processor Computing Surface ( Brommer, Needels, Larson, Joannopoulos) Thinking Machines CM2, 16,384 bit processors atom Molybdenum simulation with Qbox SC05. ( F. Gygi et al. ) BlueGene/L, 32,000 processors (LLNL) 1998 FeMn alloys (exchange bias), Gordon Bell prize ( Ujfalussy, Stocks, Canning, Y. Wang, Shelton et al.) Cray T3E, 1500 procs. first > 1 Tflop Simulation Milestones in Parallel Calculations

Electronic Structure Calculations Accurate Quantum Mechanical treatment for the electrons Each electron represented on grid or with some basis functions (eg. fourier components) Compute Intensive: Each electron requires 1 million points/basis (need 100s of electrons) 70-80% NERSC Materials Science Computer Time (first-principles electronic structure) InP quantum dot (highest electron energy state in valence band)

Motivation for Electronic Structure Calculations Most Materials Properties Only Understood at a fundamental level from Accurate Electronic Structure (Strength, Cohesion etc) Many Properties Purely Electronic eg. Optical Properties (Lasers) Complements Experiments Computer Design Materials at the nanoscale

Materials Science Methods Many Body Quantum Mechanical Approach (Quantum Monte Carlo) atoms Single Particle QM (Density Functional Theory) No free parameters atoms Empirical QM Models eg. Tight Binding atoms Empirical Classical Potential Methods thousand-million atoms Continuum Methods atoms

Ab initio Method: Density Functional Theory (Kohn 98 Nobel Prize) Kohn Sham Equation (65): The many body ground state problem can be mapped onto a single particle problem with the same electron density and a different effective potential (cubic scaling). Use Local Density Approximation (LDA) for (good Si,C) Many Body Schrodinger Equation (exact but exponential scaling )

Selfconsistent calculation Selfconsistency N electrons N wave functions lowest N eigenfunctions

Choice of Basis for DFT(LDA) Increasing basis size M GaussianFLAPWFouriergrid Percentage of eigenpairs M/N 30%2% Eigensolvers Direct (scalapack) Iterative

Plane-wave Pseudopotential Method in DFT Solve Kohn-Sham Equations self-consistently for electron wavefunctions within the Local Density Appoximation 1. Plane-wave expansion for 2. Replace “frozen” core by a pseudopotential Different parts of the Hamiltonian calculated in different spaces (fourier and real) 3d FFT used

PARATEC (PARAllel Total Energy Code) PARATEC performs first-principles quantum mechanical total energy calculation using pseudopotentials & plane wave basis set Written in F90 and MPI Designed to run on large parallel machines IBM SP etc. but also runs on PCs PARATEC uses all-band CG approach to obtain wavefunctions of electrons Generally obtains high percentage of peak on different platforms Developed with Louie and Cohen’s groups (UCB, LBNL), Raczkowski

PARATEC: Code Details Code written in F90 and MPI (~50,000 lines) 33% 3D FFT, 33% BLAS3, 33% Hand coded F90 Global Communications in 3D FFT (Transpose) Parallel 3D FFT handwritten, minimize comms. reduce latency (written on top of vendor supplied 1D complex FFT )

–Load Balance Sphere by giving columns to different procs. –3D FFT done via 3 sets of 1D FFTs and 2 transposes –Most communication in global transpose (b) to (c) little communication (d) to (e) –Flops/Comms ~ logN –Many FFTs done at the same time to avoid latency issues –Only non-zero elements communicated/calculated –Much faster than vendor supplied 3D-FFT PARATEC: Parallel Data distribution and 3D FFT (a)(b) (e) (c) (f) (d)

PARATEC: Performance  All architectures generally achieve high performance due to computational intensity of code (BLAS3, FFT)  ES achieves highest overall performance to date: 5.5Tflop/s on 2048 procs  Main ES advantage for this code is fast interconnect  SX8 achieves highest per-processor performance  X1 shows lowest % of peak  Non-vectorizable code much more expensive on X1  IBM Power5 4.8 Gflops/P (63% peak on 64 procs)  BGL got 478 Mflops/P (17% of peak on 512 procs) ProblemP NERSC (Power3) Jacquard (Opteron) Thunder (Itanium2) ORNLCray (X1)NEC ES (SX6 * )NEC SX8 Gflops/P%peakGflops/P%peakGflops/P%peakGflops/P%peakGflops/P%peakGflops/P%peak 488 Atom CdSe Quantum Dot %2.851%3.225%5.164%7.547% %1.9845%2.647%3.024%5.062%6.843% %0.9521%2.444%4.455% %1.832%3.646% Developed with Louie and Cohen’s groups (UCB, LBNL), also work with L. Oliker, J Carter

Self-consistent all band method for metallic systems Previous methods use self- consistent (SC) band by band, with Temperature smearing (eg. VASP code) drawbacks – band-by-band slow on modern computers (cannot use fast BLAS3 matrix-matrix routines) New Method uses occupancy in inner iterative loop with all band Grassman method (GMCG method) Al (100) surface, 10 layers + vacuum GMCG: new method with occupancy

Self-consistent all band method for metals Potential Mixing V out  V in KS - DFT

The Quantization Condition of Quantum-well States in Cu/Co(100) Copper substrate Cobalt layer Copper Wedge 0 54 Å 4 mm scan d photon beam in electrons out Theoretical investigation of Quantum Well states in Cu films using our codes (PARATEC, PEtot) to compare with experiments at the ALS (E. Rotenberg, Y.Z. Wu, Z.Q. Qiu) New computational methods for metallic systems used in the calculations. Lead to an understanding of surface effects on the Quantum Well States. Improves on simple Phase Accumulation Model used previously Difference between theory and experiment improved by taking surface effects into account QW states in Copper Wedge

Computational challenges (larger nanostructures) Ab initio method PARATEC atoms molecules nanostructures bulk size atoms ^6 atoms Infinite (1-10 atoms in a unit cell) method Ab initio Method PARATEC Challenge for computational nanoscience. Ab initio elements and reliability New methodology and algorithm (ESCAN) Even larger Supercomputer

Example: Quantum Dots (QD) CdSe Band gap increase CdSe quantum dot (size) Single electron effects on transport (Coulomb blockade). Mechanical properties, surface effects and no dislocations

Charge patching method for larger systems(Wang) Selfconsistent LDA calculation of a single graphite sheet Non-selfconsistent LDA quality potential for nanotube Get information from small system ab initio calc., then generate the charge densities for large systems

Motif based charge patching method (Wang) Error: 1%, ~20 meV eigen energy error.

+ Folded Spectrum Method (ESCAN) N

Charge patching: free standing quantum dots In 675 P 652 LDA quality calculations (eigen energy error ~ 20 meV) L-W Wang CBMVBM 64 processors (IBM SP3) for ~ 1 hour Total charge density motifs

Nanowire Single Electron Memory Samuelson group Lund, Sweden Nano Letters Vol2, 2, 2002.

Nanowire Single Electron Memory (LOBPCG) Comparison of LOBPCG with band by band CG (64 procs on IBM SP) Matrix Size = 2,265,837 (nano-wire InP InAs with 67,000 atoms) Using code to determine size regimes in which single electron behavior occurs (~60nm length, ~20nm diameter), also using LCBB code for larger systems. Work carried out with G. Bester S. Tomov, J. Langou

Future Directions O(N) based methods (exploit locality) gives sparse matrix problem Excited state calculations Transport Calculations

Multi-Teraflops Spin Dynamics Studies of the Magnetic Structure of FeMn and FeMn/Co Interfaces Section of an FeMn/Co (Iron Manganese/ Cobalt) interface showing the final configuration of the magnetic moments for five layers at the interface. Shows a new magnetic structure which is different from the 3Q magnetic structure of pure FeMn. Exchange bias, which involves the use of an antiferromagnetic (AFM) layer such as FeMn to pin the orientation of the magnetic moment of a proximate ferromagnetic (FM) layer such as Co, is of fundamental importance in magnetic multilayer storage and read head devices. A larger simulation of 4000 atoms of FeMn ran at 4.42 Teraflops 4000 processors. (ORNL, Univ. of Tennessee, LBNL(NERSC) and PSC) IPDPS03 A. Canning, B. Ujfalussy, T.C. Shulthess, X.-G. Zhang, W.A. Shelton, D.M.C. Nicholson, G.M. Stocks, Y. Wang, T. Dirks Contact: Andrew Canning

Conclusion First principle calculation New algorithm methodology Large scale supercomputers Accurate Nanostructures simulations + +