MULTISCALE COMPUTATION: From Fast Solvers To Systematic Upscaling A. Brandt The Weizmann Institute of Science UCLA

Major scaling bottlenecks: computing Elementary particles (QCD) Schrödinger equation molecules condensed matter Molecular dynamics protein folding, fluids, materials Turbulence, weather, combustion,… Inverse problems da, control, medical imaging Vision, recognition

Scale-born obstacles: Many variables n gridpoints / particles / pixels / … Interacting with each other O(n 2 ) Slowness Slow Monte Carlo / Small time steps / … Slowly converging iterations / due to 1.Localness of processing

0 r0r0 distance Two-atom Lennard-Jones potential r0r0 small step

small step Moving one particle at a time fast local ordering slow global move

Solving PDE : Influence of pointwise relaxation on the error Error of initial guess Error after 5 relaxation sweeps Error after 10 relaxations Error after 15 relaxations Fast error smoothing slow solution

Scale-born obstacles: Many variables n gridpoints / particles / pixels / … Interacting with each other O(n 2 ) Slowness Slow Monte Carlo / Small time steps / … Slowly converging iterations / due to 1.Localness of processing 2. Attraction basins

Fluids Gas/Liquid 1.Positional clustering Lennard-Jones r0r0 2.Electrostatic clustering Dipoles Water: 1& 2

r E(r) Optimization min E(r) multi-scale attraction basins

Macromolecule      + Lennard-Jones ~10 4 Monte Carlo passes for one T G i transition G1G1 G2G2 T Dihedral potential + Electrostatic

Potential Energy Lennard-Jones Electrostatic Bond length strain Bond angle strain torsion hydrogen bond rkrk E  ijkl riri rjrj rlrl r ij

Scale-born obstacles: Many variables n gridpoints / particles / pixels / … Interacting with each other O(n 2 ) Slowness Slow Monte Carlo / Small time steps / … Slowly converging iterations / due to 1.Localness of processing 2. Attraction basins Removed by multiscale processing

Solving PDE : Influence of pointwise relaxation on the error Error of initial guess Error after 5 relaxation sweeps Error after 10 relaxations Error after 15 relaxations Fast error smoothing slow solution

LU = F h 2h 4h L h U h = F h L 2h U 2h = F 2h L 4h V 4h = R 4h L 2h V 2h = R 2h

interpolation (order l+p) to a new grid interpolation (order m) of corrections relaxation sweeps algebraic error < truncation error residual transfer enough sweeps or direct solver * *** Full MultiGrid (FMG) algorithm... * h0h0 h 0 /2 h 0 /4 2h h

Multigrid solvers Cost: operations per unknown Linear scalar elliptic equation (~1971)

Multigrid solvers Cost: operations per unknown Linear scalar elliptic equation (~1971) Nonlinear FAS (1975)

LU = F h 2h 4h L h U h = F h L 4h U 4h = F 4h Fine-to-coarse defect correction L 2h V 2h = R 2h U 2h = U h,approximate +V 2h L 2h U 2h = F 2h

Multigrid solvers Cost: operations per unknown Linear scalar elliptic equation (~1971)* Nonlinear Grid adaptation General boundaries, BCs* Discontinuous coefficients Disordered: coefficients, grid (FE) AMG Several coupled PDEs* (1980) Non-elliptic: high-Reynolds flow Highly indefinite: waves Many eigenfunctions (N) Near zero modes Gauge topology: Dirac eq. Inverse problems Optimal design Integral equations Statistical mechanics Massive parallel processing *Rigorous quantitative analysis (1986) FAS (1975) Within one solver (1977,1982)

Multigrid solvers Cost: operations per unknown Linear scalar elliptic equation (~1971)* Nonlinear Grid adaptation General boundaries, BCs* Discontinuous coefficients Disordered: coefficients, grid (FE) AMG Several coupled PDEs* (1980) Non-elliptic: high-Reynolds flow Highly indefinite: waves Many eigenfunctions (N) Near zero modes Gauge topology: Dirac eq. Inverse problems Optimal design Integral equations Statistical mechanics Massive parallel processing *Rigorous quantitative analysis (1986) FAS (1975) Within one solver (1977,1982)

Same fast solver Local patches of finer grids Each level correct the equations of the next coarser level Each patch may use different coordinate system and anisotropic grid “Quasicontiuum” method [B., 1992] Each patch may use different coordinate system and anisotropic grid and different physics; e.g. Atomistic and differet physics; e.g. atomistic

Multigrid solvers Cost: operations per unknown Linear scalar elliptic equation (~1971)* Nonlinear Grid adaptation General boundaries, BCs* Discontinuous coefficients Disordered: coefficients, grid (FE) AMG Several coupled PDEs* (1980) Non-elliptic: high-Reynolds flow Highly indefinite: waves Many eigenfunctions (N) Near zero modes Gauge topology: Dirac eq. Inverse problems Optimal design Integral equations Statistical mechanics Massive parallel processing *Rigorous quantitative analysis (1986) FAS (1975) Within one solver (1977,1982)

Multigrid solvers Cost: operations per unknown Linear scalar elliptic equation (~1971)* Nonlinear Grid adaptation General boundaries, BCs* Discontinuous coefficients Disordered: coefficients, grid (FE) AMG Several coupled PDEs* (1980) Non-elliptic: high-Reynolds flow Highly indefinite: waves Many eigenfunctions (N) Near zero modes Gauge topology: Dirac eq. Inverse problems Optimal design Integral equations Statistical mechanics Massive parallel processing *Rigorous quantitative analysis (1986) FAS (1975) Within one solver (1977,1982)

ALGEBRAIC MULTIGRID (AMG) 1982

Classical ALGEBRAIC MULTIGRID (AMG) 1982 Ax = b, "M matrix" a ii a ij a ij = a ji  0 (i = 1,…,n) Relaxation  approximation a 23 a 34 a 35

Classical ALGEBRAIC MULTIGRID (AMG) a 23 a 34 a 35 Ax = b, "M matrix" a ii a ij a ij = a ji  0 (i = 1,…,n) Coarse variables - a subset ( x c ) 3 = ( a 23 x 2 + a 34 x 4 )/( a 23 + a 34 ) AMG Cycle Relaxation  approximation Solve A c x c = T (b - A ) (recursion)  + x c

ALGEBRAIC MULTIGRID (AMG) 1982 Coarse variables - a subset 1. “General” linear systems 2. Variety of graph problems

Graph problems Partition: min cut Clustering bioinformatics Image segmentation VLSI placement Routing Linear arrangement: bandwidth, cutwidth Graph drawing low dimension embedding Coarsening: weighted aggregation Recursion: inherited couplings (like AMG) Modified by properties of coarse aggregates General principle: Multilevel objectives

Multigrid solvers Cost: operations per unknown Linear scalar elliptic equation (~1971)* Nonlinear Grid adaptation General boundaries, BCs* Discontinuous coefficients Disordered: coefficients, grid (FE) AMG Several coupled PDEs* (1980) Non-elliptic: high-Reynolds flow Highly indefinite: waves Many eigenfunctions (N) Near zero modes Gauge topology: Dirac eq. Inverse problems Optimal design Integral equations Statistical mechanics Massive parallel processing *Rigorous quantitative analysis (1986) FAS (1975) Within one solver (1977,1982)

2h h 2  wavelength Non-local components: e i  x,  ≈ ±k Slow to converge in local processing The error after relaxation v(x) = A 1 (x) e ikx + A 2 (x) e -ikx A 1 (x), A 2 (x) smooth A r (x) are represented on coarser grids: A 1  + 2 i k A 1 ′ = f 1 = r h (x) e -ikx 1D Wave Equation: u”+k 2 u=f

 k   8,  8 )  1,  1 )  2,  2 )  3,  3 )  4,  4 )  5,  5 )  6,  6 )  7,  7 ) O(H) 2D Wave Equation: Du+k 2 u=f Non-local: e i(   x +  2 y)    +    ≈ k 2 On coarser grid (meshsize H):  Fully efficient multigrid solver  Tends to Geometrical Optics  Radiation Boundary Conditions: directly on coarsest level

Σ r = 1 m A r (x) φ r (x) Generally: LU=F Non-local part of U has the form L φ r ≈ 0 A r (x) smooth {φ r } found by local processing A r represented on a coarser grid

Multigrid solvers Cost: operations per unknown Linear scalar elliptic equation (~1971)* Nonlinear Grid adaptation General boundaries, BCs* Discontinuous coefficients Disordered: coefficients, grid (FE) AMG Several coupled PDEs* (1980) Non-elliptic: high-Reynolds flow Highly indefinite: waves Many eigenfunctions (N) Near zero modes Gauge topology: Dirac eq. Inverse problems Optimal design Integral equations Statistical mechanics Massive parallel processing *Rigorous quantitative analysis (1986) FAS (1975) Within one solver (1977,1982)

N eigenfunctions Electronic structures (Kohn-Sham eq): i = 1, …, N = # electrons O (N) gridpoints per  i O (N 2 ) storage Orthogonalization O (N 3 ) operations O (N log N) storage & operations Multiscale eigenbase 1D: Livne V = V nuclear + V(  ) One shot solver

Multigrid solvers Cost: operations per unknown Linear scalar elliptic equation (~1971)* Nonlinear Grid adaptation General boundaries, BCs* Discontinuous coefficients Disordered: coefficients, grid (FE) AMG Several coupled PDEs* (1980) Non-elliptic: high-Reynolds flow Highly indefinite: waves Many eigenfunctions (N) Near zero modes Gauge topology: Dirac eq. Inverse problems Optimal design Integral equations Full matrix Statistical mechanics Massive parallel processing *Rigorous quantitative analysis (1986) FAS (1975) Within one solver (1977,1982)

Integro-differential Equation differential, dense Multigrid solver Distributive relaxation: 1 st order 2 nd order Solution cost ≈ one fast transform (one fast evaluation of the discretized integral transform)

Integral Transforms G(x,  Transform Fourier Laplace Gauss Potential Complexity  n logn)  n) G(x,  Exp(ik  Waves  n logn)

G local G(x,y) G smooth s|x-y| G(x,y) = G smooth (x,y) + G local (x,y) s ~ next coarser scale ~ 1 / | x – y | O(n) not static!

Multigrid solvers Cost: operations per unknown Linear scalar elliptic equation (~1971)* Nonlinear Grid adaptation General boundaries, BCs* Discontinuous coefficients Disordered: coefficients, grid (FE) AMG Several coupled PDEs* (1980) Non-elliptic: high-Reynolds flow Highly indefinite: waves Many eigenfunctions (N) Near zero modes Gauge topology: Dirac eq. Inverse problems Optimal design Integral equations Statistical mechanics Monte-Carlo Massive parallel processing *Rigorous quantitative analysis (1986) FAS (1975) Within one solver (1977,1982)

Discretization Lattice for accuracy Monte Carlo cost ~ “volume factor” “critical slowing down” Multiscale ~ Multigrid moves Many sampling cycles at coarse levels

Multigrid solvers Cost: operations per unknown Linear scalar elliptic equation (~1971)* Nonlinear Grid adaptation General boundaries, BCs* Discontinuous coefficients Disordered: coefficients, grid (FE) AMG Several coupled PDEs* (1980) Non-elliptic: high-Reynolds flow Highly indefinite: waves Many eigenfunctions (N) Near zero modes Gauge topology: Dirac eq. Inverse problems Optimal design Integral equations Statistical mechanics Massive parallel processing *Rigorous quantitative analysis (1986) FAS (1975) Within one solver (1977,1982)

Same fast solver Local patches of finer grids Each level correct the equations of the next coarser level Each patch may use different coordinate system and anisotropic grid “Quasicontiuum” method [B., 1992] Each patch may use different coordinate system and anisotropic grid and different physics; e.g. Atomistic and differet physics; e.g. atomistic

Repetitive systems e.g., same equations everywhere UPSCALING: Derivation of coarse equations in small windows

Scale-born obstacles: Many variables n gridpoints / particles / pixels / … Interacting with each other O(n 2 ) Slowness Slow Monte Carlo / Small time steps / … Slowly converging iterations / due to 1.Localness of processing 2.Attraction basins Removed by multiscale processing

A solution value is NOT generally determined just by few local equations A coarse equation IS generally determined just by few local equations  O (N) operations The coarse equation can be derived ONCE for all similar neighborhoods  # operations << N

Systematic Upscaling 1.Choosing coarse variables 2.Constructing coarse-level operational rules equations Hamiltonian

Macromolecule ~ second steps

Systematic Upscaling 1.Choosing coarse variables Criterion: Fast convergence of “compatible relaxation”

Systematic Upscaling 1.Choosing coarse variables Criterion: Fast equilibration of “compatible Monte Carlo” OR: Fast convergence of “compatible relaxation” Local dependence on coarse variables 2.Constructing coarse-level operational rules Done locally In representative “windows” fast

Systematic Upscaling 1.Choosing coarse variables Criterion: Fast equilibration of “compatible Monte Carlo” Local dependence on coarse variables 2.Constructing coarse-level operational rules Done locally In representative “windows” fast

Macromolecule

Potential Energy Lennard-Jones Electrostatic Bond length strain Bond angle strain torsion hydrogen bond rkrk E  ijkl riri rjrj rlrl r ij

Macromolecule Two orders of magnitude faster simulation

Macromolecule      + Lennard-Jones ~10 4 Monte Carlo passes for one T G i transition G1G1 G2G2 T Dihedral potential + Electrostatic

Fluids £ Total mass £ Total momentum £ Total dipole moment £ average location

1 1 2

Hierarchy of Coarser levels Total mass at scale s at point x: summed recursively densities at all scales Summing recursively density variations at any scale averaged to all higher scales s s s - 1

Windows Coarser level Larger density fluctuations Still coarser level

Fluids Total mass: Summing

Lower Temperature T Summing also Still lower T: More precise crystal direction and periods determined at coarser spatial levels Heisenberg uncertainty principle : Better orientational resolution at larger spatial scales

Optimization by Multiscale annealing Identifying increasingly larger-scale degrees of freedom at progressively lower temperatures Handling multiscale attraction basins E(r) r

Systematic Upscaling Rigorous computational methodology to derive from physical laws at microscopic (e.g., atomistic) level governing equations at increasingly larger scales. Scales are increased gradually (e.g., doubled at each level) with interscale feedbacks, yielding: Inexpensive computation : needed only in some small “windows” at each scale. No need to sum long-range interactions Applicable to fluids, solids, macromolecules, electronic structures, elementary particles, turbulence, … Efficient transitions between meta-stable configurations.

Upscaling Projects QCD (elementary particles): Renormalization multigrid Ron BAMG solver of Dirac eqs. Livne, Livshits Fast update of, det Rozantsev (3n +1) dimensional Schrödinger eq. Filinov Real-time Feynmann path integrals Zlochin multiscale electronic-density functional DFT electronic structures Livne, Livshits molecular dynamics Molecular dynamics: Fluids Ilyin, Suwain, Makedonska Polymers, proteins Bai, Klug Micromechanical structures Ghoniem defects, dislocations, grains Navier Stokes Turbulence McWilliams Dinar, Diskin

THANK YOU

Aggregating Regions Adaptively e.g., by similarity of densities astrophysics heights epitaxial growth color image segmentation color variances at all scales elongation continuation deblurring shapes recognition …

Cure: Multiscale Computation  Define a Coarser system  Derive equations (or probabilistic rules) governing the coarse system  Move similarly to a still-coarser system; etc.  Small computational volumes at each scale  No need to sum far interactions  No slowness  Leading to macroscopic “equations” (or tabled rules) of the material

Exact Quantum Mechanics n masses m 1, …, m n located at r 1, …, r n r j =(x j, y j, z j ) Forcespotential V(r 1, …, r n ) Classical: r(t) Probability amplitude  (r 1, …, r n, t) Approximations: Born-Oppenheimer; Hartree-Fock; Local density; perturbations Direct Numerical Real-time path integrals 3n+1 dimensional PDE (Schrodinger): roro

F cycle h0h0 h 0 /2 h 0 /4 2h h ***... * interpolation (order l+p) to a new grid interpolation (order m) of corrections relaxation sweeps algebraic error < truncation error residual transfer enough sweeps or direct solver * residual transfer no relaxation