Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Parallel Programming in C with MPI and OpenMP Michael J. Quinn

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Chapter 10 Monte Carlo Methods

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Chapter Objectives Introduce Monte Carlo methods Introduce Monte Carlo methods Introduce techniques for parallel random number generation Introduce techniques for parallel random number generation

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Outline Monte Carlo method Monte Carlo method Sequential random number generators Sequential random number generators Parallel random number generators Parallel random number generators Generating non-uniform random numbers Generating non-uniform random numbers Monte Carlo case studies Monte Carlo case studies

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Monte Carlo Method Solve a problem using statistical sampling Solve a problem using statistical sampling Name comes from Monaco’s gambling resort city Name comes from Monaco’s gambling resort city First important use in development of atomic bomb during World War II First important use in development of atomic bomb during World War II

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Applications of Monte Carlo Method Evaluating integrals of arbitrary functions of 6+ dimensions Evaluating integrals of arbitrary functions of 6+ dimensions Predicting future values of stocks Predicting future values of stocks Solving partial differential equations Solving partial differential equations Sharpening satellite images Sharpening satellite images Modeling cell populations Modeling cell populations Finding approximate solutions to NP-hard problems Finding approximate solutions to NP-hard problems

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Example of Monte Carlo Method Area = D 2 Area =  D 2 /4 D D

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Example of Monte Carlo Method Area = D 2

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Absolute Error Absolute error is a way to measure the quality of an estimate Absolute error is a way to measure the quality of an estimate The smaller the error, the better the estimate The smaller the error, the better the estimate a: actual value a: actual value e: estimated value e: estimated value Absolute error = |e-a|/a Absolute error = |e-a|/a

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Increasing Sample Size Reduces Error nEstimateError 1/(2n 1/2 ) , , , ,000, ,000, ,000, ,000,000,

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Mean Value Theorem

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Estimating Mean Value The expected value of (1/n)(f(x 0 ) + … + f(x n-1 )) is f

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Why Monte Carlo Works

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Why Monte Carlo is Effective Error in Monte Carlo estimate decreases by the factor 1/n 1/2 Error in Monte Carlo estimate decreases by the factor 1/n 1/2 Rate of convergence independent of integrand’s dimension Rate of convergence independent of integrand’s dimension Deterministic numerical integration methods do not share this property Deterministic numerical integration methods do not share this property Hence Monte Carlo superior when integrand has 6 or more dimensions Hence Monte Carlo superior when integrand has 6 or more dimensions

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Parallelism in Monte Carlo Methods Monte Carlo methods often amenable to parallelism Monte Carlo methods often amenable to parallelism Find an estimate about p times faster Find an estimate about p times faster OR OR Reduce error of estimate by p 1/2 Reduce error of estimate by p 1/2

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Random versus Pseudo-random Virtually all computers have “random number” generators Virtually all computers have “random number” generators Their operation is deterministic Their operation is deterministic Sequences are predictable Sequences are predictable More accurately called “pseudo-random number” generators More accurately called “pseudo-random number” generators In this chapter “random” is shorthand for “pseudo- random” In this chapter “random” is shorthand for “pseudo- random” “RNG” means “random number generator” “RNG” means “random number generator”

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Properties of an Ideal RNG Uniformly distributed Uniformly distributed Uncorrelated Uncorrelated Never cycles Never cycles Satisfies any statistical test for randomness Satisfies any statistical test for randomness Reproducible Reproducible Machine-independent Machine-independent Changing “seed” value changes sequence Changing “seed” value changes sequence Easily split into independent subsequences Easily split into independent subsequences Fast Fast Limited memory requirements Limited memory requirements

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. No RNG Is Ideal Finite precision arithmetic  finite number of states  cycles Finite precision arithmetic  finite number of states  cycles  Period = length of cycle  If period > number of values needed, effectively acyclic Reproducible  correlations Reproducible  correlations Often speed versus quality trade-offs Often speed versus quality trade-offs

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Linear Congruential RNGs Multiplier Additive constant Modulus Sequence depends on choice of seed, X 0

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Period of Linear Congruential RNG Maximum period is M Maximum period is M For 32-bit integers maximum period is 2 32, or about 4 billion For 32-bit integers maximum period is 2 32, or about 4 billion This is too small for modern computers This is too small for modern computers Use a generator with at least 48 bits of precision Use a generator with at least 48 bits of precision

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Producing Floating-Point Numbers X i, a, c, and M are all integers X i, a, c, and M are all integers X i s range in value from 0 to M-1 X i s range in value from 0 to M-1 To produce floating-point numbers in range [0, 1), divide X i by M To produce floating-point numbers in range [0, 1), divide X i by M

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Defects of Linear Congruential RNGs Least significant bits correlated Least significant bits correlated  Especially when M is a power of 2 k-tuples of random numbers form a lattice k-tuples of random numbers form a lattice  Especially pronounced when k is large

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Lagged Fibonacci RNGs p and q are lags, p > q p and q are lags, p > q * is any binary arithmetic operation * is any binary arithmetic operation Addition modulo M Addition modulo M Subtraction modulo M Subtraction modulo M Multiplication modulo M Multiplication modulo M Bitwise exclusive or Bitwise exclusive or

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Properties of Lagged Fibonacci RNGs Require p seed values Require p seed values Careful selection of seed values, p, and q can result in very long periods and good randomness Careful selection of seed values, p, and q can result in very long periods and good randomness For example, suppose M has b bits For example, suppose M has b bits Maximum period for additive lagged Fibonacci RNG is (2 p -1)2 b-1 Maximum period for additive lagged Fibonacci RNG is (2 p -1)2 b-1

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Ideal Parallel RNGs All properties of sequential RNGs All properties of sequential RNGs No correlations among numbers in different sequences No correlations among numbers in different sequences Scalability Scalability Locality Locality

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Parallel RNG Designs Manager-worker Manager-worker Leapfrog Leapfrog Sequence splitting Sequence splitting Independent sequences Independent sequences

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Manager-Worker Parallel RNG Manager process generates random numbers Manager process generates random numbers Worker processes consume them Worker processes consume them If algorithm is synchronous, may achieve goal of consistency If algorithm is synchronous, may achieve goal of consistency Not scalable Not scalable Does not exhibit locality Does not exhibit locality

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Leapfrog Method Process with rank 1 of 4 processes

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Properties of Leapfrog Method Easy modify linear congruential RNG to support jumping by p Easy modify linear congruential RNG to support jumping by p Can allow parallel program to generate same tuples as sequential program Can allow parallel program to generate same tuples as sequential program Does not support dynamic creation of new random number streams Does not support dynamic creation of new random number streams

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Sequence Splitting Process with rank 1 of 4 processes

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Properties of Sequence Splitting Forces each process to move ahead to its starting point Forces each process to move ahead to its starting point Does not support goal of reproducibility Does not support goal of reproducibility May run into long-range correlation problems May run into long-range correlation problems Can be modified to support dynamic creation of new sequences Can be modified to support dynamic creation of new sequences

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Independent Sequences Run sequential RNG on each process Run sequential RNG on each process Start each with different seed(s) or other parameters Start each with different seed(s) or other parameters Example: linear congruential RNGs with different additive constants Example: linear congruential RNGs with different additive constants Works well with lagged Fibonacci RNGs Works well with lagged Fibonacci RNGs Supports goals of locality and scalability Supports goals of locality and scalability

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Other Distributions Analytical transformations Analytical transformations Box-Muller Transformation Box-Muller Transformation Rejection method Rejection method

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Analytical Transformation

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Exponential Distribution 1.0

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Example 1: Produce four samples from an exponential distribution with mean 3 Produce four samples from an exponential distribution with mean 3 Uniform sample: 0.540, 0.619, 0.452, Uniform sample: 0.540, 0.619, 0.452, Take natural log of each value and multiply by -3 Take natural log of each value and multiply by -3 Exponential sample: 1.850, 1.440, 2.317, Exponential sample: 1.850, 1.440, 2.317, 7.072

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Example 2: Simulation advances in time steps of 1 second Simulation advances in time steps of 1 second Probability of an event happening is from an exponential distribution with mean 5 seconds Probability of an event happening is from an exponential distribution with mean 5 seconds What is probability that event will happen in next second? What is probability that event will happen in next second? 1/5 1/5 Use uniform random number to test for occurrence of event Use uniform random number to test for occurrence of event

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Box-Muller Transformation Cannot invert cumulative distribution function to produce formula yielding random numbers from normal (gaussian) distribution Cannot invert cumulative distribution function to produce formula yielding random numbers from normal (gaussian) distribution Box-Muller transformation produces a pair of standard deviates g 1 and g 2 from a pair of normal deviates u 1 and u 2 Box-Muller transformation produces a pair of standard deviates g 1 and g 2 from a pair of normal deviates u 1 and u 2

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Box-Muller Transformation repeat v 1  2u v 2  2u r  v v 2 2 until r > 0 and r < 1 f  sqrt (-2 ln r /r ) g 1  f v 1 g 2  f v 2

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Example Produce four samples from a normal distribution with mean 0 and standard deviation 1 Produce four samples from a normal distribution with mean 0 and standard deviation 1 u1u1u1u1 u2u2u2u2 v1v1v1v1 v2v2v2v2rf g1g1g1g1 g2g2g2g

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Different Mean, Std. Dev. g 1  s f v 1 + m g 2  s f v 2 + m Standard deviationMean

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Rejection Method

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Example Generate random variables from this probability density function Generate random variables from this probability density function

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Example (cont.) So  h(x)  f(x) for all x

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Example (cont.) xixixixi uiuiuiui u i  h(x i ) f(x i ) Outcome Reject Accept Reject Accept Two samples from f(x) are and 1.306

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Case Studies (Topics Introduced) Neutron transport (Monte Carlo time) Neutron transport (Monte Carlo time) Temperature inside a 2-D plate (Random walk) Temperature inside a 2-D plate (Random walk) Two-dimensional Ising model (Metropolis algorithm) Two-dimensional Ising model (Metropolis algorithm) Room assignment problem (Simulated annealing) Room assignment problem (Simulated annealing) Parking garage (Monte Carlo time) Parking garage (Monte Carlo time) Traffic circle (Simulating queues) Traffic circle (Simulating queues)

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Neutron Transport

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Example D (0-  ) Angleu(0-1)L (-ln u) LcosD Dist.Absorb?(0-1) (no) (no) (no) Monte Carlo Time 3.0

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Temperature Inside a 2-D Plate Random walk

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Example of Random Walk

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. 2-D Ising Model

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Metropolis Algorithm Use current random sample to generate next random sample Use current random sample to generate next random sample Series of samples represents a random walk through the probability density function Series of samples represents a random walk through the probability density function Short series of samples highly correlated Short series of samples highly correlated Many samples can provide good coverage Many samples can provide good coverage

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Metropolis Algorithm Details Randomly select site to reverse spin Randomly select site to reverse spin If energy is lower, move to new state If energy is lower, move to new state Otherwise, move with probability  = e -  /kT Otherwise, move with probability  = e -  /kT Rejection causes current state to be recorded another time Rejection causes current state to be recorded another time

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Room Assignment Problem ABCDEF A B C D E F “Dislikes” matrix Pairing A-B, C-D, and E-F leads to total conflict value of 32.

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Physical Annealing Heat a solid until it melts Heat a solid until it melts Cool slowly to allow material to reach state of minimum energy Cool slowly to allow material to reach state of minimum energy Produces strong, defect-free crystal with regular structure Produces strong, defect-free crystal with regular structure

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Simulated Annealing Makes analogy between physical annealing and solving combinatorial optimization problem Makes analogy between physical annealing and solving combinatorial optimization problem Solution to problem = state of material Solution to problem = state of material Value of objective function = energy associated with state Value of objective function = energy associated with state Optimal solution = minimum energy state Optimal solution = minimum energy state

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. How Simulated Annealing Works Iterative algorithm, slowly lower T Iterative algorithm, slowly lower T Randomly change solution to create alternate solution Randomly change solution to create alternate solution Compute , the change in value of objective function Compute , the change in value of objective function If  < 0, then jump to alternate solution If  < 0, then jump to alternate solution Otherwise, jump to alternate solution with probability e -  /T Otherwise, jump to alternate solution with probability e -  /T

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Performance of Simulated Annealing Rate of convergence depends on initial value of T and temperature change function Rate of convergence depends on initial value of T and temperature change function Geometric temperature change functions typical; e.g., T i+1 = T i Geometric temperature change functions typical; e.g., T i+1 = T i Not guaranteed to find optimal solution Not guaranteed to find optimal solution Same algorithm using different random number streams can converge on different solutions Same algorithm using different random number streams can converge on different solutions Opportunity for parallelism Opportunity for parallelism

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Convergence Starting with higher initial temperature leads to more iterations before convergence

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Parking Garage Parking garage has S stalls Parking garage has S stalls Car arrivals fit Poisson distribution with mean A Car arrivals fit Poisson distribution with mean A Stay in garage fits a normal distribution with mean M and standard deviation M/S Stay in garage fits a normal distribution with mean M and standard deviation M/S

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Implementation Idea Current Time Times Spaces Are Available 15 Car CountCars Rejected 2

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Traffic Circle

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Traffic Circle Probabilities F N0.33 E0.50 S0.25 W0.33DNESWN E S W

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Traffic Circle Data Structures

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Summary (1/3) Applications of Monte Carlo methods Applications of Monte Carlo methods  Numerical integration  Simulation Random number generators Random number generators  Linear congruential  Lagged Fibonacci

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Summary (2/3) Parallel random number generators Parallel random number generators  Manager/worker  Leapfrog  Sequence splitting  Independent sequences Non-uniform distributions Non-uniform distributions  Analytical transformations  Box-Muller transformation  Rejection method

Copyright © The McGraw-Hill Companies, Inc. Permission required for reproduction or display. Summary (3/3) Concepts revealed in case studies Concepts revealed in case studies  Monte Carlo time  Random walk  Metropolis algorithm  Simulated annealing  Modeling queues