Lecture 3: Markov processes, master equation Outline: Preliminaries and definitions Chapman-Kolmogorov equation Wiener process Markov chains eigenvectors and eigenvalues detailed balance Monte Carlo master equation

Stochastic processes Random function x(t)

Stochastic processes Random function x(t) Defined by a distribution functional P[x], or by all its moments

Stochastic processes Random function x(t) Defined by a distribution functional P[x], or by all its moments

Stochastic processes Random function x(t) Defined by a distribution functional P[x], or by all its moments or by its characteristic functional:

Stochastic processes Random function x(t) Defined by a distribution functional P[x], or by all its moments or by its characteristic functional:

Stochastic processes (2) Cumulant generating functional:

Stochastic processes (2) Cumulant generating functional:

Stochastic processes (2) Cumulant generating functional: where

Stochastic processes (2) Cumulant generating functional: where correlation function

Stochastic processes (2) Cumulant generating functional: where etc. correlation function

Stochastic processes (3) Gaussian process:

Stochastic processes (3) Gaussian process:

Stochastic processes (3) Gaussian process: (no higher-order cumulants)

Stochastic processes (3) Gaussian process: (no higher-order cumulants) Conditional probabilities:

Stochastic processes (3) Gaussian process: (no higher-order cumulants) Conditional probabilities:

Stochastic processes (3) Gaussian process: (no higher-order cumulants) Conditional probabilities: = probability of x(t1) … x(tk), given x(tk+1) … x(tm)

Wiener-Khinchin theorem Fourier analyze x(t):

Wiener-Khinchin theorem Fourier analyze x(t): Power spectrum:

Wiener-Khinchin theorem Fourier analyze x(t): Power spectrum:

Wiener-Khinchin theorem Fourier analyze x(t): Power spectrum:

Wiener-Khinchin theorem Fourier analyze x(t): Power spectrum:

Wiener-Khinchin theorem Fourier analyze x(t): Power spectrum:

Wiener-Khinchin theorem Fourier analyze x(t): Power spectrum: Power spectrum is Fourier transform of the correlation function

Markov processes No information about the future from past values earlier than the latest available:

Markov processes No information about the future from past values earlier than the latest available:

Markov processes No information about the future from past values earlier than the latest available: Can get general distribution by iterating Q:

Markov processes No information about the future from past values earlier than the latest available: Can get general distribution by iterating Q:

Markov processes No information about the future from past values earlier than the latest available: Can get general distribution by iterating Q: where P(x(t0)) is the initial distribution.

Markov processes No information about the future from past values earlier than the latest available: Can get general distribution by iterating Q: where P(x(t0)) is the initial distribution. Integrate this over x(tn-1), … x(t1) to get

Markov processes No information about the future from past values earlier than the latest available: Can get general distribution by iterating Q: where P(x(t0)) is the initial distribution. Integrate this over x(tn-1), … x(t1) to get

Markov processes No information about the future from past values earlier than the latest available: Can get general distribution by iterating Q: where P(x(t0)) is the initial distribution. Integrate this over x(tn-1), … x(t1) to get The case n = 2 is the

Chapman-Kolmogorov equation

Chapman-Kolmogorov equation

Chapman-Kolmogorov equation (for any t’)

Chapman-Kolmogorov equation (for any t’) Examples: Wiener process (Brownian motion/random walk):

Chapman-Kolmogorov equation (for any t’) Examples: Wiener process (Brownian motion/random walk):

Chapman-Kolmogorov equation (for any t’) Examples: Wiener process (Brownian motion/random walk): (cumulative) Poisson process

Markov chains Both t and x discrete, assuming stationarity

Markov chains Both t and x discrete, assuming stationarity

Markov chains Both t and x discrete, assuming stationarity (because they are probabilities)

Markov chains Both t and x discrete, assuming stationarity (because they are probabilities) Equation of motion:

Markov chains Both t and x discrete, assuming stationarity (because they are probabilities) Equation of motion: Formal solution:

Markov chains (2): properties of T T has a left eigenvector

Markov chains (2): properties of T T has a left eigenvector (because )

Markov chains (2): properties of T T has a left eigenvector (because ) Its eigenvalue is 1.

Markov chains (2): properties of T T has a left eigenvector (because ) Its eigenvalue is 1. The corresponding right eigenvector is

Markov chains (2): properties of T T has a left eigenvector (because ) Its eigenvalue is 1. The corresponding right eigenvector is (the stationary state, because the eigenvalue is 1: )

Markov chains (2): properties of T T has a left eigenvector (because ) Its eigenvalue is 1. The corresponding right eigenvector is (the stationary state, because the eigenvalue is 1: ) For all other right eigenvectors with components

Markov chains (2): properties of T T has a left eigenvector (because ) Its eigenvalue is 1. The corresponding right eigenvector is (the stationary state, because the eigenvalue is 1: ) For all other right eigenvectors with components

Markov chains (2): properties of T T has a left eigenvector (because ) Its eigenvalue is 1. The corresponding right eigenvector is (the stationary state, because the eigenvalue is 1: ) For all other right eigenvectors with components (because they must be orthogonal to : )

Markov chains (2): properties of T T has a left eigenvector (because ) Its eigenvalue is 1. The corresponding right eigenvector is (the stationary state, because the eigenvalue is 1: ) For all other right eigenvectors with components (because they must be orthogonal to : ) All other eigenvalues are < 1.

Detailed balance If there is a stationary distribution P0 with components and

Detailed balance If there is a stationary distribution P0 with components and

Detailed balance If there is a stationary distribution P0 with components and

Detailed balance If there is a stationary distribution P0 with components and can prove (if ergodicity*) convergence to P0 from any initial state:

Detailed balance If there is a stationary distribution P0 with components and can prove (if ergodicity*) convergence to P0 from any initial state: * Can reach any state from any other and no cycles

Detailed balance If there is a stationary distribution P0 with components and can prove (if ergodicity*) convergence to P0 from any initial state: Define , * Can reach any state from any other and no cycles

Detailed balance If there is a stationary distribution P0 with components and can prove (if ergodicity*) convergence to P0 from any initial state: Define , make a similarity transformation * Can reach any state from any other and no cycles

Detailed balance If there is a stationary distribution P0 with components and can prove (if ergodicity*) convergence to P0 from any initial state: Define , make a similarity transformation * Can reach any state from any other and no cycles

Detailed balance If there is a stationary distribution P0 with components and can prove (if ergodicity*) convergence to P0 from any initial state: Define , make a similarity transformation R is symmetric, has complete set of eigenvectors , components (Eigenvalues λj same as those of T.) * Can reach any state from any other and no cycles

Detailed balance (2)

Detailed balance (2)

Detailed balance (2)

Detailed balance (2) Right eigenvectors of T:

Detailed balance (2) Right eigenvectors of T: Now look at evolution:

Detailed balance (2) Right eigenvectors of T: Now look at evolution:

Detailed balance (2) Right eigenvectors of T: Now look at evolution:

Detailed balance (2) Right eigenvectors of T: Now look at evolution:

Detailed balance (2) Right eigenvectors of T: Now look at evolution: (since )

Detailed balance (2) Right eigenvectors of T: Now look at evolution: (since )

Monte Carlo an example of detailed balance

Monte Carlo an example of detailed balance Ising model: Binary “spins” Si(t) = ±1

Monte Carlo an example of detailed balance Ising model: Binary “spins” Si(t) = ±1 Dynamics: at every time step,

Monte Carlo an example of detailed balance Ising model: Binary “spins” Si(t) = ±1 Dynamics: at every time step, (1) choose a spin (i) at random

Monte Carlo an example of detailed balance Ising model: Binary “spins” Si(t) = ±1 Dynamics: at every time step, (1) choose a spin (i) at random (2) compute “field” of neighbors hi(t) = ΣjJijSj(t)

Monte Carlo an example of detailed balance Ising model: Binary “spins” Si(t) = ±1 Dynamics: at every time step, (1) choose a spin (i) at random (2) compute “field” of neighbors hi(t) = ΣjJijSj(t) Jij = Jji

Monte Carlo an example of detailed balance Ising model: Binary “spins” Si(t) = ±1 Dynamics: at every time step, (1) choose a spin (i) at random (2) compute “field” of neighbors hi(t) = ΣjJijSj(t) Jij = Jji (3) Si(t + Δt) = +1 with probability

Monte Carlo an example of detailed balance Ising model: Binary “spins” Si(t) = ±1 Dynamics: at every time step, (1) choose a spin (i) at random (2) compute “field” of neighbors hi(t) = ΣjJijSj(t) Jij = Jji (3) Si(t + Δt) = +1 with probability

Monte Carlo an example of detailed balance Ising model: Binary “spins” Si(t) = ±1 Dynamics: at every time step, (1) choose a spin (i) at random (2) compute “field” of neighbors hi(t) = ΣjJijSj(t) Jij = Jji (3) Si(t + Δt) = +1 with probability

Monte Carlo an example of detailed balance Ising model: Binary “spins” Si(t) = ±1 Dynamics: at every time step, (1) choose a spin (i) at random (2) compute “field” of neighbors hi(t) = ΣjJijSj(t) Jij = Jji (3) Si(t + Δt) = +1 with probability (equilibration of Si, given current values of other S’s)

Monte Carlo (2) In language of Markov chains, states (n) are

Monte Carlo (2) In language of Markov chains, states (n) are Single-spin flips: transitions only between neighboring points on hypercube

Monte Carlo (2) In language of Markov chains, states (n) are Single-spin flips: transitions only between neighboring points on hypercube

Monte Carlo (2) In language of Markov chains, states (n) are Single-spin flips: transitions only between neighboring points on hypercube T matrix elements:

Monte Carlo (2) In language of Markov chains, states (n) are Single-spin flips: transitions only between neighboring points on hypercube T matrix elements: all other Tmn = 0.

Monte Carlo (2) In language of Markov chains, states (n) are Single-spin flips: transitions only between neighboring points on hypercube T matrix elements: all other Tmn = 0. Note:

Monte Carlo (2) In language of Markov chains, states (n) are Single-spin flips: transitions only between neighboring points on hypercube T matrix elements: all other Tmn = 0. Note:

Monte Carlo (3) T satisfies detailed balance:

Monte Carlo (3) T satisfies detailed balance: where p0 is the Gibbs distribution:

Monte Carlo (3) T satisfies detailed balance: where p0 is the Gibbs distribution: After many Monte Carlo steps, converge to p0:

Monte Carlo (3) T satisfies detailed balance: where p0 is the Gibbs distribution: After many Monte Carlo steps, converge to p0: S’s sample Gibbs distribution

Monte Carlo (3): Metropolis version The foregoing was for “heat-bath” MC. Another possibility is the Metropolis algorithm:

Monte Carlo (3): Metropolis version The foregoing was for “heat-bath” MC. Another possibility is the Metropolis algorithm: If hiSi < 0, Si(t+Δt) = -Si(t),

Monte Carlo (3): Metropolis version The foregoing was for “heat-bath” MC. Another possibility is the Metropolis algorithm: If hiSi < 0, Si(t+Δt) = -Si(t), If hiSi > 0, Si(t+Δt) = -Si(t) with probability exp(-hiSi)

Monte Carlo (3): Metropolis version The foregoing was for “heat-bath” MC. Another possibility is the Metropolis algorithm: If hiSi < 0, Si(t+Δt) = -Si(t), If hiSi > 0, Si(t+Δt) = -Si(t) with probability exp(-hiSi) Thus,

Monte Carlo (3): Metropolis version The foregoing was for “heat-bath” MC. Another possibility is the Metropolis algorithm: If hiSi < 0, Si(t+Δt) = -Si(t), If hiSi > 0, Si(t+Δt) = -Si(t) with probability exp(-hiSi) Thus,

Monte Carlo (3): Metropolis version The foregoing was for “heat-bath” MC. Another possibility is the Metropolis algorithm: If hiSi < 0, Si(t+Δt) = -Si(t), If hiSi > 0, Si(t+Δt) = -Si(t) with probability exp(-hiSi) Thus, In either case,

Monte Carlo (3): Metropolis version The foregoing was for “heat-bath” MC. Another possibility is the Metropolis algorithm: If hiSi < 0, Si(t+Δt) = -Si(t), If hiSi > 0, Si(t+Δt) = -Si(t) with probability exp(-hiSi) Thus, In either case, i.e., detailed balance with Gibbs

Continuous-time limit: master equation For Markov chain:

Continuous-time limit: master equation For Markov chain:

Continuous-time limit: master equation For Markov chain: Differential equation:

Continuous-time limit: master equation For Markov chain: Differential equation: In components:

Continuous-time limit: master equation For Markov chain: Differential equation: In components: (using normalization of columns of T:)

Continuous-time limit: master equation For Markov chain: Differential equation: In components: (using normalization of columns of T:) (expect , m ≠ n)

Continuous-time limit: master equation For Markov chain: Differential equation: In components: (using normalization of columns of T:) (expect , m ≠ n) transition rate matrix