1. July 11, 2006Bayesian Inference and Maximum Entropy 20061 Probing the covariance matrix Kenneth M. Hanson T-16, Nuclear Physics; Theoretical Division Los Alamos National Laboratory This presentation available at http://www.lanl.gov/home/kmh/ LA-UR-06-xxxx Bayesian Inference and Maximum Entropy Workshop, July 9-13, 2006

2. July 11, 2006Bayesian Inference and Maximum Entropy 20062 Overview Analogy between minus-log-probability and a physical potential Gaussian approximation Probing the covariance matrix with an external force ► deterministic technique to replace stochastic calculations Examples Potential applications

3. July 11, 2006Bayesian Inference and Maximum Entropy 20063 Analogy to physical system Analogy between minus-log-posterior and a physical potential ► a represents parameters d represents data I represents background information, essential for modeling Gradient ∂ a φ corresponds to forces acting on the parameters Maximum a posteriori (MAP) estimates parameters â MAP ► condition is ∂ a φ = 0 ► optimized model may be interpreted as mechanical system in equilibrium – net force on each parameter is zero This analogy is very useful for Bayesian inference ► conceptualization ► developing algorithms

4. July 11, 2006Bayesian Inference and Maximum Entropy 20064 Gaussian approximation Posterior distribution is very often well approximated by a Gaussian Then, φ is quadratic in perturbations in the model parameters a – â = δa from the minimum in φ at â: where K is the φ curvature matrix (aka Hessian); Uncertainties in the estimated parameters is summarized by the covariance matrix: Inference process reduced to finding â and C

5. July 11, 2006Bayesian Inference and Maximum Entropy 20065 External force Consider applying an constant external force to the parameters Effect is to add a linearly increasing piece to potential Gradient of perturbed potential is At the new minimum, gradient is zero, or Displacement of minimum a is proportional to covariance matrix times the force With the external force, one may “probe” the covariance

6. July 11, 2006Bayesian Inference and Maximum Entropy 20066 Effect of external force Displacement of minimizer of φ is in direction different than applied force ► its direction is affected by covariance matrix 2-D parameter space Force, f Displacement, δa a b φ contour

7. July 11, 2006Bayesian Inference and Maximum Entropy 20067 Fit straight line to data Linear model: Simulate 10 data points, exact values: Determine parameters, intercept a and slope b, by minimizing chi- squared (standard least-squares analysis) Result: Strong correlations between a and b Best fit10 data points Scatter plot

8. July 11, 2006Bayesian Inference and Maximum Entropy 20068 Apply force to solution Apply upward force to solution line at x = 0 and find new minimum in φ Effect is to pull line upward at x = 0 and reduce its slope ► data constrain solution Conclude that parameters a (intercept) and b (slope) are anti-correlated Furthermore, these relationships yield quantified results Pull upward on line

9. July 11, 2006Bayesian Inference and Maximum Entropy 20069 Straight line fit Family of lines for forces applied upward at x = 0: f = ± 1, 2 σ a -1 Upward force at x = 0 f = ± 1, 2 σ a -1

10. July 11, 2006Bayesian Inference and Maximum Entropy 200610 Straight line fit Family of lines for forces applied upward at x = 0 Plot on top shows ► perturbations proportional to f ► slope of δa = σ a -2 = C aa ► slope of δb = C ab Plot below shows φ (or χ 2 ) is quadratic function of force ► for force of f = ± σ a -1 ; min φ increases by 0.5, or min χ 2 increases by 1 Either dependence provides way to quantify variance f at x = 0

11. July 11, 2006Bayesian Inference and Maximum Entropy 200611 Simple spectrum Simulate a simple spectrum: ► Gaussian peak (ampl = 2, w = 0.2) ► quadratic background ► add random noise (rmsdev = 0.2) Fit involves 6 parameters ► nonlinear problem ► results: parameters of interest ampl. width ► fair degree of correlation

12. July 11, 2006Bayesian Inference and Maximum Entropy 200612 Simple spectrum – apply force to area To probe area under Gaussian peak, apply force appropriate to area Force should be proportional to derivatives of area wrt parameters, a = amplitude, w = rms width: Plot shows result of applying force to these two parameters in this proportion

13. July 11, 2006Bayesian Inference and Maximum Entropy 200613 Simple spectrum Plots for +/– forces applied to area Plot below shows nonlinear response, but approximately linear for small f ► slope at 0 is σ A -2 ► φ increases by 0.5 for | f | = σ A -1 Other displacements give covariance wrt area f = 3.4σ A -1 f = - 8σ A -1

14. July 11, 2006Bayesian Inference and Maximum Entropy 200614 Tomographic reconstruction from two views Problem - reconstruct uniform-density object from two projections ► 2 orthogonal, parallel projections (128 samples in each view) ► Gaussian noise added Original object Two orthogonal projections with 5% rms noise

15. July 11, 2006Bayesian Inference and Maximum Entropy 200615 The Bayes Inference Engine BIE data-flow diagram to find max. a posteriori (MAP) solution ► 0ptimizer uses gradients that are efficiently calculated by adjoint differentiation, a key capability of the BIE Boundary description Input projections

16. July 11, 2006Bayesian Inference and Maximum Entropy 200616 MAP reconstruction – two views Model object in terms of: ► deformable polygonal boundary with 50 vertices ► smoothness constraint ► constant interior density Determine boundary that maximizes posterior probability Not perfect, but very good for only two projections Question is: How do we quantify uncertainty in reconstruction? Reconstructed boundary (gray-scale) compared with original object (red line)

17. July 11, 2006Bayesian Inference and Maximum Entropy 200617 Tomographic reconstruction from two views Stiffness of model proportional to curvature of  Displacement obtained by applying a force to MAP model and re-minimizing  is proportional to a row of the covariance matrix Displacement divided by force ► at position of force is proportional to variance there ► elsewhere, proportional to covariance Applying force (white bar) to MAP boundary (red) moves it to new location (yellow-dashed)

18. July 11, 2006Bayesian Inference and Maximum Entropy 200618 Situations where probing covariance useful Technique will be most useful when ► posterior can be well approximated by Gaussian pdf in parameters ► interest is in uncertainty of one or a few quantities, but there are many parameters ► optimization easy to do ► gradient calculation can be done efficiently, e.g. by adjoint differentiation of the forward simulation code ► self-optimizing natural systems (populations, bacteria, traffic) May be useful in contexts other than probabilistic inference where Gaussian pdfs are used

19. July 11, 2006Bayesian Inference and Maximum Entropy 200619 Summary Technique has been presented ► based on interpreting minus-log-posterior as physical potential ► probe covariance matrix by applying force to estimated model ► stochastic calculation replaced by deterministic one ► may be related to fluctuation-dissipation relation from statistical mechanics

20. July 11, 2006Bayesian Inference and Maximum Entropy 200620 Bibliography ► "The hard truth," K. M. Hanson and G. S. Cunningham, Maximum Entropy and Bayesian Methods, J. Skilling and S. Sibisi, eds., pp. 157-164 (Kluwer Academic, Dordrecht, 1996) ► Uncertainty assessment for reconstructions based on deformable models," K. M. Hanson et al., Int. J. Imaging Syst. Technol. 8, pp. 506-512 (1997) ► "Operation of the Bayes Inference Engine," K. M. Hanson and G. S. Cunningham, Maximum Entropy and Bayesian Methods, W. von der Linden et al., eds., pp. 309-318 (Kluwer Academic, Dordrecht, 1999) ► “Evaluating Derivatives: Principles and Techniques of Algorithmic Differentiation,” A. Griewank (SIAM, 2000) This presentation available at http://www.lanl.gov/home/kmh/

21. July 11, 2006Bayesian Inference and Maximum Entropy 200621 Likelihood analysis – chi squared When the errors in each measurement are Gaussian distributed and independent, likelihood is related to chi squared: near minimum, χ 2 is approximately quadratic in the parameters a ► where â is the parameter vector at minimum χ 2 and K is the χ 2 curvature matrix (aka the Hessian) The covariance matrix for the uncertainties in the estimated parameters is

22. July 11, 2006Bayesian Inference and Maximum Entropy 200622 Apply force to solution Apply upward force to solution line at x = 0: f = σ a -1 = 7.874 Pull upward on line

23. July 11, 2006Bayesian Inference and Maximum Entropy 200623 example Apply force to just amplitude, a