Bayesian Modelling Harry R. Erwin, PhD School of Computing and Technology University of Sunderland.

Slides:



Advertisements
Similar presentations
Bayes rule, priors and maximum a posteriori
Advertisements

Introduction to Monte Carlo Markov chain (MCMC) methods
MCMC estimation in MlwiN
Lecture #4: Bayesian analysis of mapped data Spatial statistics in practice Center for Tropical Ecology and Biodiversity, Tunghai University & Fushan Botanical.
Markov Chain Monte Carlo Convergence Diagnostics: A Comparative Review By Mary Kathryn Cowles and Bradley P. Carlin Presented by Yuting Qi 12/01/2006.
METHODS FOR HAPLOTYPE RECONSTRUCTION
Bayesian Estimation in MARK
Ch 11. Sampling Models Pattern Recognition and Machine Learning, C. M. Bishop, Summarized by I.-H. Lee Biointelligence Laboratory, Seoul National.
Gibbs Sampling Qianji Zheng Oct. 5th, 2010.
Introduction to Bayesian Statistics Harry R. Erwin, PhD School of Computing and Technology University of Sunderland.
Bayesian statistics – MCMC techniques
Stochastic approximate inference Kay H. Brodersen Computational Neuroeconomics Group Department of Economics University of Zurich Machine Learning and.
BAYESIAN INFERENCE Sampling techniques
Computing the Posterior Probability The posterior probability distribution contains the complete information concerning the parameters, but need often.
The University of Texas at Austin, CS 395T, Spring 2008, Prof. William H. Press IMPRS Summer School 2009, Prof. William H. Press 1 4th IMPRS Astronomy.
Particle filters (continued…). Recall Particle filters –Track state sequence x i given the measurements ( y 0, y 1, …., y i ) –Non-linear dynamics –Non-linear.
Today Introduction to MCMC Particle filters and MCMC
Using ranking and DCE data to value health states on the QALY scale using conventional and Bayesian methods Theresa Cain.
Bayesian Analysis for Extreme Events Pao-Shin Chu and Xin Zhao Department of Meteorology School of Ocean & Earth Science & Technology University of Hawaii-
G. Cowan Lectures on Statistical Data Analysis Lecture 10 page 1 Statistical Data Analysis: Lecture 10 1Probability, Bayes’ theorem 2Random variables and.
Analyzing iterated learning Tom Griffiths Brown University Mike Kalish University of Louisiana.
Introduction to Monte Carlo Methods D.J.C. Mackay.
ECE 8443 – Pattern Recognition LECTURE 06: MAXIMUM LIKELIHOOD AND BAYESIAN ESTIMATION Objectives: Bias in ML Estimates Bayesian Estimation Example Resources:
Introduction to MCMC and BUGS. Computational problems More parameters -> even more parameter combinations Exact computation and grid approximation become.
Priors, Normal Models, Computing Posteriors
Machine Learning Lecture 23: Statistical Estimation with Sampling Iain Murray’s MLSS lecture on videolectures.net:
R2WinBUGS: Using R for Bayesian Analysis Matthew Russell Rongxia Li 2 November Northeastern Mensurationists Meeting.
CSC321: 2011 Introduction to Neural Networks and Machine Learning Lecture 11: Bayesian learning continued Geoffrey Hinton.
A Comparison of Two MCMC Algorithms for Hierarchical Mixture Models Russell Almond Florida State University College of Education Educational Psychology.
Bayesian Reasoning: Tempering & Sampling A/Prof Geraint F. Lewis Rm 560:
Latent Class Regression Model Graphical Diagnostics Using an MCMC Estimation Procedure Elizabeth S. Garrett Scott L. Zeger Johns Hopkins University
Bayesian Statistics Lecture 8 Likelihood Methods in Forest Ecology October 9 th – 20 th, 2006.
- 1 - Overall procedure of validation Calibration Validation Figure 12.4 Validation, calibration, and prediction (Oberkampf and Barone, 2004 ). Model accuracy.
Beam Sampling for the Infinite Hidden Markov Model by Jurgen Van Gael, Yunus Saatic, Yee Whye Teh and Zoubin Ghahramani (ICML 2008) Presented by Lihan.
MCMC reconstruction of the 2 HE cascade events Dmitry Chirkin, UW Madison.
Markov Chain Monte Carlo for LDA C. Andrieu, N. D. Freitas, and A. Doucet, An Introduction to MCMC for Machine Learning, R. M. Neal, Probabilistic.
Lecture #9: Introduction to Markov Chain Monte Carlo, part 3
Bayesian Travel Time Reliability
Nonparametric Methods II 1 Henry Horng-Shing Lu Institute of Statistics National Chiao Tung University
Reducing MCMC Computational Cost With a Two Layered Bayesian Approach
Item Parameter Estimation: Does WinBUGS Do Better Than BILOG-MG?
Statistics Sampling Distributions and Point Estimation of Parameters Contents, figures, and exercises come from the textbook: Applied Statistics and Probability.
1 Chapter 8: Model Inference and Averaging Presented by Hui Fang.
Introduction to Sampling Methods Qi Zhao Oct.27,2004.
The Unscented Particle Filter 2000/09/29 이 시은. Introduction Filtering –estimate the states(parameters or hidden variable) as a set of observations becomes.
CS Statistical Machine learning Lecture 25 Yuan (Alan) Qi Purdue CS Nov
Bayesian statistics named after the Reverend Mr Bayes based on the concept that you can estimate the statistical properties of a system after measuting.
G. Cowan Lectures on Statistical Data Analysis Lecture 10 page 1 Statistical Data Analysis: Lecture 10 1Probability, Bayes’ theorem 2Random variables and.
Kevin Stevenson AST 4762/5765. What is MCMC?  Random sampling algorithm  Estimates model parameters and their uncertainty  Only samples regions of.
SIR method continued. SIR: sample-importance resampling Find maximum likelihood (best likelihood × prior), Y Randomly sample pairs of r and N 1973 For.
CSC321: Lecture 8: The Bayesian way to fit models Geoffrey Hinton.
Hierarchical Models. Conceptual: What are we talking about? – What makes a statistical model hierarchical? – How does that fit into population analysis?
CS498-EA Reasoning in AI Lecture #19 Professor: Eyal Amir Fall Semester 2011.
Generalization Performance of Exchange Monte Carlo Method for Normal Mixture Models Kenji Nagata, Sumio Watanabe Tokyo Institute of Technology.
Markov Chain Monte Carlo in R
Advanced Statistical Computing Fall 2016
Bayesian data analysis
ICS 280 Learning in Graphical Models
Introduction to the bayes Prefix in Stata 15
Markov Chain Monte Carlo
Markov chain monte carlo
Remember that our objective is for some density f(y|) for observations where y and  are vectors of data and parameters,  being sampled from a prior.
CAP 5636 – Advanced Artificial Intelligence
Predictive distributions
Multidimensional Integration Part I
Where did we stop? The Bayes decision rule guarantees an optimal classification… … But it requires the knowledge of P(ci|x) (or p(x|ci) and P(ci)) We.
Ch13 Empirical Methods.
CS 188: Artificial Intelligence
CS 594: Empirical Methods in HCC Introduction to Bayesian Analysis
CS639: Data Management for Data Science
Presentation transcript:

Bayesian Modelling Harry R. Erwin, PhD School of Computing and Technology University of Sunderland

Resources Kéry, Marc (2010) Introduction to WinBUGS for Ecologists, Academic Press Ntzoufras, Ioannis (2009) Bayesian Modelling Using WinBUGS, Wiley

Purpose The purpose of Bayesian modelling is to predict the response of a statistical process as a function of parameters and variables. Note both parameters and variables may vary. Bayesian models differ from frequentist models in the use of a prior distribution in addition to likelihood. The result of the analysis is a posterior distribution that is the best explanation for the data in light of the prior.

Basic Bayesian Modelling Direct computation of the posterior distribution from the prior distribution and the likelihood function representing the data. This is particularly efficient when the prior and posterior distributions are in the same family. For example, if the prior and likelihood are Gaussian, so will be the posterior. Unfortunately, complex processes cannot be analysed this way, and some sort of sampling approach is needed.

Markov Chain Monte Carlo (MCMC) Methods Invented originally by Metropolis et al., and later developed by Hastings. Under fairly unrestrictive conditions, a Markov chain will converge to an equilibrium distribution. If that distribution is the probability distribution of interest, life is good, as the chain can be sampled. If the convergence is relatively fast, life is very good.

Gibbs Sampling Gibbs sampling is a variant on the Metropolis-Hastings algorithm that is particularly efficient. It defines the equilibrium distribution using the distribution of each variable and parameter conditional on the other variables and parameters. During an epoch (a cycle through the variables and parameters), each is resampled using the values of the remaining. Repeat, periodically recording the values (at an interval long enough that the autocorrelation of the process is minimised to produce pseudo-independence).

BUGS WinBUGS (or OpenBUGS) is a tool for doing Bayesian modelling of statistical distributions. It allows you to use Gibbs sampling to go from a series of relationships to a statistical model implied by those relationships. For systems too complicated to be modelled in the frequentist paradigm (see last lecture), it will give you answers. For systems that can be modelled in the frequentist paradigm, it gives very similar or identical answers.

Kéry’s Argument Kéry recommends this approach for six reasons: 1.Numerical tractability 2.Absence of asymptotics 3.Ease of error propagation 4.Formal framework for combining information 5.Intuition 6.Coherence

Numerical tractability Gibbs sampling can handle statistical models too complex to be fitted using classical statistics. When the model is fitted using classical statistics, Gibbs sampling gives answers close to those produced by classical methods.

Absence of asymptotics Classical inference using maximum likelihood is unbiased in the infinite limit. For small samples, classical inference is often biased. Gibbs sampling for small samples is unbiased.

Ease of error propagation In classical statistics, measuring the distribution of parameters often requires approximation methods. In Gibbs sampling, the distribution of parameters can be sampled and reported directly.

Formal framework for combining information Bayesian methods define a theoretically correct approach for fusing data with existing knowledge of the process being studied.

Intuition Bayesian probability is concerned with calculating the distribution of parameters in the model, not with being able to reject null hypotheses on the data.

Coherence Bayesian statistics is much simpler in concept than classical statistics.

Evaluation of Results Suppose you have done your Gibbs sampling and you have a series of pseudo-random numbers from what appears to be the joint posterior distribution. How do you check it? –Verify the numbers come from what appears to be a stationary distribution. –Verify that the numbers are not auto-correlated. –Verify that the results are robust.

Worked Example 1 Using BUGS, demonstrate how to do the analysis.

Worked Example 2 Running BUGS from R.