Bayesian Model Robust and Model Discrimination Designs William Li Operations and Management Science Department University of Minnesota (joint work with Chris Nachtsheim and Vincent Agtobo)

Outline Introduction Bayesian model-robust designs Bayesian model discrimination designs

Part I: Introduction Main objective: find designs that are efficient over a class of models –Model estimation: Are all models estimable? –Model discrimination: Can estimable models be discriminated? Brief literature review –Early work: Lauter (1974), Srivastava (1975), Cook and Nachtsheim (1982) –Cheng, Steinberg, and Sun (1999) –Li and Nachtsheim (2000) –Jones, Li, Nachtsheim, Ye (2006)

More literature review Bingham and Chipman (2002): Bayesian Hellinger distance Miller and Sitter (2005): the probability that the true model is identified Montgomery et al. (2005): application of new tools in model- robust designs Loeppky, Sitter, and Tang (2005): projection model space Jones, Li, Nachtsheim, Ye (2006): model-robust supersaturated designs

A general framework Li (2006): a review on model-robust designs Framework: three main elements –Model space: F={f 1, f 2, …, f u } –Criterion (e.g., EC, D-, EPD) –Candidates designs (e.g., orthogonal designs) Objective (rephrase): select an optimal design from candidate designs, such that it is optimal over all models in F, with respect to a criterion

Model spaces Srivastava (1975): search designs –F = {all effects of type (ii) + up to g effects of type (iii)} Sun (1993), Li and Nachtsheim (2000) –F g = {all main effects + up to g 2f interactions} Supersaturated designs –F = {any g out of m main effects} Loeppky, Sitter, and Tang (2005) –F g = {g out of m main effects + all 2f interactions}

Criteria and candidate designs Criteria –Bayesian model-robust criterion (related to EC and IC of LN) –Bayesian model discrimination criteria (related to EPD of Jones et al.) Candidate designs –Orthogonal designs –Optimal designs

Bayesian optimal designs Main elements –Prior distribution: p(  ) –Distribution of data: p(y |  ) –Utility function: U(d,  y) –Design space

Selected literature “Bayesian Experimental Design: A Review” – Chaloner and Verdinelli (1995) DuMouchel and Jones (1994): Bayesian D- optimal designs Jones, Lin, and Nachtsheim (2006): Bayesian supersaturated designs

Part II: Bayesian model-robust designs Focus: estimability of designs –Estimation capacity (EC): percentage of estimable models Model-robust designs: EC=100% –Information capacity (IC): average D-criterion value over all models Model space –LN (2000): main effects + g 2fi’s –Loeppky et al. (2006): g main effects + all 2fi’s among g factors

Bayesian criterion

Bayesian criterion for model- robust designs

Bayesian model-robust design Prior probabilities –Uniform prior –Hierarchical prior Chipman, Hamada, and Wu (1997) Bayesian model-robust (BMR) criterion Bayesian model-robust design (BMRD)

Design evaluations Evaluating existing orthogonal designs –12-, 16-, and 20-run designs (Sun, Li, and Ye, 2002) –Two model spaces –Compute BMR values and rank designs –Compare BMR ranks with generalized WLP ranks Generalized WLP: Deng and Tang (1999) Ranks for GWLP: given in Li, Lin, and Ye (2003)

Rank comparison plot (for 16*7 designs)

Design constructions Optimal designs –Balanced (equal # of +’s and –’s) CP algorithm of Li and Wu (1997) –General (unbalanced) optimal designs Coordinate-exchange algorithm of Meyer and Nachtsheim (1995)

Part III: Bayesian model discrimination designs Issues beyond model estimation –How well can estimable models be distinguished from each other? –If true model is known, is it fully aliased with other models through the design?

Criteria Atkinson and Fedorov (1975) EPD (expected prediction differnce) criterion (Jones et al. 2006)

Expected non-centrality parameter (ENCP) criterion

Bayesian EPD criterion

Design results Evaluating orthogonal designs –A comprehensive study of designs –Candidate designs: 12-, 16-, 20-run designs –Model space: both LN and the projected space of Loeppky et al. (2006) –Criteria: all model discrimination criteria (Bayesian and non-Bayesian) Constructing optimal designs –CP: balanced –Coordinate-exchange: general (unbalanced)

An example mEPD aEPD mAF aAF mENCP aENCP (n=16, m=5, g=2) 1-3 EC < 100% EC < 100% EC < 100%

THANK YOU! More information: