1. 1 Advances in the Construction of Efficient Stated Choice Experimental Designs John Rose 1 Michiel Bliemer 1,2 1 The University of Sydney, Australia 2 Delft University of Technology, The Netherlands

2. 2 Contents Efficient Designs Defined State of Practice in Experimental Design Efficient Designs for Stated Choice Experiments Bayesian Efficient Designs Example When is an Orthogonal Design Appropriate? How can I do this?

3. 3 Efficient Designs Defined

4. 4 What are efficient designs? Based on a design, a survey is composed and the outcomes of the survey are used to estimate the model parameters The more reliable the parameter estimates are (i.e., having small standard error), the more efficient the design is experimental design respondents 0 0 1 0 0 1 0 0 0 0 0 1 1 0 0 0 0 1 0 0 1 0 0 0 dataestimationresults

5. 5 What are efficient designs? The asymptotic variance-covariance (AVC) matrix is an approximation of the true variance-covariance matrix “Asymptotic” means –assuming a very large sample; or –assuming a large number of repetitions using a small sample The roots of the diagonals of the variance-covariance matrix denote the standard errors variance- covariance matrix where is the standard error of parameter

6. 6 Asymptotic variance-covariance matrix Efficient Design: –Generate a design that when applied in practice will likely yield standard errors that are as small as possible experimental design respondents 0 0 1 0 0 1 0 0 0 0 0 1 1 0 0 0 0 1 0 0 1 0 0 0 dataestimationAVC matrix

7. 7 State of Practice in Experimental Design

8. 8 State of Practice In linear regression models: variance-covariance matrix = where is the data or design If X is orthogonal, then The diagonal elements of will be made as large as possible and the off diagonals equal to zero If we take the inverse of the diagonal elements will be minimised whilst the off diagonals remain zero

9. 9 State of Practice In linear regression models: variance-covariance matrix = where is the data or design If the diagonal elements as small as possible… will be maximised And the zero off-diagonals suggest no multicolinearity

10. 10 State of Practice Question: Is the variance-covariance matrix of the logit model represented by ? Answer: Logit model Question: For discrete choice data, what type of econometric model do we typically employ?

11. 11 Efficient Designs for Stated Choice Experiments

12. 12 Efficient Designs and Logit Models The variance-covariance matrix for logit models is related to the log-likelihood of the model Note that: In estimation, given the (design) data X and the observations y, one aims to determine estimates such that is maximised [maximum likelihood estimation] When generating an experimental design, these parameter estimates are unknown The values of depend on the model used (MNL, NL, ML)

13. 13 The second derivatives of the log-likelihood gives the Fisher information matrix: The negative inverse of the Fisher information matrix yields the model variance-covariance matrix [Hessian matrix of second derivatives] Efficient Designs and Logit Models [negative inverse matrix]

14. 14 Efficient Designs and Logit Models Example: MNL model with generic parameters (McFadden, 1974) First derivative: Second derivative: with Note: y drops out!

15. 15 Efficient Designs and Logit Models Assuming that all responds observe the same choice situations, Therefore, the AVC matrix becomes: Example: MNL model with generic parameters (cont’d)

16. 16 Efficient Designs and Logit Models Example: MNL model with generic parameters (cont’d) 05101520253035404550 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 (sample size) “A design that yields 50% lower standard errors requires 4 x less respondents”

17. 17 standard error 0 10 20 30 40 50 standard error 0 10 20 30 40 50 Investing in more respondents Investing in better design (sample size) Efficient Designs and Logit Models

18. 18 Efficient Designs and Logit Models Numerical example MNL model: Priors: Design:

19. 19 Efficient Designs and Logit Models Sample size and designs

20. 20 Efficient Designs and Logit Models Design 1: Design 2: Which design is more efficient?

21. 21 Efficiency measures In order to assess the efficiency of different designs, several efficiency measures have been proposed The most widely used ones are: –D-error –A-error The lower the D-error or A-error, the more efficient the design [determinant of AVC matrix] [trace of AVC matrix] number of parameters (size of the matrix), used as a scaling factor for the efficiency measure

22. 22 Bayesian Efficient Designs

23. 23 Bayesian efficient designs Efficient design –Example: Find D-efficient design based on priors Bayesian efficient design –Example: Find Bayesian D-efficient design based on priors

24. 24 Bayesian efficiency measures Bayesian D-error = Bayesian efficiency is difficult to compute, it needs to evaluate a complex (multi-dimensional) integral However, it is nothing more than a simple average of D-errors: where are random draws from the distribution function (we take r = 1,…,R draws) Bayesian D-error ≈

25. 25 How to obtain priors? Prior parameter estimates can be obtained from: –the literature –pilot studies –focus groups –expert judgement If no prior information is available, what to do? 1.Create a design using zero priors or use an orthogonal design 2.Give design to 100% of respondents 1.Create a design using zero priors or use an orthogonal design 2.Give design to 10% of respondents 3.Estimate parameters, use as priors 4.Create efficient design 5.Give design to 90% of respondents

26. 26 Example

27. 27 Example Let S = 12

28. 28 Orthogonal Design Alternative AAlternative BAlternative C SABCABCAC 11280 80 0 2841881100 3881 4180 4640124060 5 40108161 6 40641120 710811241101 8880640 1 9 8168060 641 80121 1168088181 12 4184081 Alternative AAlternative BAlternative C ABCABCAC Alternative A A1....... B01...... C001..... Alternative B A0001.... B00001... C000001.. Alternative C A0000001. C 00000001 Constant1ABConstant2C1C2C3 Constant16.580-0.238-0.796-1.5045.8320.0920.805 A-0.2380.1320.040-0.053-0.271-0.1070.143 B-0.7960.0400.151-0.018-1.0070.1130.132 Constant2-1.504-0.053-0.0185.3910.4200.084-0.327 C15.832-0.271-1.0070.4209.170-2.656-0.109 C20.092-0.1070.1130.084-2.6564.2101.100 C30.8050.1430.132-0.327-0.1091.1003.722 D b -error = 1.058 N = 316.28

29. 29 Efficient Design D b -error = 0.6617 N = 158.22 Alternative AAlternative BAlternative C SABCABCAC 1104168080 2840124180 3 80104060 41240108181 56801240101 6 4168061 76811241 1 8880881 0 9681840100 1240680101 111041881120 881104161 Alternative AAlternative BAlternative C ABCABCAC Alternative A A1....... B-0.601...... C-0.3001..... Alternative B A-0.470.45-0.301.... B0.60-0.670-0.751... C-0.15000.4501.. Alternative C A-0.400.1500.070.150.301. C0000.1500-0.151 Constant1ABConstant2C1C2C3 Constant16.860-0.669-0.981-0.7736.3890.9790.081 A-0.6690.1190.136-0.069-0.745-0.1630.197 B-0.9810.1360.199-0.063-1.089-0.1720.244 Constant2-0.773-0.069-0.0632.0930.2600.250-0.185 C16.389-0.745-1.0890.2607.5300.232-0.242 C20.979-0.163-0.1720.2500.2321.985-0.019 C30.0810.1970.244-0.185-0.242-0.0192.699

30. 30 When is an orthogonal design appropriate?

31. 31 Example Let S = 12

32. 32 Orthogonal Design Alternative AAlternative BAlternative C SABCABCAC 11280 80 0 2841881100 3881 4180 4640124060 5 40108161 6 40641120 710811241101 8880640 1 9 8168060 641 80121 1168088181 12 4184081 Alternative AAlternative BAlternative C ABCABCAC Alternative A A1....... B01...... C001..... Alternative B A0001.... B00001... C000001.. Alternative C A0000001. C 00000001 Constant1ABConstant2C1C2C3 Constant1 2.9380.000-0.2812.313 0.000-0.750 A 0.0000.025-0.281 0.000 B -0.2810.0000.0470.000 Constant2 2.313-0.2810.0001.5000.000 0.750 C1 2.3130.000 2.9380.000 C2 0.000 1.5000.000 C3 -0.7500.000 0.7500.000 1.500 D b -error = 0.3572 N = ?

33. 33 So how can I do this?

34. 34

35. 35 Ngene Software

36. 36 Ngene Software