1. Scottish Social Survey Network: Master Class 1 Data Analysis with Stata Dr Vernon Gayle and Dr Paul Lambert 23 rd January 2008, University of Stirling The SSSN is funded under Phase II of the ESRC Research Development Initiative

2. Handling coefficients 1)Some general issues (Some thoughts on statistical modelling in Stata, and some tricks and tips …) 2) Using Quasi-variance

3. Statistical Modelling Process Model formulation [make assumptions] Model fitting [quantify systematic relationships & random variation] (Model criticism) [review assumptions] Model interpretation [assess results] Davies and Dale, 1994 p.5

4. Building Models REMEMBER – Real data is much more messy, badly behaved (people do odd stuff), harder to interpret etc. than the data used in books and at workshops

5. Building Models Always be guided by substantive theory (the economists are good at this – but a bit rigid) Form of the outcome variable (or process) Main effects – more complicated models later Don’t use stepwise regression (stepwise, pr(.05):regress wage married children educ age) An example…

6. A regression model GHS Data Y = age left education (years) X Vars Female Social Class (Advantaged; Lower Supervisory; Semi-routine; Routine) Age (centred at 40)

7. Regression Estimates ABCDE Female-0.32-0.34-0.27 Age (40)-0.06 -0.05 Supervisory-1.83-1.85 Semi- Routine-1.98-1.88 Routine-2.40-2.33 Constant17.5217.517.7518.2218.54

8. Linear Regression Models 1 unit change in X leading to a  change in Y The  is consistent – minor insignificant random variation (survey data) As long as the X vars are uncorrelated (a classical regression assumption)

9. A logit model (non-linear) GHS Data Y = Graduate / Non Graduate X Vars Female Social Class (Advantaged; Lower Supervisory; Semi-routine; Routine) Age (centred at 40)

10. Estimates Logit (log scale) ABCDE Female-0.24-0.23-0.20 Age (40)-0.03 -0.04 Supervisory-1.46-1.52 Semi-Routine-1.82-1.87 Routine-2.65-2.70 Constant-0.90-0.80-0.39-0.68-0.04 Parameterization ??

11. Logit Model Estimates on a log scale The  estimates a shift from X 1 =0 to X 1 =1 leads to a change in the log odds of y=1 Even when the X vars are uncorrelated, including additional variables can lead to changes in  estimates The  estimates the effect given all other X vars in the model Fixed variance in the logit model(   / 3 )

12. Non-Linear Models Be sensible about how you parameterize them Be careful interpreting them… Don’t throw variables in like a ‘bull in a china shop’ Model checking – make sure you understand how the ‘left hand side’ (lhs) is working Some bad examples using SARs (large dataset with many significant X variables)

13. Reference A technical explanation of the issue is given in Davies, R.B. (1992) ‘Sample Enumeration Methods for Model Interpretation’ in P.G.M. van der Heijden, W. Jansen, B. Francis and G.U.H. Seeber (eds) Statistical Modelling, Elsevier.

14. A Few Tricks The outreg2 command was written by John Luke Gallup and appears in the Stata Tech. Bullet. #59 You can download outreg2 from within Stata Outputs regression results in a more ‘publishable’ format e.g. in a word document

15. A Few Tricks statsby – tells Stata to collect statistics for a command across a by list Attractive because it saves the data simply, and can be used in Graphs In our experience statsby can save a lot of manual editing of results You can re-run the models with small adjustments; subsequent operations such as graph generation can be better automated

16. Handling coefficients 1)Some general issues (Some thoughts on statistical modelling in Stata, and some tricks and tips …) 2) Using Quasi-variance

17. Using Quasi-variance to Communicate Sociological Results from Statistical Models Vernon Gayle & Paul S. Lambert University of Stirling Gayle and Lambert (2007) Sociology, 41(6):1191-1208.

18. “One of the useful things about mathematical and statistical models [of educational realities] is that, so long as one states the assumptions clearly and follows the rules correctly, one can obtain conclusions which are, in their own terms, beyond reproach. The awkward thing about these models is the snares they set for the casual user; the person who needs the conclusions, and perhaps also supplies the data, but is untrained in questioning the assumptions….

19. …What makes things more difficult is that, in trying to communicate with the casual user, the modeller is obliged to speak his or her language – to use familiar terms in an attempt to capture the essence of the model. It is hardly surprising that such an enterprise is fraught with difficulties, even when the attempt is genuinely one of honest communication rather than compliance with custom or even subtle indoctrination” (Goldstein 1993, p. 141).

20. A little biography (or narrative)… Since being at Centre for Applied Stats in 1998/9 I has been thinking about the issue of model presentation Done some work on Sample Enumeration Methods with Richard Davies Summer 2004 (with David Steele’s help) began to think about “quasi-variance” Summer 2006 began writing a paper with Paul Lambert

21. Statistical Models Statistical models offer an attractive way for sociological researchers to summarize patterns from social survey datasets They offer techniques to summarize the joint relative effects of several different variables in a research study This is achieved by estimating statistical values (‘parameters’ or ‘coefficient estimates’) that indicate the magnitude and direction of the effect of each explanatory variable The appropriate sociological interpretation of the parameter estimates from statistical models is by no means trivial

22. The Reference Category Problem In standard statistical models the effects of a categorical explanatory variable are assessed by comparison to one category (or level) that is set as a benchmark against which all other categories are compared The benchmark category is usually referred to as the ‘reference’ or ‘base’ category

23. The Reference Category Problem An example of Some English Government Office Regions 0 = North East of England ---------------------------------------------------------------- 1 = North West England 2 = Yorkshire & Humberside 3 = East Midlands 4 = West Midlands 5 = East of England

24. Government Office Region

25. 1234 BetaStandard Error Prob.95% Confidence Intervals No Higher qualifications - ---- Higher Qualifications 0.65 0.0056<.0010.640.66 Males - ---- Females -0.20 0.0041<.001-0.21-0.20 North East - ---- North West 0.09 0.0102<.0010.070.11 Yorkshire & Humberside 0.12 0.0107<.0010.100.14 East Midlands 0.15 0.0111<.0010.130.17 West Midlands 0.13 0.0106<.0010.110.15 East of England 0.32 0.0107<.0010.290.34 South East 0.36 0.0101<.0010.340.38 South West 0.26 0.0109<.0010.240.28 Inner London 0.17 0.0122<.0010.150.20 Outer London 0.27 0.0111<.0010.250.29 Constant 0.48 0.0090<.0010.460.50 Table 1: Logistic regression prediction that self-rated health is ‘good’ (Parameter estimates for model 1 )

26. BetaStandard Error Prob.95% Confidence Intervals North East----- North West0.090.070.11 Yorkshire & Humberside0.120.100.14

28. Conventional Confidence Intervals Since these confidence intervals overlap we might be beguiled into concluding that the two regions are not significantly different to each other However, this conclusion represents a common misinterpretation of regression estimates for categorical explanatory variables These confidence intervals are not estimates of the difference between the North West and Yorkshire and Humberside, but instead they indicate the difference between each category and the reference category (i.e. the North East) Critically, there is no confidence interval for the reference category because it is forced to equal zero

29. Formally Testing the Difference Between Parameters - The banana skin is here!

30. Standard Error of the Difference Variance North West (s.e. 2 ) Variance Yorkshire & Humberside (s.e. 2 ) Only Available in the variance covariance matrix

31. Table 2: Variance Covariance Matrix of Parameter Estimates for the Govt Office Region variable in Model 1 Column123456789 Row North West Yorkshire & Humberside East Midlands West Midlands East England South EastSouth WestInner London Outer London 1North West.00010483 2Yorkshire & Humberside.00007543.00011543 3East Midlands.00007543.00012312 4West Midlands.00007543.00011337 5East England.00007544.00007543.0001148 6South East.00007545.00007544.00007545.00010268 7South West.00007544.00007543.00007544.00007543.00007544.00007546.00011802 8Inner London.00007552.00007548.0000755.00007547.00007554.00007572.00007558.00015002 9Outer London.00007547.00007545.00007546.00007545.00007548.00007555.00007549.00007598.00012356 Covariance

32. Standard Error of the Difference Variance North West (s.e. 2 ) Variance Yorkshire & Humberside (s.e. 2 ) Only Available in the variance covariance matrix 0.0083 =

33. Formal Tests t = -0.03 / 0.0083 = -3.6 Wald  2 = (-0.03 /0.0083) 2 = 12.97; p =0.0003 Remember – earlier because the two sets of confidence intervals overlapped we could wrongly conclude that the two regions were not significantly different to each other

34. Comment Only the primary analyst who has the opportunity to make formal comparisons Reporting the matrix is seldom, if ever, feasible in paper-based publications In a model with q parameters there would, in general, be ½q (q-1) covariances to report

35. Firth’s Method (made simple) s.e. difference ≈

36. Table 1: Logistic regression prediction that self-rated health is ‘good’ (Parameter estimates for model 1, featuring conventional regression results, and quasi-variance statistics ) 12345 BetaStandard Error Prob.95% Confidence Intervals Quasi- Variance No Higher qualifications------ Higher Qualifications0.650.0056<.0010.640.66- Males------ Females-0.200.0041<.001-0.21-0.20- North East-----0.0000755 North West0.090.0102<.0010.070.11 0.0000294 Yorkshire & Humberside0.120.0107<.0010.100.14 0.0000400

37. Firth’s Method (made simple) s.e. difference ≈ 0.0083 = t = (0.09-0.12) / 0.0083 = -3.6 Wald  2 = (-.03 / 0.0083) 2 = 12.97; p =0.0003 These results are identical to the results calculated by the conventional method

38. The QV based ‘comparison intervals’ no longer overlap

39. Firth QV Calculator (on-line)

40. Table 2: Variance Covariance Matrix of Parameter Estimates for the Govt Office Region variable in Model 1 Column123456789 Row North WestYorkshire & Humberside East Midlands West Midlands East England South EastSouth WestInner London Outer London 1North West.00010483 2Yorkshire & Humberside.00007543.00011543 3East Midlands.00007543.00012312 4West Midlands.00007543.00011337 5East England.00007544.00007543.0001148 6South East.00007545.00007544.00007545.00010268 7South West.00007544.00007543.00007544.00007543.00007544.00007546.00011802 8Inner London.00007552.00007548.0000755.00007547.00007554.00007572.00007558.00015002 9Outer London.00007547.00007545.00007546.00007545.00007548.00007555.00007549.00007598.00012356

41. Information from the Variance-Covariance Matrix Entered into the Data Window (Model 1) 0 0 0.00010483 0 0.00007543 0.00011543 0 0.00007543 0.00007543 0.00012312 0 0.00007543 0.00007543 0.00007543 0.00011337 0 0.00007544 0.00007543 0.00007543 0.00007543 0.00011480 0 0.00007545 0.00007544 0.00007544 0.00007544 0.00007545 0.00010268 0 0.00007544 0.00007543 0.00007544 0.00007543 0.00007544 0.00007546 0.00011802 0 0.00007552 0.00007548 0.00007550 0.00007547 0.00007554 0.00007572 0.00007558 0.00015002 0 0.00007547 0.00007545 0.00007546 0.00007545 0.00007548 0.00007555 0.00007549 0.00007598 0.00012356

43. Conclusion – We should start using method Benefits Overcomes the reference category problem when presenting models Provides reliable results (even though based on an approximation) Easy(ish) to calculate Has extensions to other models Costs Extra Column in results Time convincing colleagues that this is a good thing

44. Conclusion – Why have we told you this… Categorical X vars are ubiquitous Interpretation of coefficients is critical to sociological analyses –Subtleties / slipperiness –Emphasis often on precision rather than communication (e.g. in economics)