1. A trial of incentives to attend adult literacy classes Carole Torgerson, Greg Brooks, Jeremy Miles, David Torgerson Classes randomised to incentive or no incentive. Outcome variable: number of sessions attended.

2. Classes randomised to incentive or no incentive. Two groups of 14 classes. Labeled “X” and “Y” in this data set. Blinded for analysis. Group X: 77 students Group Y: 86 students

3. Outcome variable: number of sessions attended.

4. Compare mean number of sessions ignoring clustering:. ttest sessions, by(group) Two-sample t test with equal variances ------------------------------------------------------------------------------ Group | Obs Mean Std. Err. Std. Dev. [95% Conf. Interval] ---------+-------------------------------------------------------------------- X | 70 6.685714.4177941 3.495516 5.852238 7.519191 Y | 82 5.280488.2991881 2.709263 4.685197 5.875778 ---------+-------------------------------------------------------------------- combined | 152 5.927632.2566817 3.164585 5.42048 6.434783 ---------+-------------------------------------------------------------------- diff | 1.405226.5037841.4097968 2.400656 ------------------------------------------------------------------------------ Degrees of freedom: 150 Ho: mean(X) - mean(Y) = diff = 0 Ha: diff 0 t = 2.7893 t = 2.7893 t = 2.7893 P |t| = 0.0060 P > t = 0.0030

5. Compare mean number of sessions ignoring clustering:. ttest sessions, by(group) Two-sample t test with equal variances ------------------------------------------------------------------------------ Group | Obs Mean Std. Err. Std. Dev. [95% Conf. Interval] ---------+-------------------------------------------------------------------- X | 70 6.685714.4177941 3.495516 5.852238 7.519191 Y | 82 5.280488.2991881 2.709263 4.685197 5.875778 ---------+-------------------------------------------------------------------- combined | 152 5.927632.2566817 3.164585 5.42048 6.434783 ---------+-------------------------------------------------------------------- diff | 1.405226.5037841.4097968 2.400656 ------------------------------------------------------------------------------ Degrees of freedom: 150 Ho: mean(X) - mean(Y) = diff = 0 Ha: diff 0 t = 2.7893 t = 2.7893 t = 2.7893 P |t| = 0.0060 P > t = 0.0030 Stata version 8.

6. Compare mean number of sessions ignoring clustering:. ttest sessions, by(group) Two-sample t test with equal variances ------------------------------------------------------------------------------ Group | Obs Mean Std. Err. Std. Dev. [95% Conf. Interval] ---------+-------------------------------------------------------------------- X | 70 6.685714.4177941 3.495516 5.852238 7.519191 Y | 82 5.280488.2991881 2.709263 4.685197 5.875778 ---------+-------------------------------------------------------------------- combined | 152 5.927632.2566817 3.164585 5.42048 6.434783 ---------+-------------------------------------------------------------------- diff | 1.405226.5037841.4097968 2.400656 ------------------------------------------------------------------------------ Degrees of freedom: 150 Ho: mean(X) - mean(Y) = diff = 0 Ha: diff 0 t = 2.7893 t = 2.7893 t = 2.7893 P |t| = 0.0060 P > t = 0.0030 P = 0.006 — a highly significant difference!

7. Compare mean number of sessions ignoring clustering:. ttest sessions, by(group) Two-sample t test with equal variances ------------------------------------------------------------------------------ Group | Obs Mean Std. Err. Std. Dev. [95% Conf. Interval] ---------+-------------------------------------------------------------------- X | 70 6.685714.4177941 3.495516 5.852238 7.519191 Y | 82 5.280488.2991881 2.709263 4.685197 5.875778 ---------+-------------------------------------------------------------------- combined | 152 5.927632.2566817 3.164585 5.42048 6.434783 ---------+-------------------------------------------------------------------- diff | 1.405226.5037841.4097968 2.400656 ------------------------------------------------------------------------------ Degrees of freedom: 150 Ho: mean(X) - mean(Y) = diff = 0 Ha: diff 0 t = 2.7893 t = 2.7893 t = 2.7893 P |t| = 0.0060 P > t = 0.0030 P = 0.006 — a highly significant difference! But it is wrong — it ignores the clustering!

8. Compare mean number of sessions ignoring clustering, regression:. regress sessions group Source | SS df MS Number of obs = 152 -------------+------------------------------ F( 1, 150) = 7.78 Model | 74.5694526 1 74.5694526 Prob > F = 0.0060 Residual | 1437.63449 150 9.58422997 R-squared = 0.0493 -------------+------------------------------ Adj R-squared = 0.0430 Total | 1512.20395 151 10.0145957 Root MSE = 3.0958 ------------------------------------------------------------------------------ sessions | Coef. Std. Err. t P>|t| [95% Conf. Interval] -------------+---------------------------------------------------------------- group | -1.405226.5037841 -2.79 0.006 -2.400656 -.4097968 _cons | 8.090941.8152001 9.93 0.000 6.480183 9.701699 ------------------------------------------------------------------------------ P = 0.006 — identical to two sample t method. It is still wrong — it ignores the clustering!

9. Compare mean number of sessions including clustering, two sample t method on cluster means:. ttest sessions, by(group) Two-sample t test with equal variances ------------------------------------------------------------------------------ Group | Obs Mean Std. Err. Std. Dev. [95% Conf. Interval] ---------+-------------------------------------------------------------------- 1 | 14 6.69932.7457716 2.790422 5.088178 8.310461 2 | 14 5.189229.3974616 1.487165 4.330565 6.047893 ---------+-------------------------------------------------------------------- combined | 28 5.944274.439363 2.32489 5.042776 6.845773 ---------+-------------------------------------------------------------------- diff | 1.510091.8450746 -.226985 3.247166 ------------------------------------------------------------------------------ Degrees of freedom: 26 Ho: mean(1) - mean(2) = diff = 0 Ha: diff 0 t = 1.7869 t = 1.7869 t = 1.7869 P |t| = 0.0856 P > t = 0.0428 P = 0. 0856 — not significant.

10. Compare mean number of sessions including clustering, two sample t method on cluster means:. ttest sessions, by(group) Two-sample t test with equal variances ------------------------------------------------------------------------------ Group | Obs Mean Std. Err. Std. Dev. [95% Conf. Interval] ---------+-------------------------------------------------------------------- 1 | 14 6.69932.7457716 2.790422 5.088178 8.310461 2 | 14 5.189229.3974616 1.487165 4.330565 6.047893 ---------+-------------------------------------------------------------------- combined | 28 5.944274.439363 2.32489 5.042776 6.845773 ---------+-------------------------------------------------------------------- diff | 1.510091.8450746 -.226985 3.247166 ------------------------------------------------------------------------------ Degrees of freedom: 26 Ho: mean(1) - mean(2) = diff = 0 Ha: diff 0 t = 1.7869 t = 1.7869 t = 1.7869 P |t| = 0.0856 P > t = 0.0428 P = 0. 0856 — not significant. Almost correct — it takes the data structure into account, but not the variation in class size.

11. Compare number of sessions including clustering, two sample t method on cluster means Almost correct — it takes the data structure into account, but not the variation in class size.

12. Compare mean number of sessions including clustering, regression method, weighted by class size:. regress session group [aweight=learner] (sum of wgt is 1.6300e+02) Source | SS df MS Number of obs = 28 -------------+------------------------------ F( 1, 26) = 2.77 Model | 13.3075302 1 13.3075302 Prob > F = 0.1082 Residual | 124.992713 26 4.80741204 R-squared = 0.0962 -------------+------------------------------ Adj R-squared = 0.0615 Total | 138.300243 27 5.12223123 Root MSE = 2.1926 ------------------------------------------------------------------------------ sessions | Coef. Std. Err. t P>|t| [95% Conf. Interval] -------------+---------------------------------------------------------------- group | -1.380902.8299839 -1.66 0.108 -3.086958.3251548 _cons | 8.00502 1.33388 6.00 0.000 5.26319 10.74685 ------------------------------------------------------------------------------ P = 0. 108 — not significant. Correct — it takes the data structure into account, including the variation in class size.

13. Compare individual number of sessions including clustering, robust standard error method (Huber-White-sandwich method):. regress sessions group, cluster(class) Regression with robust standard errors Number of obs = 152 F( 1, 27) = 2.79 Prob > F = 0.1062 R-squared = 0.0493 Number of clusters (class) = 28 Root MSE = 3.0958 ------------------------------------------------------------------------------ | Robust sessions | Coef. Std. Err. t P>|t| [95% Conf. Interval] -------------+---------------------------------------------------------------- group | -1.405226.8407909 -1.67 0.106 -3.130387.319934 _cons | 8.090941 1.535933 5.27 0.000 4.939466 11.24242 ------------------------------------------------------------------------------ P = 0.106 — not significant. Correct — it takes the data structure into account. Very similar estimate and P value to method using means.

14. Compare individual number of sessions including clustering, robust standard error method (Huber-White-sandwich method). Correct — it takes the data structure into account. Very similar estimate and P value to method using means.

15. Compare individual number of sessions including clustering, robust standard error method (Huber-White-sandwich method. Correct — it takes the data structure into account. Very similar estimate and P value to method using means. I can do that using SPSS. So what is the advantage?

16. Compare individual number of sessions including clustering, robust standard error method (Huber-White-sandwich method): Correct — it takes the data structure into account. Very similar estimate and P value to method using means. I can do that using SPSS. So what is the advantage? We can use subject-level covariates.

17. Mid-score = reading score before randomisation.

18. Compare individual number of sessions including clustering, robust standard error method, adjusting for mid-score:. regress sessions group midscl, cluster(class) Regression with robust standard errors Number of obs = 152 F( 2, 27) = 11.91 Prob > F = 0.0002 R-squared = 0.1956 Number of clusters (class) = 28 Root MSE = 2.8572 ------------------------------------------------------------------------------ | Robust sessions | Coef. Std. Err. t P>|t| [95% Conf. Interval] -------------+---------------------------------------------------------------- group | -1.533053.6128085 -2.50 0.019 -2.790433 -.2756742 midscl | -.049151.0104713 -4.69 0.000 -.0706363 -.0276658 _cons | 10.56678 1.304614 8.10 0.000 7.889936 13.24363 ------------------------------------------------------------------------------ P = 0.019 — significant. Correct — it takes the data structure into account.

19. Compare individual number of sessions including clustering, robust standard error method, adjusting for mid-score:. regress sessions group midscl, cluster(class) Regression with robust standard errors Number of obs = 152 F( 2, 27) = 11.91 Prob > F = 0.0002 R-squared = 0.1956 Number of clusters (class) = 28 Root MSE = 2.8572 ------------------------------------------------------------------------------ | Robust sessions | Coef. Std. Err. t P>|t| [95% Conf. Interval] -------------+---------------------------------------------------------------- group | -1.533053.6128085 -2.50 0.019 -2.790433 -.2756742 midscl | -.049151.0104713 -4.69 0.000 -.0706363 -.0276658 _cons | 10.56678 1.304614 8.10 0.000 7.889936 13.24363 ------------------------------------------------------------------------------ P = 0.019 — significant. Correct — it takes the data structure into account. Adjustment produces true significant difference.