1. Outline National Assessment of Educational Progress (NAEP) Multivariate Design Problem Implications for analysis Example with similar structure in Biostatistics

2. NAEP On-going surveys at national and state levels 4th, 8th, and 12th grade students and their teachers math, reading, writing background demographic and educational environment questions

3. Excellent web site http://www.nces.ed.gov/nationsreportcard/

4. NAEP Mathematics Mathematics –5 domains/sub-scales/traits/latent proficiencies –Algebra –Geometry Several hundred potential test questions

5. NAEP Objectives and Constraints Goal is population estimates Individual students (and schools) are NOT rated based on NAEP 45 minutes for cognitive questions (items) 15 minutes for background questions/administration

6. Matrix Sampling (1984) AlgebraGeometry

7. Model for Cognitive Data Longitudinal data model – : Student algebra proficiency – : Student geometry proficiency IRT (Item response theory)

8. Design Issue Fixed number of items per student –How many algebra items? –How many geometry items? Obtain (equally) accurate population estimates of both algebra and geometry proficiencies

9. Balanced Designs Give each student approximately the same number of algebra and geometry items Up to 5 or 6 sub-domains, so the number of items per sub-domain is very small Extended collections of related items may make a balanced design infeasible

10. Split Designs (symmetric) Some students assigned only algebra items Same number of other students assigned only geometry items Remaining students are assigned equal number of algebra and geometry items

11. Optimal Design and Estimation The balanced design is optimal Maximum likelihood estimation –The joint MLE for the algebra distribution and the geometry distribution is the same as the univariate MLE with the geometry and algebra proficiencies estimated separately

12. Balanced design –There is no gain from multivariate estimation –Estimates for individual student proficiencies are much improved by multivariate estimation Split design –Multivariate estimation is much better than univariate –Multivariate estimation for the split design approaches balanced design efficiency as the proficiency correlation approaches 1

13. Bivariate outcomes (Jessica Mancuso) Experimental biomarker for stroke patients –Measurement error –It can be applied to the infarct and non-infarct sides of the brain –Anticipated that the non-infarct side of the brain will be predictive of the infarct side –Evaluate an oral compound using the biomarker

14. Study design Placebo/drug in parallel blinded randomized groups Measurements –Baseline –On-dosing measurements (longitudinal) –Measurements on the infarct and non-infarct sides of the brain (bivariate)

15. Estimation The primary goal is to estimate the treatment effect on the infarct side of the brain What is the role of the measurements on the non-infarct side in the primary estimation?

16. Depends on other information If there is no effect (or a known effect) on the non-infarct side of the brain, the non- infarct data can improve estimation –Baseline non-infarct measurement may be very helpful –If the treatment does not effect the non-infarct side of the brain, the on-dosing measurement(s) are like covariates and may improve estimation

17. Depends on design Balanced design –Both sides of brain measured each time –No planned or unplanned missing measurements –On-dosing non-infarct measurements do not contribute to estimation of the drug effect on the infarct side (mostly true) –Lack of contribution despite improvement in estimation for individual patients

18. Split design –At some on-dosing times, the non-infarct measurement is available but the infarct is not available –The on-dosing non-infarct data may contribute substantially

19. Summary The use of multiple outcomes to improve inference is very complex The fact that an outcome can be used to improve the estimation/prediction of another outcome at the level of an individual person is not sufficient