1. Getting Started with Large Scale Datasets Dr. Joni M. Lakin Dr. Margaret Ross Dr. Yi Han

2. Presentation Files Are Available: http://www.auburn.edu/~jml0035/ (Under “Conference materials and resources” at the bottom of the page)

3. Opening questions How many of you primarily use SPSS for data analysis? How many are comfortable with using syntax (in SPSS or other programs)? How many already have plans to use a specific dataset? How many just curious about what’s available?

4. What Data is Available? Dr. Yi Han

5. U.S. National Datasets NCES

6. U.S. National Datasets Restricted use licenses http://nces.ed.gov/nationsreportcard/researchcenter/license.aspx

8. International Datasets

9. PISAPIAAC

10. Accessing Data and Getting Started Dr. Margaret Ross See PDFs

11. Key Issues in Working with Large Datasets Dr. Joni Lakin

12. Key issues 1. Statistical weighting in SPSS 2. Practical significance and large samples 3. Matrix sampling 4. Plausible values SPSS skills that make working with large datasets easier: 5. Keeping and managing syntax 6. Merging datasets 7. Checking for duplicate cases 8. Missing data imputation

13. 1. Statistical weighting in SPSS Weights allow us to better approximate the full population If African American students are 18% of population but 9% of my sample, I could weight each AA student 2.0 (so each observation is included twice in analyses) to get results that better reflect population-level effects. Types of weights Scale weights = multiplies observations to create a weighted sample of same size as population Proportional weights = may be below 1 to keep overall sample size the same as the sample Note When you’re reporting results, you can report weighted sample size, but you should also report unweighted sample sizes too

14. Using weights These “weight” values are already in large datasets

15. ELS:2002 Race UNWEIGHTED ELS:2002 Race WEIGHTED Freq.% Amer. Indian/Alaska Native130.8 Asian, Hawaii/Pac. Islander14609.0 Black or African American202012.5 Hispanic, no race specified9966.1 Hispanic, race specified12217.5 More than one race7354.5 White, non-Hispanic868253.6 Total16197100.0 Freq.% Amer. Indian/Alaska Native327811.0 Asian, Hawaii/Pac. Islander1425184.2 Black or African American49132114.4 Hispanic, no race specified2436077.1 Hispanic, race specified2986488.8 More than one race1478964.3 White, non-Hispanic205410360.2 Total3410873100.0

16. 2. Practical significance and large datasets Because of large sample size, many negligible effects (and ALL correlations) will be significant Must consider effect sizes and practical significance ELS:2002 variables Independent Samples Test tdfSig. Math test score 8.718593<.001 Reading test score -4.148593<.001 Mathematics self-efficacy 14.658593<.001 English self-efficacy scale -2.198593.029 Wow!! All significant!!

17. Practical significance and large datasets Actually negligible differences for reading and small differences for math ELS:2002 variables Independent Samples Test tdfSig.Cohen’s d Math test score 8.718593<.001 0.19 Reading test score -4.148593<.001 -0.09 Mathematics self-efficacy 14.658593<.001 0.32 English self-efficacy scale -2.198593.029 -0.05

18. 3. Matrix sampling (be aware of…) Used in large-scale assessments when Large domain being sampled (e.g., world history) Need to cover many topics in limited time Individual estimates of the constructs are less important than aggregate estimates (state level achievement) Usually requires IRT (item response theory) scoring methods to allow for comparable scores across examinees completing different items Table from von Davier et al., http://www.ierinstitute.org/fileadmin/Documents/IERI_Monograph/IERI_Monograph_Volume_02_Chapter_01.pdf http://www.ierinstitute.org/fileadmin/Documents/IERI_Monograph/IERI_Monograph_Volume_02_Chapter_01.pdf

19. 4. Plausible values Can result from matrix sampling (with IRT models), bootstrapping, and missing data imputation In matrix sampling, individual estimates of skills are less reliable and plausible values better capture this error variance compared to single scores Results in multiple estimates of the student’s true score on the construct (will appear as multiple variables) Poor practice = averaging plausible values before analysis Produces biased estimates (von Davier et al., see notes) Better practice = using methods that analyze the different estimates together and produce standard error bars Refer to von Davier et al. link in notes

20. 5. Keeping and managing syntax From any command window, can select “Paste” Makes sure analyses start with the same data selections: Sample weights, split files, selecting relevant cases Good for keeping record of computed and recoded variables

21. 6. Merging datasets Add cases = add more participants’ data Add variables = add variables for same participants from another dataset

22. Merging datasets--Adding variables Have to exclude duplicate variables from one dataset Check that values are really identical (if not, change variable name) Use Key Variables to match cases

23. 7. Checking for duplicate cases

24. Duplicate cases output Will appear as a new variable “PrimaryLast” Will need to decide how to handle on case-by-case basis Merging datasets incorrectly can result in duplicates If variables are identical, delete one If variables are different, check that identification variables are correct

25. 8. Missing data Methods that bias results: Mean substitution, listwise or pairwise deletion Methods that can provide less biased estimates Single imputation regression (better than above, but restricts variability) Expectation-maximization (EM)—best of SPSS options, works well when data is missing at random Analyze  Missing Value Analysis Be sure to read up on “missing completely at random, missing at random”, and “missing not at random”

26. Other Resources Dr. Lakin

27. AERA Research Grants and Dissertation Grants “The program seeks to stimulate research on U.S. education issues using data from the large-scale, national and international data sets supported by the National Center for Education Statistics (NCES), NSF, and other federal agencies, and to increase the number of education researchers using these data sets.” Suggestions based on personal observations and the RFP: Must use a strong quasi-experimental design ( Schneider et al., Estimating Causal Effects: Using Experimental and Observational Designs ) Regression discontinuity, propensity score matching, etc. Bringing in new quantitative approaches for other fields also very appealing (economics, epidemiology, etc.) Check past grants to see which datasets are “neglected” (more recent datasets better) Prefer ideas that involve multiple datasets in meaningful research are more successful Analyses of international datasets have been more successful recently

28. Other opportunities IES Research Grants do fund secondary data analyses with Exploration grant goals (any subject area) http://ies.ed.gov/funding/ IES data training workshops http://ies.ed.gov/whatsnew/conferences/?cid=2 AERA annual meeting usually has data training events: PDC02: Analyzing NAEP Assessment Data with Plausible Values… PDC13: Advanced Analysis using Adult International Large Scale Assessment Databases PDC16: Using NAEP Data on the Web for Educational Policy Research Several on quantitative methods (including propensity scores) AERA Institute on Statistical Analysis for Education Policy (summer)Institute on Statistical Analysis for Education Policy IES/NCES hosts STATS-DC conferences and summer institutes to train researchers in using specific datasets

29. Q&A Presentation files are available from http://www.auburn.edu/~jml0035/ (Under “Conference materials and resources”)