2. Far North Data Disaggregation Workshop September 2017
Small Sample Size
Far North Data Disaggregation Workshop September 2017

3. Sample Size is Important
Reliability, Accuracy, Power to Detect Meaningful Differences
Sample Size

4. Small Sample Size
Large sample sizes are needed for a statistic to be accurate and reliable, especially if its findings are to be extrapolated to a larger population or group of data.  The smaller your sample size, the more likely outliers -- unusual pieces of data -- are to skew your findings.

5. Example: Tiny CC Foster Youth Transfer Rate
2009 – 2010 cohort
Two Foster Youth students
Both transfer to 4-year schools
Is this reliable as a Tiny CC foster youth transfer rate?

6. What is too small? N = 30 N >= 5 for FERPA reporting (too small?)
to satisfy the central limit theorem
However, many critiques (Cohen, 1990; Chrakrapani, 2011)
N >= 5 for FERPA reporting (too small?)
what is the risk of unmasking?
Data perturbation for small groups
How many additional FY would it take for FY transfer rate in above example to go from above to below average, assuming overall transfer rate is 55%?

7. Small Sample Sizes and Statistical Significance
Smaller sample size
Larger Sample (n = 400); success = 50% PI 10
10.0%
0.20 20
25.0%
0.50 30
30.0%
0.60 40
32.5%
0.65 50
34.0%
0.68 75
37.3%
0.75
100
39.0%
0.78
Differences are only significant when success rate of smaller sample is equal to or less than the percentage in the table.
Significance calculator:

8. Small sample size No single minimum sample size.
Power to detect meaningful differences increases with larger sample size.

9. Small sample size No single minimum sample size.
Power to detect meaningful differences increases with larger sample size.

10. Small Sample Size: Considerations/Caveats
When comparing the proportion between 2 groups, an n < 10 in each group is basically never enough to make meaningful comparisons
An n > 100 in each group is generally enough to show that there is a meaningful difference between groups
Often, we are dealing with sample sizes between 10 and 100.
What should we do?

11. Solutions Dealing with small sample sizes is a recurrent problem.
How you deal with it depends on your project's timeline, data sources, and requirements
More data is better than less
Combine data across terms/years
Multivariate analysis not always best
Use alternative sources of evidence
“Although we do not have sufficient data to determine whether FY are disproportionately impacted, most studies have shown disproportionate impacts for FY in CC completion, so…”

12. Solutions
Consider qualitative research such as case studies and focus groups
Qualitative evidence can be powerful
Can be used alone or in combination with quantitative evidence
Consider combining categories
Ex: all small groups lumped into "other“
Focus on the larger categories
Comparing only three largest ethnic groups
Oversampling (for survey data)
Purposely sample disproportionately from undersized group(s)

13. Solutions Acknowledge sample size limitations
Sometimes there is not a clear-cut solution
A lack of a definitive answer should not preclude action
Continue collecting data…

14. Group Work: Is there a difference in success rates between FY and comparison group? Why or why not?
Foster Youth
Comparison Group
Cohort
# in Cohort
# Completed
Proportion PI
2005-6 10 2
0.20
100 50
0.50
2006-7 4 3
0.75
2007-8 9
0.44
2008-9 13
0.15 6
0.67 8 1
0.13

15. Group Work: Is there a difference in success rates between FY and comparison group? Why or why not?
Foster Youth
Comparison Group
Cohort
# in Cohort
# Completed
Proportion PI
2005-6 10 2
0.20
100 50
0.50
.40
2006-7 4 3
0.75
1.50
2007-8 9
0.44
.88
2008-9 13
0.15
.30 6
0.67
1.34 8 1
0.13
.26

16. Are FY doing better or worse than comparison group?
Completion Rate

17. Group Work: Is there a difference in success rates between FY and comparison group? Why or why not?
Foster Youth
Comparison Group
Cohort
# in Cohort
# Completed
Proportion PI
2005-6 10 2
0.20
100 50
0.50
.40
2006-7 4 3
0.75
1.50
2007-8 9
0.44
.88
2008-9 13
0.15
.30 6
0.67
1.34 8 1
0.13
.26
Total 16
0.32
600
300
.64

18. Additional Resources
A short, mostly non-technical review of issues around sample size. May be useful for general framing of the issues: 
A more technical review of the issue of small sample size. Has a nice glossary that might be worth incorporating and/or expanding: 
Short but sweet website that includes useful metaphors and links to online calculators (tools!) that people can use when exploring these issues: 
Online calculator that is more flexible and probably easier to use than the grid included in the paper: 
Check out Box 1 of this paper for some interesting terminology (e.g., "Winner's Curse"): 
Why there is not single “magic number” for sample size: