Eastern Michigan University Student Success Between Honors and Non-Honors Students: A Comparative Analysis Using Matching Case Method Bin Ning & Meng Chen Eastern Michigan University MIAIR, November 3, 2016

Outline Context Analysis and Results Model Evaluation Conclusion

Higher Education Context Student Success, frequently measured by retention and graduation rates for quantitative research purposes, is undoubtedly one of the most critical subjects in higher education Many institutions put great effort into improving student retention and graduation rates through a variety of programs Establishing or expanding honors program has become an important tool to improve overall student success Does honors program actually improve student success? The question remains unanswered

Institutional Context EMU: ~18,000 UG and 4,000 GR Sizes of entering FTIAC cohort fluctuated around 2,700 Carnegie: before 2016—master’s large; starting 2016—Doctoral R3 Relatively good 1st-year retention rate 72-77% Low 6-year graduation rate: 36-40% Fast expansion of honors programs from 875 (AY2012) to 1,686 (AY2016) As a primary result from the expansion, EMU expects the improvement of overall student success

EMU’s Investment in Honors College Staffing Acquisition of properties Major marketing campaign Financial Aid

About the Study In this study, we want to test the hypothesis: Whether students in honors program presents higher first-to-second year retention and six-year degree completion rates or not than non-honors students, after controlling other input factors such as high school GPA, ACT score, and other demographic variables

Research Questions Are higher 1st-year retention and 6-year graduation rates for honors students due to participation in honors college or students with better starting qualifications? Is there a significant difference on aforementioned rates between honors and non-honors students?

Dataset The study included approximately 13,000 records from six FTIAC cohorts between 2005 and 2010 For the purpose of conducting matching case analysis, records with missing values were removed A total of 9,436 valid individual records were used Missing value has to be manually removed. The model does not recognize the missing values.

Research Design Experimental Group: 951 FTIAC honors students were enrolled and stayed in honors college for at least 3 semesters Control Group: 8,485 non-honors FITAIC students

Direct Comparison Results Cohort N Retained after Year 1 Retention Rate Honors 951 930 97.8% Non-Honors 8,485 7,085 83.5% Cohort N Graduated in Six Years Graduation Rate Honors 951 794 71.8% Non-Honors 8,485 3,213 37.9%

Matching Case Comparison Propensity score matching Four matching variables High School GPA ACT Composite Score Ethnicity First-generation McNemar’s test McNemar: paired non-parametric test, similar to chi-square and fisher’s exact test (for small sample size). Strength: the concept is similar to paired t test.

Propensity Score Matching Propensity score matching is a statistical matching technique that one treatment case is matched with one or more control cases The matching can strengthen causal-effect arguments in quasi- experimental and observational data by reducing selection bias In this study, the propensity score matching was performed in R by using package: MatchIt Both“ exact” and “nearest” match methods have been selected and results have been compared

Matching Methods Exact Matching: matches each treated unit with a control unit that has exactly same value on each covariate Sub-classification Matching: distributions of the covariates are similar in each subclass Nearest Neighbor Matching: control units are closest in terms of distance measure Optimal Matching: minimizing the average absolute distance across all match pairs Genetic Matching: using computationally intensive genetic search algorithm to match Coarsened Exact Matching: matches on a covariate while maintaining the balances of other covariates

Exact Match HONORS~HIGHSCHOOL_GPA+ACT+ETHNICITY+FIRST_GENERATION (method="exact") Sample sizes: Non-Honors Honors All 8,485 951 Matched 362 329 Discarded 8,123 622

Nearest Match HONORS~HIGHSCHOOL_GPA+ACT+ETHNICITY+FIRST_GENERATION (method="nearest") Honors: non-honors=1:3 Sample sizes: Non-Honors Honors All 8,485 951 Matched 2,853 Unmatched 5,632 __

Retention Rates Graduation Rates Results Retention Rates Graduation Rates Honors= 96.96% Non-Honors= 81.49% Method= “exact” Honors= 97.79% Non-Honors= 78.27% Method= “nearest” Honors= 82.07% Non-Honors= 63.54% Method= “exact” Honors= 83.49% Non-Honors= 54.15% Method= “nearest”

McNemar’s Test McNemar’s test Statistical test used to determine consistency in responses across two paired variables Basically a paired version of Chi-square test McNemar’s test usually considered paired binary response data displayed in a 2X2 contingency table It can be extended to a 3X3 or higher square tables by expanding the test statistic to include the sum of values obtained from all possible pairs of 2X2 tables

McNemar’s Test (Retention $ “Exact” Method) McNemar's chi-squared = 264.45, df = 1, p-value < 2.2e-16 College Retained Non-Retained Honors 319 10 Non-Honors 295 67 McNemar’s test is frequently used in life sciences

McNemar’s Test (Graduation $ “Exact” Method) McNemar's chi-squared = 100, df = 1, p-value < 2.2e-16 College Graduated Non-Graduated Honors 270 59 Non-Honors 230 132

McNemar’s Test (Retention $ “Nearest” Method) McNemar's chi-squared = 2168.8, df = 1, p-value < 2.2e-16 College Retained Non-Retained Honors 930 21 Non-Honors 2,233 620

McNemar’s Test (Graduation $ “Nearest” Method) McNemar's chi-squared = 1130.3, df = 1, p-value < 2.2e-16 College Non-Honors Honors 794 157 1,545 1,308

Model Evaluation Mean Diff. 16.1 52.8 45.0 47.5 Data dependent -90.7 Distance 16.1 HIGHSCHOOL_GPA 52.8 ACT 45.0 American Indian/Alaskan Native 47.5 Asian -90.7 Black/African American 75.6 Hispanic/Latino 100.0 Native Hawaiian/Other Pacific Islander Race/Ethnicity Unknown -7.1 White 79.0 First-generation 44.0 Data dependent Lowest mean differences between groups Perfect balance improvement Percent improvement of mean difference between control and experiment group before and after matching Data for non-honors after match. Data dependent means match process depends on data distribution, both groups must have overlap in each variable; lowest mean difference should be realized after match; after match, the balance of distribution of two groups are similar

Model Evalution Data dependent Lowest mean differences between groups Propensity score = probability

Distribution of High School GPA Before match “Nearest” match “ Exact” match Non-Honors Honors Non-Honors Honors Non-Honors Honors

Distribution of ACT Composite Score Before match “Nearest” match “ Exact” match The difference even after match is due to sample size Non-Honors Honors Non-Honors Honors Non-Honors Honors

Conclusion There is a significant difference on both retention rates and graduation rates between honors and non-honors students Honors program can improve student retention and graduations rate Differences are significant and are independent from students’ starting qualifications

Future Considerations Other factors to be considered Gender Living arrangements Pell Scaled GPA (high school vigor)

Thoughts on Matching Case Method Pros Quasi-experimental design Strengthen causal-effect arguments Allow the control of input variables Open source software available Cons Increase of number of matching variables will drastically decrease the sample size