1. Propensity Score Matching Makes Program Evaluation Easy
Propensity Score Matching Makes Program Evaluation Easy! Robert M Roe, PhD and Emma A Gyasi, MS Executive Director of Institutional Research and Planning

2. Outline Challenges with program Evaluation Importance of Evaluation
What good IR/Assesement people do well
Method – Propensity score matching
Examples

3. Program Evaluation Challenges
Those who develop and implement intervention programs are not necessarily data knowledgeable
Since they are motivated to show programs work they tend to have biases:
Bias to search for information that confirms success of program and ignore information that does not
Bias to remember positive outcomes and forget negative outcomes
Often have a few anecdotal stories that show success (Alfred was a success!)

4. Role of Assessment/IR
Unless look critically at data, no way to know if program is effective
Our role is to add to the understanding of the program outcomes to:
Improve the program
Eliminate programs that don’t work
All in an objective way that can be understood by non data people

5. Program Evaluation
Need to compare students in some program, course, or intervention to an appropriate comparison group
Often comparison to overall student population is not appropriate
However it is usually not clear what the appropriate group is
That is where Propensity Scores come in

6. Caveat
If program Evaluation is positive, no one questions the methodology
If Evaluation is negative……..
Example

7. Evidence of Effectiveness No Evidence
Program Evaluation
Evidence of Effectiveness
No Evidence
Done
Question Methodology
Convinced?
Not Convinced?
Done
Suggest reasons why analyses are wrong and the program still works
Students did not take our advice
Maybe X has an impact?
Can you look at just females?
Can you look at last year?

8. Program Evaluation: Academic Empowerment Program (AEP)
Designed to help academically at risk students
AEP is basically a suggestion to take course AAD 102 – College Learning Strategies
The proposal for funding AEP defined success as:
AEP students will have higher retention, graduation, and GPA’s
Higher than ‘What’ or ‘Whom’ was not discussed

9. Academic Empowerment Program (AEP)
Eligibility requirement: HSGPA <=2.7 or ACT <= 18
Not all who are eligible participate:
17.5% of students are eligible
33% of eligible participate
Implies a simple analysis between eligible who participate and eligible who don’t
Analyses indicated a potential negative impact for the program

10. First Analysis Results
Persistence to 2nd Term
Persistence to 2nd Year
1st Term GPA
1st Year GPA
NON-AEP
86.9%
67.9%
2.28
2.46
AEP
85.0%
65.1%
2.20
2.33
Total
86.3%
67.0%
2.25
2.42

11. Response AEP administrators did not like results
Then informed us that students with other risk factors are chosen for the program (first gen, Pell, etc.)
Is this true?
Does the distribution of risk factors vary across groups?

12. Risk Factors At Risk ACT HSGPA Low Income 1st Gen Minority NON-AEP
18.85
2.96
26.8%
28.4%
29.0%
AEP
18.93
2.70
26.7%
25.6%
31.8%
Total
18.88
2.87
27.5%
30.0%

13. Risk Factors
The negative impact could be due to other risk factors and not AEP
Need to match those in AEP with those not in AEP by ACT, HSGPA, Low Income, First Generation and Minority

14. Propensity Score Matching (PSM)
PSM is a statistical matching technique that attempts to estimate the effect of a treatment by accounting for the covariates that predict receiving the treatment.
Match on the basis of the propensity score
Instead of attempting to create a match for each participant with exactly the same values of risk factors, we can instead match on the probability of participation:
Propensity = P(AEP|HSGPA, ACT, PELL, 1st Gen, Minority)

15. Steps Get data from treatment and potential comparison group
Run a logistic regression:
Response variable: treatment (AEP)or not (non AEP)
Independent variables: HSGPA, ACT, PELL, 1st Gen, Minority
Calculate propensity score for each case from the logit
Each case in the treatment group is matched with one in the comparison group on propensity score using: nearest neighbor matching (Full matching, Optimal matching)
Pairs are analyzed using paired t-test to determine impact of the treatment

16. Example of Propensity Match
Student
HSGPA
PELL
1st Gen
Minority
ps_t 1
2.76
0.407 12 2
2.66
0.422 13
2.67
0.415 3
3.09
0.172 14
3.17 4
2.57
0.520 15
2.59
0.495 5
2.77
0.378 16
2.75
0.381 6
2.63
0.443 17
2.73
0.428 7
2.65 18 8
2.54
0.507 19
2.58
0.479

17. PSM Results

18. Outcomes AEP No significant impact after propensity score matching
Next Question is why?
The only real intervention of AEP is a suggested participation in AAD College Learning Strategies
Recall the requirements are HSGPA <=2.7 or ACT <= 18

19. ACTUAL Course Description!
College Learning Strategies is a 16-week, two-credit course. The class meets twice a week for 50 minutes. Academic content is an integral component in these classes. Topics covered are: learning styles, critical thinking, life management, test taking, note taking, memorization techniques, and reading for increased comprehension. The second component is growth and development in appropriate behaviors for students in higher education. This area includes regular and punctual class attendance, life management assessment via use of time management logs, and integration of overall study skills. Combining these activities to the content requirements, this class is notably a process course coupled with the presentation of study techniques for individual experimentation and acquisition.

20. Is the Course Taught Consistently?
Grades vary dramatically by faculty member
Average Grade in course is a 2.72

21. AAD 102 Analysis
Grade
Count
Percentage A
298
19.4% A-
284
18.5% B+
145
9.5% B
167
10.9% B-
100
6.5% C+ 96
6.3% C
114
7.4% C- 64
4.2% D+ 39
2.5% D 54
3.5% D- 29
1.9% E
143
9.3%
Total Graded
1,533 W 74
4.6%
Total
1,607
21.4% receive a C- or lower while another 4.6% withdraw

22. OIR Outcome of AEP Analysis
After thorough analyses there is no evidence the program has an impact on retention, graduation, or GPA
Furthermore, AAD 102 has no impact (and potentially negative)
We should have started with PSM - with all of the prior analyses, OIR may have lost some credibility
Currently the AEP group is looking for other data to support the program (instead of revising it)

23. Library 197 (LIB 197)
The Dean of the Library believes students who take LIB 197 are significantly more likely to be retained, have higher GPAs, and graduate at a higher rate than those who do not take the course
If we simply look at those students who took the course vs everyone else, it indeed looks like this is the case

24. LIB 197
However, it turns out that students who take LIB 197 are better students in general (in terms of HSGPA and ACT scores)
Is the positive impact from the course or simply selection bias?
Propensity Score Matching (PSM)

25. Academic and Risk Factors Profile
Year
Status
High school GPA
ACT Composite score
Pell Eligible
1st Generation Status
Minority
2014
Non-Lib 197
3.33
22.7
15.3%
21.3%
16.5%
Lib 197
3.59
23.8
15.2%
19.2%
7.1%
All
3.34
16.3%
2015
3.37
22.8
16.2%
21.0%
19.9%
3.46
23.3
15.5%
21.6%
17.5%
22.9
19.8%
LIB 197 students have higher entering scores and fewer risk factors than the overall student body

26. PSM Results
Year
Status
1st Term GPA
1st Year GPA
Persist to 2nd Term
Persist to 2nd year
2014
Non-LIB 197
3.17
3.19
97.0%
83.7%
LIB 197
3.25
3.29
92.4%
2015
NON-LIB 197
3.12
3.05
92.6%
75.0%
3.07
2.95
95.6%
83.3%
2014: It appears LIB 197 students have slightly better performance (difference not significant at α=0.05)
2015: The trend appears to indicate non LIB 197 outperform LIB 197 students in terms of GPA, whiles LIB 197 performs slightly better in terms of persistence (difference not significant at α=0.05)

27. Outcomes LIB 197
Students who take LIB 197 tend to be more academically prepared than the rest of the entering class based on high school GPA and ACT scores.
Students who take LIB 197 tend to have fewer risk factors that the rest of the entering class.
Students who take LIB 197 tend to outperform the rest of the student body in terms of first term GPA, first year GPA, retention to second term, and retention to second year.
However, when compared to the appropriate comparison group, there are no significant differences in performance.

28. Summary
Propensity scores are a good solution to identifying appropriate control groups necessary for proper program analyses
Propensity scores are easy to describe to non data people
Using appropriate control groups increase confidence in outcomes (even when outcomes are negative)
Are retention and GPA the best outcome measures?

29. Thank You
Questions?