1. Predicting Success Using Placement Tests for STEM and Non STEM Math Courses
Michael Pilant, Texas A&M University
Jennifer Whitfield, Texas A&M University
Robert Hall, Texas A&M University
JoungDong Kim, Texas A&M University
March 15, 2019
Scottsdale, AZ

2. Brief History of Texas A&M Math Placement Exam (MPE)
MPE1 (STEM MPE) consists of 33 algorithmically generated questions covering Pre-Calculus concepts. It was developed, in house, by two lecturers. Subsequently, it was delivered commercially (Cengage->WebAssign->PlaceU).
Before an MPE cutoff was established, students could register for Calc I regardless of MPE score. Based on historical analysis, a cutoff of 22 was established.
Since fitting MPE data in range [0,33] to binary data [pass/fail], is not feasible, we used a logistic curve to relate the probability of passing vs MPE score.
To study retention, and the effect of cutoffs, we also study the cumulative logistic distribution function.
A summer prep/review program for the MPE was established in 2010
A minimum score of 22 on the MPE was made mandatory in 2011
MPE2 (NON-STEM MPE) was developed in 2015
The MPE was un-proctored until Summer 2018
March 15, 2019
Scottsdale, AZ

3. Before cutoff established, this shows the pass rate versus MPE score
A score of 22 on MPE corresponds to 70% passing, using linear regression.
Based on historical data, a cutoff score of MPE=22 was established for enrollment in
Calc I beginning in Fall 2011.
March 15, 2019
Scottsdale, AZ

4. After cutoff implemented, the number of students with MPE<22 is greatly reduced.
A score of 22 on MPE still corresponds to 70% passing, using linear regression.
The cumulative pass rate, shows the effect of changing the cutoff score (within the class).
March 15, 2019
Scottsdale, AZ

5. Modeling MPE using Logistic Distribution Function
Since Pass/Fail is binary data (Fail=0, Pass=1), a scatterplot of outcome (pass/fail) vs MPE score results in two horizontal lines. Fitting a curve is not feasible …
If we consider the number of students passing (or failing) for each MPE score, we are led to analyzing the probability of passiing, for each MPE score. Using actual data, this is modeled by the ratio of NP/(NP+NF). Curve is quite “noisy.”
If we want to consider the cumulative pass rate, that is the probability of passing the class with an MPE score greater than or equal to a given score, S, we get the integral of the distribution, which is much smoother (until one reaches the Max MPE=33).
March 15, 2019
Scottsdale, AZ

6. Calc I Retention Rates, Fall 2000-Fall 2018
Algorithmic MPE
Course Changes
Cutoff Implemented
Increase of 18% over 18 years…
Summer PPP
Proctored
NSF STEP grants
March 15, 2019
Scottsdale, AZ

7. Development of MPE for NON-STEM Courses
Based on the success of the 33 question MPE for Calc I, a 33 question MPE for non-STEM courses was developed. It was increased to 39 questions in 2017.
There are two finite math courses (Math 140, 141, 166), two applied calculus (Math 131 and 142), a survey course (Math 167).
Major differences:
STEM courses Calc I, II, III have common exams, non-STEM courses do not.
Calc I has well-defined content, developing an MPE for both Finite Math and Applied Calculus is challenging.
Non-STEM courses can be taken anytime during the students career, and not necessarily in any order (part of core currriculum). Often students wait year they graduate to take math courses!
March 15, 2019
Scottsdale, AZ

8. Non-STEM course retention rates vs MPE2
Math 141 is a Finite Math course
A cutoff of 11 (out of 22) was established for MPE2.
This corresponds to a pass rate of 70% on the regression line.
March 15, 2019
Scottsdale, AZ

9. Non-STEM course retention rates vs MPE2
Math 131 is a Calculus Concepts course
A cutoff of 11 (out of 22) was established for MPE2.
This corresponds to a pass rate of 70% on the regression line.
March 15, 2019
Scottsdale, AZ

10. Exam1 Scores in Calc I
MPE33 is the 33 question version of MPE1 (for STEM).
For this data set, it was un-proctored.
The correlation between exam1 and MPE33 is 0.37
(R^2=0.138)
March 15, 2019
Scottsdale, AZ

11. Exam1 Scores in Calc I
MPE16 is the 16 question version of MPE1 (for STEM). It was administered during first week of class, and was proctored.
The correlation between exam1 and MPE16 is 0.491
(R^2=0.241)
March 15, 2019
Scottsdale, AZ

12. Exam1 Scores as an Early Warning Indicator
In a previous talk (ICTCM 2017), we analyzed the relationship between exam1, MPE16, MPE33 and final average.
Students were defined to be “at risk” if their first exam score was <70% or their computed MPE33 score (MPE16 * 33 / 16) was in the range [0,16]
The results of using MPE16 only, and MPE16 along with exam1 scores, are shown
In the following slides.
March 15, 2019
Scottsdale, AZ

13. (Given during first week of class)
Predicting At-Risk Students Using Only MPE16
(Given during first week of class)
Outcome
At-Risk
Not At-Risk
Total
Prediction
305
236
541
259
1221
1480
564
1457
2021
False Positive = 236/541 = 43.62%
False Negative = 259/1480 = 17.50%
Correctly Identified At Risk = 305/564 = 54.08%
Correctly Identified Not At-Risk = 1221/1457 = 83.80%
March 15, 2019
Scottsdale, AZ

14. Outcome Prediction Predicting At-Risk Students Using
MPE 16 and Exam 1 Scores (Week 6)
Outcome
At-Risk
Not At-Risk
Total
Prediction
386
142
528
178
1315
1493
564
1457
2021
False Positive = 142/528 = 26.89%
False Negative = 178/1493 = 11.92%
Correctly Identified At Risk = 386/528 = 68.44%
Correctly Identified Not At-Risk = 1315/1457 = 90.25%
March 15, 2019
Scottsdale, AZ

15. Proctored vs Non-Proctored Results
MPE vs SAT
Proctored
Non-Proctored
March 15, 2019
Scottsdale, AZ

16. Proctored vs Non-Proctored Results
Midterm Grades
March 15, 2019
Scottsdale, AZ

17. Summary For STEM course (Calc I and Calc II)
An MPE1 score of 22 out of 33 (on proctored exam) is required (or passing pre-Calculus course) for enrollment
Exams 1, 2, 3 are common exams, across all sections (except honors)
Exam 1 can be retaken if student scores below 70
Final Exam replaces lowest common exam score
Cutoff for C is now 67%, D is 57% (A, B unchanged at 90, 80)
Summer PPP is available (before first semester)
Week in Review and Help Sessions are available to all students
March 15, 2019
Scottsdale, AZ

18. Summary For non-STEM course:
An MPE2 score of 13 out of 26 is required for enrollment
Cutoff for C is now 67%, D is 57% (A, B unchanged at 90, 80)
No common exams
Week in Review and Help Sessions are available to all students
PPP equivalents being developed
Analog of “pre-calculus” for non-stem courses has been developed
March 15, 2019
Scottsdale, AZ