Predicting Success Using Placement Tests for STEM and Non STEM Math Courses Michael Pilant, Texas A&M University Jennifer Whitfield, Texas A&M University Robert Hall, Texas A&M University JoungDong Kim, Texas A&M University March 15, 2019 Scottsdale, AZ

Brief History of Texas A&M Math Placement Exam (MPE) MPE1 (STEM MPE) consists of 33 algorithmically generated questions covering Pre-Calculus concepts. It was developed, in house, by two lecturers. Subsequently, it was delivered commercially (Cengage->WebAssign->PlaceU). Before an MPE cutoff was established, students could register for Calc I regardless of MPE score. Based on historical analysis, a cutoff of 22 was established. Since fitting MPE data in range [0,33] to binary data [pass/fail], is not feasible, we used a logistic curve to relate the probability of passing vs MPE score. To study retention, and the effect of cutoffs, we also study the cumulative logistic distribution function. A summer prep/review program for the MPE was established in 2010 A minimum score of 22 on the MPE was made mandatory in 2011 MPE2 (NON-STEM MPE) was developed in 2015 The MPE was un-proctored until Summer 2018 March 15, 2019 Scottsdale, AZ

Before cutoff established, this shows the pass rate versus MPE score A score of 22 on MPE corresponds to 70% passing, using linear regression. Based on historical data, a cutoff score of MPE=22 was established for enrollment in Calc I beginning in Fall 2011. March 15, 2019 Scottsdale, AZ

After cutoff implemented, the number of students with MPE<22 is greatly reduced. A score of 22 on MPE still corresponds to 70% passing, using linear regression. The cumulative pass rate, shows the effect of changing the cutoff score (within the class). March 15, 2019 Scottsdale, AZ

Modeling MPE using Logistic Distribution Function Since Pass/Fail is binary data (Fail=0, Pass=1), a scatterplot of outcome (pass/fail) vs MPE score results in two horizontal lines. Fitting a curve is not feasible … If we consider the number of students passing (or failing) for each MPE score, we are led to analyzing the probability of passiing, for each MPE score. Using actual data, this is modeled by the ratio of NP/(NP+NF). Curve is quite “noisy.” If we want to consider the cumulative pass rate, that is the probability of passing the class with an MPE score greater than or equal to a given score, S, we get the integral of the distribution, which is much smoother (until one reaches the Max MPE=33). March 15, 2019 Scottsdale, AZ

Calc I Retention Rates, Fall 2000-Fall 2018 Algorithmic MPE Course Changes Cutoff Implemented Increase of 18% over 18 years… Summer PPP Proctored NSF STEP grants March 15, 2019 Scottsdale, AZ

Development of MPE for NON-STEM Courses Based on the success of the 33 question MPE for Calc I, a 33 question MPE for non-STEM courses was developed. It was increased to 39 questions in 2017. There are two finite math courses (Math 140, 141, 166), two applied calculus (Math 131 and 142), a survey course (Math 167). Major differences: STEM courses Calc I, II, III have common exams, non-STEM courses do not. Calc I has well-defined content, developing an MPE for both Finite Math and Applied Calculus is challenging. Non-STEM courses can be taken anytime during the students career, and not necessarily in any order (part of core currriculum). Often students wait year they graduate to take math courses! March 15, 2019 Scottsdale, AZ

Non-STEM course retention rates vs MPE2 Math 141 is a Finite Math course A cutoff of 11 (out of 22) was established for MPE2. This corresponds to a pass rate of 70% on the regression line. March 15, 2019 Scottsdale, AZ

Non-STEM course retention rates vs MPE2 Math 131 is a Calculus Concepts course A cutoff of 11 (out of 22) was established for MPE2. This corresponds to a pass rate of 70% on the regression line. March 15, 2019 Scottsdale, AZ

Exam1 Scores in Calc I MPE33 is the 33 question version of MPE1 (for STEM). For this data set, it was un-proctored. The correlation between exam1 and MPE33 is 0.37 (R^2=0.138) March 15, 2019 Scottsdale, AZ

Exam1 Scores in Calc I MPE16 is the 16 question version of MPE1 (for STEM). It was administered during first week of class, and was proctored. The correlation between exam1 and MPE16 is 0.491 (R^2=0.241) March 15, 2019 Scottsdale, AZ

Exam1 Scores as an Early Warning Indicator In a previous talk (ICTCM 2017), we analyzed the relationship between exam1, MPE16, MPE33 and final average. Students were defined to be “at risk” if their first exam score was <70% or their computed MPE33 score (MPE16 * 33 / 16) was in the range [0,16] The results of using MPE16 only, and MPE16 along with exam1 scores, are shown In the following slides. March 15, 2019 Scottsdale, AZ

(Given during first week of class) Predicting At-Risk Students Using Only MPE16 (Given during first week of class) Outcome At-Risk Not At-Risk Total Prediction 305 236 541 259 1221 1480 564 1457 2021 False Positive = 236/541 = 43.62% False Negative = 259/1480 = 17.50% Correctly Identified At Risk = 305/564 = 54.08% Correctly Identified Not At-Risk = 1221/1457 = 83.80% March 15, 2019 Scottsdale, AZ

Outcome Prediction Predicting At-Risk Students Using MPE 16 and Exam 1 Scores (Week 6) Outcome At-Risk Not At-Risk Total Prediction 386 142 528 178 1315 1493 564 1457 2021 False Positive = 142/528 = 26.89% False Negative = 178/1493 = 11.92% Correctly Identified At Risk = 386/528 = 68.44% Correctly Identified Not At-Risk = 1315/1457 = 90.25% March 15, 2019 Scottsdale, AZ

Proctored vs Non-Proctored Results MPE vs SAT Proctored Non-Proctored March 15, 2019 Scottsdale, AZ

Proctored vs Non-Proctored Results Midterm Grades March 15, 2019 Scottsdale, AZ

Summary For STEM course (Calc I and Calc II) An MPE1 score of 22 out of 33 (on proctored exam) is required (or passing pre-Calculus course) for enrollment Exams 1, 2, 3 are common exams, across all sections (except honors) Exam 1 can be retaken if student scores below 70 Final Exam replaces lowest common exam score Cutoff for C is now 67%, D is 57% (A, B unchanged at 90, 80) Summer PPP is available (before first semester) Week in Review and Help Sessions are available to all students March 15, 2019 Scottsdale, AZ

Summary For non-STEM course: An MPE2 score of 13 out of 26 is required for enrollment Cutoff for C is now 67%, D is 57% (A, B unchanged at 90, 80) No common exams Week in Review and Help Sessions are available to all students PPP equivalents being developed Analog of “pre-calculus” for non-stem courses has been developed March 15, 2019 Scottsdale, AZ