1. Using Decision Trees to Predict Course Success in the Multiple Measures Assessment Pilot RP CONFERENCE APRIL 9, 2015

2. Your MMAP to Today's Presentation Overview of Multiple Measures Assessment Project (MMAP) Data sources for MMAP Why multiple measures? Why decision trees? Code for creating decision trees in R Impact of disjunctive application of MM rule sets Equity implications Implementation

3. Multiple Measures Assessment Project MMAP research team & MMAP Work Group Component of the Common Assessment Initiative Analyzing high school performance data to create models that predict success in CC English & math Also working on assessing predictive validity of noncognitive variables Focused on predictive validity and reducing inappropriate underplacement Webinars, summits, professional development

4. Tests alone have proven to have poor predictive validity MMS can reduce underplacement Provide a more holistic picture Multiple measures assessment is mandated Multiple measures have traditionally involved a few survey-type questions More of a nudge than a true determinant Why MMs?

5. Correlations with College Grade: 11 th grade Cumulative GPA vs. Accuplacer English11 th Grade GPAAccuplacer Transfer 0.270.10 1 level below 0.240.12 2 levels below 0.250.12 3 levels below 0.180.12 4 levels below 0.210.07 Math11 th Grade GPAAccuplacer Transfer - STEM 0.240.19 Transfer – Stats 0.310.16 Transfer – LAM 0.260.09 1 level below 0.280.21 2 levels below 0.260.11 3 levels below 0.230.11 4 levels below 0.190.05

6. Lessons From Other States A number of assessment reform challenges have been identified (RFA, 2013) Using data to inform the work: Developmental education student performance data helps make the case for system-wide reform. Using the data to educate about the need for developmental education reform is an essential first step in developing the political will to move forward. Including faculty in the process: Engaging faculty in every part of the work has been central to the successes experienced to date. Communicating across system colleges: Important to inform the field about the work that is taking place and also improve the results by incorporating feedback.

7. National legislative & policy context State-level ◦North Carolina – 2.6 GPA & 4 years of math  transfer-level ◦Florida – developmental educational optional ◦Virginia – re-design of assessment in, modularization of math ◦Connecticut – dev. education cut to 1 semester, coreq only ◦Hawaii CCs- pilots using HSGPA in assessment ◦Colorado, Montana, Mississippi* - move to corequisite models ◦Massachusetts – 2.7 GPA and coreq. for non-STEM math National level ◦2013: US Governmental Accountability Office report on Developmental Education in Community Colleges ◦2014: Center for the Analysis of Postsecondary Readiness established by CCRC, MDRC ◦2015: Foundation policy advocacy and funding priority

8. Data for MMAP Sources COMIS K-12 transcript data CST, EAP, ACCUPLACER, CAHSEE Structure High School (grade) Community College (CB21) 8

10. Pros and Cons of Decision Trees Strengths Weaknesses Visualization Easy to understand output Easy to code rules Model complex relationships easily Linearity, normality, not assumed Handles large data sets Can use categorical and numeric inputs Results dependent on training data set – can be unstable esp. with small N Can easily overfit data Out of sample predictions can be problematic Greedy method selects only ‘best’ predictor Must re-grow trees when adding new observations

11. A B Z Z A B B Z A A B A B A B AAABBB ZZZ Increasing Homogeneity with each split Branch Internal Node Leaf Node Root Node

12. How is homogeneity measured? Gini-Simpson Index If selecting two individual items randomly from a collection, what is the probability they are in different categories. The Gini coefficient is a measure of the inequality of a distribution, a value of 0 expressing total equality and a value of 1 maximal inequality. Shannon Information Index Measures diversity of a collection of items. Higher values indicate greater diversity.

13. Libraries and Code for R: Your Basic Classification Decision Tree Data <- read.csv(“C:/Folder/Document.csv", header=T) Data.df <- data.frame(Data) DataTL <- (subset (Data.df,EnglishLevel==“Transfer Level”)) library(rpart) library(rpart.plot) ctrl <- rpart.control(minsplit = 100, minbucket = 1, cp = 0.001) DataTransferLevel <- (subset (Data.df,CourseLevel==1)) fitTL <- rpart(formula = success ~ Delay + CBEDS_rank + course_gp + A2G + cst_ss + grade_level + GPA_sans, data=DataTL, method = "class", control = ctrl) printcp(fitTL) prp(fitTL) rsq.rpart(fitTL) print(fitTL) print(fitTL, minlength=0, spaces=2, digits= getOption("digits")) summary(fitTL) More code available at: http://bit.ly/MMAP_resources_2015http://bit.ly/MMAP_resources_2015

14. http://www-users.cs.umn.edu/~kumar/dmbook/ch4.pdf

15. Considerations Splitting criterion: how small should the leaves be? What are the minimum # of splits? Stopping criterion: when should one stop growing the branch of the tree? Pruning: avoiding overfitting of the tree and improving Understanding classification performance

16. Two Approaches to Avoid Overfitting Forward pruning: Stop growing the tree earlier. Stop splitting the nodes if the number of samples is too small to make reliable decisions. Stop if the proportion of samples from a single class (node purity) is larger than a given threshold Post-pruning: Allow overfit and then post-prune the tree. Estimation of errors and tree size to decide which subtree should be pruned.

17. Fit Statistics: Evaluating your tree Misclassification rate - the number of incorrect predictions divided by the total number of classifications. Sensitivity - the percentage of cases that actually experienced the outcome (e.g., "success") that were correctly predicted by the model (i.e., true positives). Specificity - the percentage of cases that did not experience the outcome (e.g., "unsuccessful") that were correctly predicted by the model (i.e., true negatives). Positive predictive value - the percentage of correctly predicted successful cases relative to the total number of cases predicted as being successful.

18. Variables that predict CC success English – Cumulative HS GPA – C+ or better in AP English class – Score on English CST Math – Cumulative HS GPA – Grades in Algebra II, Calculus, Statistics, Trigonometry, Geometry, Pre-calculus – Taking a challenging CST – Score on math CST – Delay

19. Decision tree predicting grade points in transfer-level English, non-direct matriculants 19

20. English transfer-level rule sets Direct matriculants 11th grade HS Cume GPA >= 2.7 OR C+ or better in AP English Non-direct matriculants 12th grade HS Cume GPA >= 2.7 OR C or better in AP English OR Cumulative GPA >= 2.3 & 12th Course GP >= B-

21. GE Math/Liberal Arts Math Decision Tree, direct matriculants

22. GE Math transfer-level rule sets Direct matriculants 11th grade Cume GPA >= 3.2 OR Cumulative GPA >= 2.5 & CST >= 288 & Algebra II C or better OR Cumulative GPA >= 2.9 & C or better in Algebra II Non-direct matriculants 12th grade Cume GPA >= 2.8 OR CST >= 284 & Took Trigonometry in high school OR Trigonometry B or better

25. Impact of applying math models

26. Impact of applying English models

27. Disproportionate Impact Renewed interest in equity across gender, ethnicity, age, disability, foster youth and veteran status Does a student’s demographics predict placement level? If so, what is the degree of impact and what can be done to mitigate?

29. Nominal impact on success rates

30. Impact on throughput

31. Timeline and feedback Pilot running now through fall 2015 Testing rule sets via retroactive “what ifs” Collecting Social-Psychological and other MMs Active implementations (some venues) Need to collect & analyze fall 2015 in winter/early spring of 2016 Fall 2016 system go-live deadline All feedback appreciated and encouraged

32. Contacts : Terrence Willett The RP Group twillett@rpgroup.org Craig Hayward The RP Group chayward@rpgroup.org Alyssa Nguyen The RP Group anguyen@rpgroup.org John Hetts Educational Results Partnership/CPP jhetts@edresults.org Ken Sorey Educational Results Partnership/CPP ken@edresults.org Danielle Duran Educational Results Partnership/CPP dduran@edresults.org Daniel Lamoree Educational Results Partnership/CPP dlamoree@edresults.org