Multiple Measures Susan Barbitta Associate Director, Special Projects North Carolina Community College System Office barbittas@nccommunitycolleges.edu

Multiple Measures: NC Policy HS Degree within 5 years Unweighted high school GPA ≥ 2.6 4 math classes The student must have graduated from a NC high school within the previous 5 years. The 4th math class must have Algebra II as a prereq

NC Students: Credit Math Enrollment Total Students Enrolling 2008-09 310,989 52,015 2009-10 334,879 60,013 2010-11 338,431 63,305 2011-12 329,714 62,346 2012-13 326,172 64,911 2013-14 321,158 69,619 2014-15 309,297 67,979 2015-16 298,873 71,222 Enrolling 16.7% 17.9% 18.7% 18.9% 19.9% 21.7% 22.0% 23.8% Dev Ed redesign in 2012, math prereqs reevaluated by all disciplines 2008-09 310,989 total number of s 52,015 enrolled in credit bearing math 2009-10 334,879 60,013 2010-11 338,431 63,305 2011-12 329,714 62,346 2012-13 326,172 64,911 2013-14 321,158 69,619 2014-15 309,297 67,979 Includes ALL students enrolled: first time, transfers, continuing students; whether need college-level math or not, and may have already taken in past—we don’t know the mix here.

NC Students College-Level Math: Attempts *Includes ALL students enrolled: first time, transfers, continuing students; whether need college-level math or not, and may have already taken in past—we don’t know the mix here. *This shows a steady increase in both the number and % of students enrolling in college level math during the academic year. This increase is due in part to MM and in part to the DE redesign – students needing less DE 17%

NC Students: College Level Math Success 71,222 % represents students that are successful 2008-09 310,989 total number of s 52,015 enrolled in credit bearing math 2009-10 334,879 60,013 2010-11 338,431 63,305 2011-12 329,714 62,346 2012-13 326,172 64,911 2013-14 321,158 69,619 2014-15 309,297 67,979 Includes ALL students enrolled: first time, transfers, continuing students; whether need college-level math or not, and may have already taken in past—we don’t know the mix here.

Multiple Measures: NC Early Data Fall 2013-Fall 2014 Success Rates Self reported data from 12 colleges, large and small, that were early implementers of MM. No intentional interventions were in place. N= several hundred Although the “sky is not falling” the data indicates that additional supports are needed for the students in the 2.6-3.0 range. Overall includes all students, not just MM students. Placement via MM, no additional academic supports

Multiple Measures: Research Study Funded by Bill & Melinda Gates Foundation 6 NC Community Colleges Target population: Students exempting developmental education with HS GPAs between 2.60 - 2.99 Focus: Gateway math and/or English The student must have graduated from a NC high school within the previous 5 years. The 4th math class must have Algebra II as a prereq

Colleges in Research Study Campus Approx. unduplicated curriculum head count (fall 2015) Number of students served in target group through fall 2015 Multi-campus, urban college 21,000 782 Main campus and one satellite campus, rural 3,808 62 Multi-campus, rural 6,550 243 11,000 67 Main campus and satellite centers 2,694 28 Multi-campus, urban 25,000 770 DRAFT

MM Waiver Students Pre-Support SP 14 & FA 14 This begins the evidence as to why we targeted the 2.6-3.0 students

MM Waiver Students Pre-Support SP 14 & FA 14 Based on: about 2500 students entering 4 NC CCs during spring 2014 and fall 2014 ; 44% target, 56% control All students in Overall for the Semester cohorts—ft/pt

Multiple Measures Supports Co-requisite approach: 4 Colleges Supplemental Instruction Sessions: 1 College Remediation Modules: 1 College Based on: 3481 students entering 5 NC CCs during the 2014 AY; 44% (1548) target, 56% (1933) control All students in Overall for the Semester cohorts—ft/pt

Research Questions Addressed Is there a difference in outcomes between the target and control groups? 
 Are target groups benefiting from the supports? 
 Are there adjustments necessary to the design and/or implementation? Coreq labs – design change – less work in NROC, implementation change – - better alignment of gateway course to coreq lab - stats and precalc in same coreq lab (might change again)

Research Questions Addressed Are there outcome differences by specific student groups, among the targeted student group, and are these the same differences seen with non-targeted students? 
 Which student groups’ outcomes appear to be most impacted by the implementation strategies and related supports? 


Multiple Measures Supports…. Co-requisite Labs Goal: Strengthen pre-requisite math skills and knowledge 1-credit labs Multiple classes one lab Instructor led Class design Lecture Computer work Group work -All interventions are mandatory for students in the target group and optional for all others- Labs required for all students within the target group -Multiple classes feed into one lab -taught by a curriculum instructor or a dev ed instructor aligned dev ed with curriculum didn’t want same class instr teaching the lab fairly sustainable - student pays for one credit -Some labs have similar characteristics of a corequisite model. Two of the major differences are that the gateway instructor does not teach the lab and multiple gateway classes, with different instructors, feed into one lab Some labs are being taught similarly to a math freshman seminar – soft skills as they relate to math. This is primarily with the Quantitative Literacy math classes

Multiple Measures Supports …. Remediation Modules Goal: Refresh basic skills, improve pre-requisite knowledge, level the playing field Uses on-line assessment and review tools Pre-test [identify weaknesses], complete review [close knowledge gaps], post-test [ensure gaps closed] One college took a very different route incorporating an on-line assessment and review process within their gateway courses; initially, this ran for the initial three weeks of the semester. Refresh basic skills & improve pre-requisite knowledge; ensure confident & ready to take on the demands of their curriculum coursework Distinctive in form and character relative to the other colleges: Chose this strategy b/c wanted to focus on and ensure strength in basic skills, upfront and early. Also chose this approach because they are a small school. They didn’t think they would have enough target-group students to sufficiently fill lab classes. Does not provide support throughout the semester, the academic assistance is front-loaded at the beginning of the semester

Co-requisite Support Labs Outcomes, Target Group Co-requisite Support Labs This data is from one college in the study. We’ll see an aggregate shortly. 172 pre n=150, post n=153 152 pre n=84, post n=70 Success is ABC/ABCDFW Precalc n=163 Statistics n = 91 This is data from CPCC

Outcomes, Target Group Remediation Modules Each semester the n for the target group <20

MM Waiver Students Pre/Post-Support

MM Waiver Students Pre/Post-Support Average grade 2.7 1.4 On average no change from pre to post in grade by FTIC who attempted math during their first term. Students earned a 2.3 on average. - BUT –during the pre intervention control group student grades were over an entire grade higher than the target groups. HOWEVER target students in post intervention attained an average grade of 1.9, one-half grade increase while the control group actually declined very slightly. -RESULT – closing of the achievement gap between target and control students -pre control group was 1.3 higher than target -post control group was only .6 higher than target ***the increase in grade was pronounced and statistically significant for students with the lowest GPA (2.6-2.799) when students with deeper academic needs are provided support, there is room for larger gains to be achieved; however, providing better well-prepared students with additional support does not result in large gains, as there is not as much room available for improvement.

MM Waiver Students Pre/Post-Support Percentage of FTIC MM Students Attaining D, F, W in Targeted Math Course 22 17 13 Decline in D grades and withdraws for the target group. 10 12 6

MM Waiver Students Pre/Post-Support Average GPA over all courses

MM Waiver Students Pre/Post-Support There are so many more part-time students – this is affecting the number of credits accumulated. If we disaggregate the data by spring term and fall term we can see that spring 2014, 2015, 2016 the # of credits accumulated has taken a slight down turn. However the same metric for fall entry students has remained relatively steady. The post data includes 2 spring cohorts (sp 15 and sp 16) while the pre data includes only one spring, thus exacerbating the downward effect on the post student outcomes. Overall

Next Steps Evaluating the Multiple Measures Policy Disaggregating the high school GPA by tenths ACT/SAT Refining the Developmental Education reform