1. Multiple Measures Susan Barbitta Associate Director, Special Projects
North Carolina Community College System Office

2. Multiple Measures: NC Policy
HS Degree within 5 years
Unweighted high school GPA ≥ 2.6
4 math classes
The student must have graduated from a NC high school within the previous 5 years.
The 4th math class must have Algebra II as a prereq

3. NC Students: Credit Math Enrollment
Total Students
Enrolling
310,989
52,015
334,879
60,013
338,431
63,305
329,714
62,346
326,172
64,911
321,158
69,619
309,297
67,979
298,873
71,222
Enrolling
16.7%
17.9%
18.7%
18.9%
19.9%
21.7%
22.0%
23.8%
Dev Ed redesign in 2012, math prereqs reevaluated by all disciplines
,989 total number of s 52,015 enrolled in credit bearing math
, ,013
, ,305
, ,346
, ,911
, ,619
, ,979
Includes ALL students enrolled: first time, transfers, continuing students; whether need college-level math or not, and may have already taken in past—we don’t know the mix here.

4. NC Students College-Level Math: Attempts
*Includes ALL students enrolled: first time, transfers, continuing students; whether need college-level math or not, and may have already taken in past—we don’t know the mix here.
*This shows a steady increase in both the number and % of students enrolling in college level math during the academic year.
This increase is due in part to MM and in part to the DE redesign – students needing less DE
17%

5. NC Students: College Level Math Success
71,222
% represents students that are successful
,989 total number of s 52,015 enrolled in credit bearing math
, ,013
, ,305
, ,346
, ,911
, ,619
, ,979
Includes ALL students enrolled: first time, transfers, continuing students; whether need college-level math or not, and may have already taken in past—we don’t know the mix here.

6. Multiple Measures: NC Early Data
Fall 2013-Fall 2014
Success Rates
Self reported data from 12 colleges, large and small, that were early implementers of MM. No intentional interventions were in place. N= several hundred
Although the “sky is not falling” the data indicates that additional supports are needed for the students in the range. Overall includes all students, not just MM students.
Placement via MM, no additional academic supports

7. Multiple Measures: Research Study
Funded by Bill & Melinda Gates Foundation
6 NC Community Colleges
Target population: Students exempting developmental education with HS GPAs between
Focus: Gateway math and/or English
The student must have graduated from a NC high school within the previous 5 years.
The 4th math class must have Algebra II as a prereq

8. Colleges in Research Study
Campus
Approx. unduplicated curriculum head count (fall 2015)
Number of students served in target group through fall 2015
Multi-campus, urban college
21,000
782
Main campus and one satellite campus, rural
3,808 62
Multi-campus, rural
6,550
243
11,000 67
Main campus and satellite centers
2,694 28
Multi-campus, urban
25,000
770
DRAFT

9. MM Waiver Students Pre-Support
SP 14 & FA 14
This begins the evidence as to why we targeted the students

10. MM Waiver Students Pre-Support
SP 14 & FA 14
Based on: about 2500 students entering 4 NC CCs during spring 2014 and fall 2014 ; 44% target, 56% control
All students in
Overall for the
Semester cohorts—ft/pt

11. Multiple Measures Supports
Co-requisite approach: 4 Colleges
Supplemental Instruction Sessions: 1 College
Remediation Modules: 1 College
Based on: 3481 students entering 5 NC CCs during the 2014 AY; 44% (1548) target, 56% (1933) control
All students in
Overall for the
Semester cohorts—ft/pt

12. Research Questions Addressed
Is there a difference in outcomes between the target and control groups? 

Are target groups benefiting from the supports? 

Are there adjustments necessary to the design and/or implementation?
Coreq labs – design change – less work in NROC,
implementation change –
- better alignment of gateway course to coreq lab
- stats and precalc in same coreq lab (might change again)

13. Research Questions Addressed
Are there outcome differences by specific student groups, among the targeted student group, and are these the same differences seen with non-targeted students? 

Which student groups’ outcomes appear to be most impacted by the implementation strategies and related supports? 


14. Multiple Measures Supports….
Co-requisite Labs
Goal: Strengthen pre-requisite math skills and knowledge
1-credit labs
Multiple classes one lab
Instructor led
Class design
Lecture
Computer work
Group work
-All interventions are mandatory for students in the target group and optional for all others-
Labs required for all students within the target group
-Multiple classes feed into one lab
-taught by a curriculum instructor or a dev ed instructor
aligned dev ed with curriculum
didn’t want same class instr teaching the lab
fairly sustainable - student pays for one credit
-Some labs have similar characteristics of a corequisite model. Two of the major differences are that the gateway instructor does not teach the lab and multiple gateway classes, with different instructors, feed into one lab
Some labs are being taught similarly to a math freshman seminar – soft skills as they relate to math. This is primarily with the Quantitative Literacy math classes

15. Multiple Measures Supports ….
Remediation Modules
Goal: Refresh basic skills, improve pre-requisite knowledge, level the playing field
Uses on-line assessment and review tools
Pre-test [identify weaknesses], complete review [close knowledge gaps], post-test [ensure gaps closed]
One college took a very different route incorporating an on-line assessment and review process within their gateway courses; initially, this ran for the initial three weeks of the semester.
Refresh basic skills & improve pre-requisite knowledge; ensure confident & ready to take on the demands of their curriculum coursework
Distinctive in form and character relative to the other colleges:
Chose this strategy b/c wanted to focus on and ensure strength in basic skills, upfront and early. Also chose this approach because they are a small school. They didn’t think they would have enough target-group students to sufficiently fill lab classes.
Does not provide support throughout the semester, the academic assistance is front-loaded at the beginning of the semester

16. Co-requisite Support Labs
Outcomes, Target Group
Co-requisite Support Labs
This data is from one college in the study. We’ll see an aggregate shortly.
172 pre n=150, post n= pre n=84, post n=70
Success is ABC/ABCDFW
Precalc n=163
Statistics n = 91
This is data from CPCC

17. Outcomes, Target Group Remediation Modules
Each semester the n for the target group <20

18. MM Waiver Students Pre/Post-Support

19. MM Waiver Students Pre/Post-Support
Average grade
2.7
1.4
On average no change from pre to post in grade by FTIC who attempted math during their first term. Students earned a 2.3 on average. - BUT –during the pre intervention control group student grades were over an entire grade higher than the target groups. HOWEVER target students in post intervention attained an average grade of 1.9, one-half grade increase while the control group actually declined very slightly.
-RESULT – closing of the achievement gap between target and control students
-pre control group was 1.3 higher than target
-post control group was only .6 higher than target
***the increase in grade was pronounced and statistically significant for students with the lowest GPA ( )
when students with deeper academic needs are provided support, there is room for larger gains to be achieved; however, providing better well-prepared students with additional support does not result in large gains, as there is not as much room available for improvement.

20. MM Waiver Students Pre/Post-Support
Percentage of FTIC MM Students Attaining D, F, W in Targeted Math Course 22 17 13
Decline in D grades and withdraws for the target group. 10 12 6

21. MM Waiver Students Pre/Post-Support
Average GPA over all courses

22. MM Waiver Students Pre/Post-Support
There are so many more part-time students – this is affecting the number of credits accumulated.
If we disaggregate the data by spring term and fall term we can see that spring 2014, 2015, 2016 the # of credits accumulated has taken a slight down turn. However the same metric for fall entry students has remained relatively steady. The post data includes 2 spring cohorts (sp 15 and sp 16) while the pre data includes only one spring, thus exacerbating the downward effect on the post student outcomes.
Overall

23. Next Steps Evaluating the Multiple Measures Policy
Disaggregating the high school GPA by tenths
ACT/SAT
Refining the Developmental Education reform