1. CMC3 South Conference October 15, 2016 Loris Fagioli, PhD
Multiple Measures Assessment Project (MMAP)
CMC3 South Conference
October 15, 2016
Loris Fagioli, PhD

4. The Facts

5. O = T + e Classical Test Theory Observed score = True score + error
N(0,1)
O = T + e
Observed score = True score + error
We are ok with error since it is normally distributed and 0 on average
- however, every test taker suggests error is inflated, and every test instructor thinks error is nonexistent
Since we cannot observe T or e, how can we get a sense of e?
- predictive validity with outcomes. The assumption is that T predicts GPA (assumption of perfect relationship=1)

6. O = T + e Classical Test Theory Observed score = True score + error
“[The woman at the test center] said, ‘It doesn’t matter
how you place. It’s just to see where you are.’ Looking
back, that’s not true. It’s really important.”
“I thought it was one of those tests that you take just to
see what kind of field they were going to recommend.
And then I found out it places you in classes.
O = T + e
Observed score = True score + error
“Normally I don’t really like to prepare for anything that
has to do with things like placement tests, because in a
way it feels like I’m cheating myself a little. I’m thinking,
‘Well, I didn’t know these concepts before the test, and
all of a sudden they tell me that I have a test coming
up. So let me prepare for it.’ And it feels like I’m sort of
cheating.”
CCC study
Venezia et al (2010). One-shot deal? WestEd

7. O = T + e Classical Test Theory ≠ Observed score = True score + error1
GPA/Achievement

8. Predictive validity of Tests (transfer level)
Correlation with Grade Point
Accuplacer – English
.10
Accuplacer – Math

9. Growing body of evidence
Weak relationship between assessment tests and college course outcomes if used as single predictor: bit.ly/CCRCAssessment
Incredible variability in cutscores; CCs often use HIGHER cutscores than 4-year: bit.ly/NAGB2012
Underestimates students of color, women, first generation college students, low SES: bit.ly/DefiningPromise
Long thread of research in the CCCs
Willett, Hayward, & Dahlstrom,
Hetts, Fuenmayor, & Rothstein,
Willett & Karanjeff,
The College Board agrees that the most successful placement models are those that take a comprehensive approach [with] multiple variables, including high school GPA …
e.g., Belfield & Crosta, 2012; Edgescombe, 2011; Scott-Clayton, 2012; Scott-Clayton & Rodriguez, 2012:
Hiss & Franks, 2014;
BC & Sierra – equivalent or high success rates among multiple measure students
SDCCD – doubling to tripling the rate of assessment into transfer-level among students selected for pilot.

10. Why Multiple Measures? ~ Multiple measures
Effort (HW, on time …)
Time management
Life-Work balance
Etc. 1
Multiple measures
provide a more complete picture of student ability
provide a way to increase the accuracy of placement, particularly reducing under-placement
are required by law

11. Predicting transfer-level grades Test vs GPA
Correlation with GPs
Accuplacer – English
.10
Accuplacer – Math
11th Grade GPA - English
.37
11th Grade GPA – Math
.38

13. Variables Explored in MMAP Models
High School Cumulative GPA
Grades in high school courses
CST scores
Advanced Placement course taking
Taking higher level courses (math)
Delay between HS and CCC (math)
Future:
Non-cognitive variables
Smarter Balance
Common Core
CCC Assess
etc.
The MMAP project took an empirical approach to finding predictors of success in CCC courses.

14. Example Decision Rules– 70% criterion
English Transfer
Math Transfer (Statistics)
HS 12 GPA >= 2.6
Minimum: Passed Algebra I (or higher)
AND
HS 12 GPA >= 3.0 OR
HS 12 GPA >= 2.6 AND Pre-Calculus C (or better)
Complete rules available here

15. Potential Statewide Impact
Projections for your college available here: bit.ly/MMAPProjections

16. Potential Impact on Equity (Math Transfer Level)

17. Example Decision Tree (transfer-level Statistics)
60%
70%
50%
Example Decision Tree (transfer-level Statistics)
Percent of sample in leaf
Success rate
Internal Node / Split
Leaf

18. Effects of different thresholds (transfer-level Statistics)
Criterion
Course Success rate
Statistics .8
85%
Statistics .7
78%
Statistics .6
Statistics .5
74%
% Eligible
8-15%
16-30%
17-30%
26-45%
Starting level Math
% complete transfer*
1 level below transfer
35%
2 levels below transfer
15%
3 levels below transfer 6%
* Completing Transfer-Level Math in 3 Years. Statewide data, Basic Skills Cohort Tracker, Fall 2009-Spring 2012

19. HSGPA and Transfer Level English Success Rates
SOCCCD

20. HSGPA and Transfer Level English Success Rates
SOCCCD

21. “Alternatives” to MMAP
Direct articulation with local high schools
Local strong multiple measures
State-wide MMAP with different thresholds
Self-placement (Moorpark, American River College*)

24. Grade Inflation Across Years State Mean Median SD N 2.64 2.69 .69
1,131,196
Zhang & Sanchez, 2014:

25. Grade Inflation Overall State Mean Median SD N 2.64 2.69 .69 1,131,196
Zhang & Sanchez, 2014:

27. Common Concerns about MMAP
Grade Inflation
Self-reported GPA not reliable
Students placed via MMs will not be successful
Our courses will have lower pass rates
Our test is different
Students would be better off in remedial coursework
We are only looking at GPA
Students will only get a “C” in transfer-level work
Students who get a “C” in transfer-level won’t be able to transfer
High school GPA is only good for recent graduates
What is disjunctive, conjunctive, and compensatory?
Next Steps

29. Availability of Cal-PASS+ data
Self-reported GPA
Availability of Cal-PASS+ data
Overall
Feeder
Local USD HS
Cal-PASS+ availability
Match
67.1%
83.5%
86.2%
Out of State/Missing
13.1%
Placement-Math
50.7%
63.3%
65.2%
Placement -English
55.0%
68.6%
70.6%

30. GPA vs. Self-reported HSGPA
College Board, 2009:
ACT, 2013:
HSGPA Level N
Mean HSGPA
Mean diff.
Actual
Self-reported
3.50–4.00
599
3.79
3.75
–0.04
3.00–3.49
451
3.24
3.23
–0.01
2.50–2.99
408
2.81
2.76
–0.05
2.00–2.49
265
2.24
2.35
0.11
1.50–1.99
172
1.77
2.04
0.27
0.00–1.49 85
1.03
1.85
0.82
Total
1,980
2.95
3.02
0.07
Under-reporting was 2-4X as common as over-reporting.

31. Using Self-Reported GPA from CCC Apply
New optional items included in Open CCCApply
Grade Point Average
Highest English Course Taken
Highest English Course Taken Grade
Highest Math Course Taken
Highest Math Course Taken Grade
Highest Math Course Passed
Highest Math Course Passed Grade
Need to opt-in!
Contact CCCAssess Product Manager, John Hadad, to opt-in
Requesting all MMAP pilot colleges to opt-in and share data with the MMAP team for validation

32. Success Story: Sierra College
Students will not be successful
Success Story: Sierra College

33. Success Story: Cañada College
N = 66
N = 170
N = 70
N = 116

34. Success Rate at Transfer Level
Success Story: Bakersfield
Success Rate at Transfer Level

35. Success Story: San Diego

36. Our tests are different – IVC Tests (correlations with transfer level grade point)
Uncorrected r
Corrected r
CTEP Grammar
.10
.16
CTEP Reading
.06
.09
CTEP Syntax
.08
.12
Math Assessment Test
.21
.29
11th Grade GPA - English
.37
.38
11th Grade GPA – Math
.39

37. Remedial courses are better for students
Efficiency Rates
3 levels below
% complete transfer
2 levels below
1 level below .9
48%
59%
73% .8
21%
33%
51% .7 8%
17%
34% .6 3%
22%
source: Hayward and Willett (2014)
Belfield & Crosta (2012): Given the frequency of underplacement, the poor predictive validity of assessment tests and the lack of positive outcomes for student placed into remediation, it would be statistically defensible and really quite reasonable to just put all students into transfer-level work.

38. We are only looking at GPA Will only get a “C” in transfer course

40. HSGPA and Transfer Level English Success Rates
SOCCCD

41. HSGPA and Transfer Level English Success Rates
SOCCCD

42. Will only get a “C” in transfer course
SOCCCD

43. Will only get a “C” in transfer course
Distribution of Statistics Node 8

44. Transfer-oriented students are better off in remediation than getting a “C” in transfer-level
Irvine Valley College, first course enrolled in, Spring 2000 to Fall 2011 who took an English course. N= 28,279, transfer within 4 years.

45. High school GPA is only good for recent graduates
Correlation between Predictor and 1st CC Math Grade
Semesters of Delay (approx. 6 months each)

46. High school GPA is only good for recent graduates
Correlation between Predictor and 1st CC English Grade
Semesters of Delay (approx. 6 months each)

48. Compensatory vs Disjunctive (Math Transfer Projections)
Craig analysis

49. Some lessons learned (pilot colleges)
MMAP rules performing as expected
Implementation of MM rules is nuanced, requiring careful compliance with details
Communication to students should specify which math classes they are recommended for
Carful consideration if communication of placement includes MMAP and Test information.
Projected impact of statewide MMAP rules based on scenario of little or no existing MM policies
Outreach to local high schools
Change takes time, or does it?

50. Retrospective Impact Analysis by College
4 graphics available for each college and overall based on disjunctive model for Math and English bit.ly/MMAPProjections
Overall
Overall with range
By Ethnicity
By Ethnicity with range
Range = Expected estimates vary depending on data availability
Limitations:
Does not include self-reported data
Based on students with Cal-PASSPlus data
Retrospective projections
Placement vs Enrollment
Loris -

51. Irvine Valley College Loris slide 1

52. Math and English Transfer Placement by Ethnicity - IVC
Loris

53. MMAP Prospective Projections South Orange County District (example)
Loris

54. Contacts Loris Fagioli John Hetts Educational Results Partnership
The RP Group
Mallory Newell
Terrence Willett
Craig Hayward
John Hetts
Educational Results Partnership
Ken Sorey
Daniel Lamoree
Peter Bahr University of Michigan ‎