1. Accountability Reporting for California Community Colleges
Patrick Perry
Vice Chancellor of Technology, Research, & Info. Systems
CCC Chancellors Office

2. Data Preamble
“Information is the currency of democracy.” -Thomas Jefferson
“Get your facts first, then you can distort them as you please.” -Mark Twain
“In the twenty-first century, whoever controls the screen controls consciousness, information and thought.” -Timothy Leary

3. The CCC System 109 campuses, 72 districts, all locally governed
2.6 million students (annual unduplicated)
1.1 million FTES (annual)
35% white; half over age 25; 70% part-time
No admissions requirements
$20/unit; 40% get fees waived
Highest participation rate of any CC system in US; 25% of all CC students are CCC

4. Participation (and Fees)

5. CCC Chancellor’s Office
Weak authority; powers vested locally
Unitary MIS data collection (1992)
Student, faculty, course, section, session, grade level detail
Data collected end of term, 3x/yr
Used for IPEDS, apportionment, accountability, research, online data mart

6. History of CCC Accountability
Simple reporting, fact books until 1998
1998: State provides $300m ongoing in exchange for accountability reporting
“Partnership for Excellence” was born
CCC developed report in isolation
CCC allowed to determine “adequate progress”
“Contingent funding” never triggered
Used 5 metrics to measure system and college-level performance

7. PFE Metrics Annual volume of transfers to CSU/UC
Annual volume of awards/certificates
Rate of successful course completions
Annual volume of Voc. Ed. Course completions
Annual volume of basic skills improvements (lower to higher level)
4 of 5 are volume metrics, only 1 rate

8. The State Said: Your metrics allow for no adequate college comparisons
Your method of determining “adequate progress” is suspicious
You only look good because you are growing
Partnership over (2001), but keep reporting, (until 2004)
we have to spend your money buying energy from Enron

9. What Happened Next
Gov. Gray Davis: recalled for spending money buying energy from Enron
Replaced by
“The Governator”

10. The Governator Likes Community Colleges
Comes from a country that has European “academic bifurcation” (Austria)-university vs trade paths
Attended Santa Monica Community College
Took ESL, PE, bookkeeping, micro/macroeconomics
Transferred to U. Wisconsin-Superior

11. And Arnold Said:
We shall haves deez accountabeelity seeztem for de community collegez.
A bill was passed to create the framework, and eventually the framework was enacted.
Named: Accountability Reporting for Community Colleges (ARCC).

12. Arnold Said:
There shall be no pay for performance, but there will be the ability to compare performance.

13. We Said:
Some metrics will be system only; others will be at college-level
College metrics will be rates (to mitigate size for comparison)
No rankings—we will compare colleges against their “peers”
No $$$=ARCC is a “dashboard” accountability report.

14. Arnold Said:
Colleges need to address their performance annually to the State.

15. We Said:
Colleges are more responsive to their local district Board; annual requirement to take local ARCC results to local Board and submit minutes to State
Colleges must submit 500 word response, which becomes a part of the final report.

16. Arnold Said:
The report shall be done in collaboration with the State, not in isolation.

17. We Said:
The Dep’t of Finance, Leg Analyst, and Secretary of Education shall be a part of the technical advisory committee (along with CCC researchers and stakeholders).
We will either succeed or fail together.
This was a really smart move.

18. ARCC The Model: Measures 4 areas with 13 metrics:
Student Progress & Achievement-Degree/Certificate/Transfer
Student Progress & Achievement-Vocational/Occupational/Workforce Dev.
Pre-collegiate improvement/basic skills/ESL
Participation
“Process” is not measured

19. Student Prog. & Achievement: Degree/Cert/Xfer
College:
Student Progress & Achievement Rate(s) (SPAR)
“30 units” Rate for SPAR cohort
1st year to 2nd year persistence rate
System:
Annual volume of transfers
Transfer Rate for 6-year cohort of FTF’s
Annual % of BA/BS grads at CSU/UC who attended a CCC

20. Student Prog. & Achievement: Voc/Occ/Wkforce Dev
College:
Successful Course Completion rate: vocational courses
System:
Annual volume of degrees/certificates by program
Increase in total personal income as a result of receiving degree/certificate

21. Precollegiate Improvement/Basic Skills/ESL
College:
Successful Course Completion rate: basic skills courses
ESL Improvement Rate
Basic Skills Improvement Rate
System:
Annual volume of basic skills improvements

22. Participation College: System: None yet…but coming.
Statewide Participation Rate (by demographic)

23. Major Advancements of ARCC
Creating a viable alternative to the GRS Rate for grad/transfer rate.
Finding transfers to private/out of state institutions.
Doing a wage study.
Geo-mapping district boundaries.
Creating peer groups.

24. Defining Grad/Transfer Rate
Student Progress & Achievement Rate (SPAR Rate)
IPEDS-GRS for 2-yr colleges stinks:
No part-timers
How do you define degree-seeking?
Tracking period too short
Outcomes counting methodology terrible
AA/AS/Cert counted before transfer
Transfer to 2-yr college is counted

25. SPAR Rate Defining the cohort:
Scrub “first-time” by checking against past records (CCC, UC, CSU, NSC)

26. SPAR Rate Define “degree-seeking” behaviorally for CC populations
Not by self-stated intent; this is a poor indicator
Behavior: did student ever attempt transfer/deg-applicable level math OR English (at any point in academic history)
Students don’t take this for “fun”

27. Defining Degree-Seeking Behaviorally
Separates out remedial students not yet at collegiate aptitude
Measure remedial progression to this threshold elsewhere
Creates common measurement “bar” of student aptitude between colleges
Same students measured=viable comparison

28. SPAR Rate-Unit Threshold
CCC provides a lot of CSU/UC remediation
Lots of students take transfer math/Eng and leave/take in summer
Should not count these as success or “our” student
Set minimum unit completed threshold (12) for cohort entrance
Any 12 units in 6 years anywhere in system

29. SPAR Denominator: First-Time (scrubbed)
Degree-seeking (at any point in 6 years, attempt transfer/degree applicable math or English)
12 units (in 6 years)
This represents about 40% of students in our system

30. SPAR Numerator Outcomes the State wants:
Earned an AA/AS/certificate; OR
Transfer: to a 4-yr institution; OR
Become “transfer-prepared”;OR
Completed 60 xferable units
Became “transfer-directed”:
Completed both xfer level math AND English
No double-counting, but any outcome counts
SPAR Rate=51%

31. Tracking Transfers SSN-level matches with CSU, UC
Nat’l Student Clearinghouse for private, proprietary, for-profit, out of state
Match 2x/yr, send all records since 1992
Update internal “xfer bucket”
Works great for cohort tracking
Needed method for “annual volume”

32. Tracking Transfers Annual Volume of Transfers
CSU/UC: they provide these figures based on their criteria
We didn’t want to redefine this
Private/Out of State: NSC “cross-section” cut method
Validated against CSU/UC xfers from NSC source
Added another 30% to annual volumes

33. 97-98
98-99
99-00
00-01
01-02
02-03
03-04
04-05
05-06
06-07
FTF → T R A N S F
MIN E 12
UNITS

34. Sector
01-02
02-03
03-04
04-05
05-06
06-07
CSU
50,473
50,746
48,321
53,695
52,642
54,391 UC
12,291
12,780
12,580
13,211
13,462
13,874
ISP
17,070
15,541
18,100
18,365
17,840
18,752
OOS
10,762
10,540
11,150
11,709
11,726
11,825
Total
90,596
89,607
90,151
96,980
95,670
98,842

35. Transfer: Sector of Choice
% to UC
% to CSU
% to Instate Private
% to Out of State
White
17.9%
60.7%
11.0%
10.4%
AfrAm
11.5%
51.2%
18.1%
19.2%
Hisp/Lat
15.1%
67.7%
12.1%
5.1%
Asian
37.0%
49.9%
9.2%
3.9%

36. Demography of Transfer
FTF Stdents
All Stdents
XFER-CSU
XFER-UC
XFER-ISP
XFER-OOS
AfrAm 9% 8% 5% 3%
11%
13%
Asian
12%
26% 7%
Hisp/
Latino
35%
29%
23%
16%
White
37%
40%
44%
55%

37. The Rise of The Phoenix 96-97 2,166 97-98 2,829 98-99 3,374 99-00
4,194
00-01
5,055
01-02
5,586
02-03
6,515
03-04
8,222
04-05
8,585
05-06
8,134
06-07
9,216

38. Who Transfers to Phoenix?
Ethnicity UC
CSU
Phoenix
Asian
29.3%
14.2%
4.6%
African American
2.4%
5.2%
16.8%
Hispanic/Latino
13.6%
23.8%
28.6%
White
39.1%
43.6%
37.5%

39. Wage Study
What was the economic value of the degrees (AA/AS/certificate) we were conferring?
Required data match with EDD
Had to pass a bill changing EDD code to allow match

40. Wage Study Take all degree recipients in a given year
Subtract out those still enrolled in a CCC
Subtract out those who transferred to a 4-yr institution
Match wage data 5 years before/after degree

41. Wage Study Separate out two groups:
Those with wages of basically zero before degree
Those with >$0 pre wage
The result: The Smoking Gun of Success

43. Mapping Districts
CC Districts in CA are legally defined, have own elections, pass own bonds
We did not have a district mapping for all 72 districts
So we couldn’t do district participation rates

44. Mapping Project Get a cheap copy of ESRI Suite
Collect all legal district boundary documents
Find cheap labor—no budget for this

46. Peer Grouping “Peers” historically have been locally defined:
My neighbor college
Other colleges with similar demography
Other colleges with similar size

47. Peer Grouping Taking peering to another level:
Peer on exogenous factors that predict the accountability metric’s outcome
Thus leaving the “endogenous” activity as the remaining variance
Cluster to create groups
We picked 6 clusters, with a min of 3 in a cluster
Each metric produces different factors, peers, clusters

48. Peer Grouping: Example
Peering the SPAR Rate:
109 rates as outcomes
Find data for all 109 that might predict outcomes/explain variance
Perform regression and other magical SPSS things
See how high you can get your R2

49. Finding Data
What might affect a grad/transfer rate on an institutional level?
Student academic preparedness levels
Socioeconomic status of students
First-gen status of students
Distance to nearest transfer institution
Student age/avg unit load

50. Finding Data
We had to create proxy indices for much of these (142 tried)
GIS system: geocode student zipcode/ZCTA
Census: lots of data to be crossed by zip/ZCTA
Create college “service areas” based on weighted zip/ZCTA values
Different than district legal boundaries

52. Finding Data The Killer Predictor “Bachelor Plus Index” a proxy for:
“Bachelor Plus Index”, or what % of service area population of college has a bachelor’s degree or higher
“Bachelor Plus Index” a proxy for:
First gen
Academic preparedness
Socioeconomic status
Distance to nearest transfer institution

53. Peering SPAR Rate Exogenous factors that predict SPAR Rate: R2 = .67
Bachelor Plus Index
% older students
% students in basic skills
R2 = .67
What’s left is implied institutional variance
Demo

54. Peering: What’s Bad
Its complex and somewhat confusing and labor intensive.
Colleges traditional notion of “peer” is shaken
Multiple peers for multiple metrics; can change every year
You could do well vs. State average, increasing over time, but last in your peer group

55. Peering: What’s Good Its complex and somewhat confusing
You will likely look good in some areas, OK in others, and low in others
Its not very likely anyone will be high or low in all 6 metrics
It eliminated rankings.

56. The ARCC Report Is almost 800 pages. Comes out every March.
Takes 4 PY’s to complete (about 6 months/yr)
Is generally regarded highly in CA academic and Legislative circles.
DOF and LAO and Sec. of Ed love it.
Local Trustees/Boards love it.

57. The ARCC Collaboration
Has brought the system more money:
$33 mil in basic skills
Increased noncredit reimbursement rates by $300/FTE
Has brought about trust between system and State stakeholders.
Has educated both sides tremendously.

58. No More “Girlie-Man” Accountability!