2. 1 Accountability Reporting for California Community Colleges Patrick Perry Vice Chancellor of Technology, Research, & Info. Systems CCC Chancellors Office

3. 2 Data Preamble u “Information is the currency of democracy.” -Thomas Jefferson u “Get your facts first, then you can distort them as you please.” -Mark Twain u “In the twenty-first century, whoever controls the screen controls consciousness, information and thought.” -Timothy Leary

4. 3 The CCC System u 109 campuses, 72 districts, all locally governed u 2.6 million students (annual unduplicated) u 1.1 million FTES (annual) u 35% white; half over age 25; 70% part-time u No admissions requirements u $20/unit; 40% get fees waived u Highest participation rate of any CC system in US; 25% of all CC students are CCC

5. 4 Participation (and Fees)

6. 5 CCC Chancellor’s Office u Weak authority; powers vested locally u Unitary MIS data collection (1992) l Student, faculty, course, section, session, grade level detail l Data collected end of term, 3x/yr l Used for IPEDS, apportionment, accountability, research, online data mart

7. 6 History of CCC Accountability u Simple reporting, fact books until 1998 u 1998: State provides $300m ongoing in exchange for accountability reporting l “Partnership for Excellence” was born u CCC developed report in isolation u CCC allowed to determine “adequate progress” u “Contingent funding” never triggered l Used 5 metrics to measure system and college- level performance

8. 7 PFE Metrics u Annual volume of transfers to CSU/UC u Annual volume of awards/certificates u Rate of successful course completions u Annual volume of Voc. Ed. Course completions u Annual volume of basic skills improvements (lower to higher level) l 4 of 5 are volume metrics, only 1 rate

9. 8 The State Said: u Your metrics allow for no adequate college comparisons u Your method of determining “adequate progress” is suspicious u You only look good because you are growing u Partnership over (2001), but keep reporting, (until 2004) l we have to spend your money buying energy from Enron

10. 9 What Happened Next u Gov. Gray Davis: recalled for spending money buying energy from Enron u Replaced by “The Governator”

11. 10 The Governator u Likes Community Colleges l Comes from a country that has European “academic bifurcation” (Austria)- university vs trade paths l Attended Santa Monica Community College u Took ESL, PE, bookkeeping, micro/macroeconomics u Transferred to U. Wisconsin-Superior

12. 11 And Arnold Said: u We shall haves deez accountabeelity seeztem for de community collegez. u A bill was passed to create the framework, and eventually the framework was enacted. u Named: Accountability Reporting for Community Colleges (ARCC).

13. 12 Arnold Said: u There shall be no pay for performance, but there will be the ability to compare performance.

14. 13 We Said: u Some metrics will be system only; others will be at college-level u College metrics will be rates (to mitigate size for comparison) u No rankings—we will compare colleges against their “peers” u No $$$=ARCC is a “dashboard” accountability report.

15. 14 Arnold Said: u Colleges need to address their performance annually to the State.

16. 15 We Said: u Colleges are more responsive to their local district Board; annual requirement to take local ARCC results to local Board and submit minutes to State u Colleges must submit 500 word response, which becomes a part of the final report.

17. 16 Arnold Said: u The report shall be done in collaboration with the State, not in isolation.

18. 17 We Said: u The Dep’t of Finance, Leg Analyst, and Secretary of Education shall be a part of the technical advisory committee (along with CCC researchers and stakeholders). u We will either succeed or fail together. u This was a really smart move.

19. 18 ARCC u The Model: l Measures 4 areas with 13 metrics: u Student Progress & Achievement- Degree/Certificate/Transfer u Student Progress & Achievement- Vocational/Occupational/Workforce Dev. u Pre-collegiate improvement/basic skills/ESL u Participation l “Process” is not measured

20. 19 Student Prog. & Achievement: Degree/Cert/Xfer u College: l Student Progress & Achievement Rate(s) (SPAR) l “30 units” Rate for SPAR cohort l 1 st year to 2 nd year persistence rate u System: l Annual volume of transfers l Transfer Rate for 6-year cohort of FTF’s l Annual % of BA/BS grads at CSU/UC who attended a CCC

21. 20 Student Prog. & Achievement: Voc/Occ/Wkforce Dev u College: l Successful Course Completion rate: vocational courses u System: l Annual volume of degrees/certificates by program l Increase in total personal income as a result of receiving degree/certificate

22. 21 Precollegiate Improvement/Basic Skills/ESL u College: l Successful Course Completion rate: basic skills courses l ESL Improvement Rate l Basic Skills Improvement Rate u System: l Annual volume of basic skills improvements

23. 22 Participation u College: l None yet…but coming. u System: l Statewide Participation Rate (by demographic)

24. 23 Major Advancements of ARCC u Creating a viable alternative to the GRS Rate for grad/transfer rate. u Finding transfers to private/out of state institutions. u Doing a wage study. u Geo-mapping district boundaries. u Creating peer groups.

25. 24 Defining Grad/Transfer Rate u Student Progress & Achievement Rate (SPAR Rate) l IPEDS-GRS for 2-yr colleges stinks: u No part-timers u How do you define degree-seeking? u Tracking period too short u Outcomes counting methodology terrible v AA/AS/Cert counted before transfer v Transfer to 2-yr college is counted

26. 25 SPAR Rate u Defining the cohort: l Scrub “first-time” by checking against past records (CCC, UC, CSU, NSC)

27. 26 SPAR Rate u Define “degree-seeking” behaviorally for CC populations l Not by self-stated intent; this is a poor indicator u Behavior: did student ever attempt transfer/deg-applicable level math OR English (at any point in academic history) l Students don’t take this for “fun”

28. 27 Defining Degree-Seeking Behaviorally u Separates out remedial students not yet at collegiate aptitude l Measure remedial progression to this threshold elsewhere u Creates common measurement “bar” of student aptitude between colleges l Same students measured=viable comparison

29. 28 SPAR Rate-Unit Threshold u CCC provides a lot of CSU/UC remediation l Lots of students take transfer math/Eng and leave/take in summer l Should not count these as success or “our” student u Set minimum unit completed threshold (12) for cohort entrance l Any 12 units in 6 years anywhere in system

30. 29 SPAR Denominator: u First-Time (scrubbed) u Degree-seeking (at any point in 6 years, attempt transfer/degree applicable math or English) u 12 units (in 6 years) u This represents about 40% of students in our system

31. 30 SPAR Numerator u Outcomes the State wants: l Earned an AA/AS/certificate; OR l Transfer: to a 4-yr institution; OR l Become “transfer-prepared”;OR u Completed 60 xferable units l Became “transfer-directed”: u Completed both xfer level math AND English l No double-counting, but any outcome counts l SPAR Rate=51%

32. 31 Tracking Transfers u SSN-level matches with CSU, UC u Nat’l Student Clearinghouse for private, proprietary, for-profit, out of state l Match 2x/yr, send all records since 1992 l Update internal “xfer bucket” u Works great for cohort tracking u Needed method for “annual volume”

33. 32 Tracking Transfers u Annual Volume of Transfers l CSU/UC: they provide these figures based on their criteria u We didn’t want to redefine this l Private/Out of State: NSC “cross-section” cut method u Validated against CSU/UC xfers from NSC source l Added another 30% to annual volumes

34. 33 97-9898-9999-0000-0101-0202-0303-0404-0505-0606-07 FTF→→→→→→→→T →→→→→→→R →→→→→→A →→→→→N →→→→S →→→F MIN FTF→→E 12 FTF→R UNITS FTF06-07

35. 34 Sector01-0202-0303-0404-0505-0606-07 CSU50,47350,74648,32153,69552,64254,391 UC12,29112,78012,58013,21113,46213,874 ISP17,07015,54118,10018,36517,84018,752 OOS10,76210,54011,15011,70911,72611,825 Total90,59689,60790,15196,98095,67098,842

36. 35 Transfer: Sector of Choice % to UC % to CSU % to Instate Private % to Out of State White17.9%60.7%11.0%10.4% AfrAm11.5%51.2%18.1%19.2% Hisp/Lat15.1%67.7%12.1%5.1% Asian37.0%49.9%9.2%3.9%

37. 36 Demography of Transfer Demog (06-07) FTF Stdents All Stdents XFER- CSU XFER- UC XFER- ISP XFER- OOS AfrAm9%8%5%3%11%13% Asian11%12% 26%8%7% Hisp/ Latino35%29%23%16%23%13% White29%35%37%40%44%55%

38. 37 The Rise of The Phoenix 96-972,166 97-982,829 98-993,374 99-004,194 00-015,055 01-025,586 02-036,515 03-048,222 04-058,585 05-068,134 06-079,216

39. 38 Who Transfers to Phoenix? EthnicityUCCSUPhoenix Asian29.3%14.2%4.6% African American2.4%5.2%16.8% Hispanic/Latino13.6%23.8%28.6% White39.1%43.6%37.5%

40. 39 Wage Study u What was the economic value of the degrees (AA/AS/certificate) we were conferring? u Required data match with EDD l Had to pass a bill changing EDD code to allow match

41. 40 Wage Study u Take all degree recipients in a given year l Subtract out those still enrolled in a CCC l Subtract out those who transferred to a 4- yr institution u Match wage data 5 years before/after degree

42. 41 Wage Study u Separate out two groups: l Those with wages of basically zero before degree l Those with >$0 pre wage u The result: The Smoking Gun of Success

43. 42

44. 43 Mapping Districts u CC Districts in CA are legally defined, have own elections, pass own bonds u We did not have a district mapping for all 72 districts l So we couldn’t do district participation rates

45. 44 Mapping Project u Get a cheap copy of ESRI Suite u Collect all legal district boundary documents u Find cheap labor—no budget for this

46. 45

47. 46 Peer Grouping u “Peers” historically have been locally defined: l My neighbor college l Other colleges with similar demography l Other colleges with similar size

48. 47 Peer Grouping u Taking peering to another level: l Peer on exogenous factors that predict the accountability metric’s outcome u Thus leaving the “endogenous” activity as the remaining variance l Cluster to create groups u We picked 6 clusters, with a min of 3 in a cluster l Each metric produces different factors, peers, clusters

49. 48 Peer Grouping: Example u Peering the SPAR Rate: l 109 rates as outcomes l Find data for all 109 that might predict outcomes/explain variance l Perform regression and other magical SPSS things l See how high you can get your R2

50. 49 Finding Data u What might affect a grad/transfer rate on an institutional level? l Student academic preparedness levels l Socioeconomic status of students l First-gen status of students l Distance to nearest transfer institution l Student age/avg unit load

51. 50 Finding Data u We had to create proxy indices for much of these (142 tried) l GIS system: geocode student zipcode/ZCTA l Census: lots of data to be crossed by zip/ZCTA l Create college “service areas” based on weighted zip/ZCTA values u Different than district legal boundaries

52. 51

53. 52 Finding Data u The Killer Predictor l “Bachelor Plus Index”, or what % of service area population of college has a bachelor’s degree or higher u “Bachelor Plus Index” a proxy for: l First gen l Academic preparedness l Socioeconomic status l Distance to nearest transfer institution

54. 53 Peering SPAR Rate u Exogenous factors that predict SPAR Rate: l Bachelor Plus Index l % older students l % students in basic skills u R2 =.67 l What’s left is implied institutional variance u Demo

55. 54 Peering: What’s Bad u Its complex and somewhat confusing and labor intensive. u Colleges traditional notion of “peer” is shaken u Multiple peers for multiple metrics; can change every year u You could do well vs. State average, increasing over time, but last in your peer group

56. 55 Peering: What’s Good u Its complex and somewhat confusing u You will likely look good in some areas, OK in others, and low in others u Its not very likely anyone will be high or low in all 6 metrics u It eliminated rankings.

57. 56 The ARCC Report u Is almost 800 pages. u Comes out every March. u Takes 4 PY’s to complete (about 6 months/yr) u Is generally regarded highly in CA academic and Legislative circles. u DOF and LAO and Sec. of Ed love it. u Local Trustees/Boards love it.

58. 57 The ARCC Collaboration u Has brought the system more money: l $33 mil in basic skills l Increased noncredit reimbursement rates by $300/FTE u Has brought about trust between system and State stakeholders. u Has educated both sides tremendously.

59. 58 No More “Girlie-Man” Accountability!