October 28, 2010 Use of Research Data to Assess and Improve Educational Quality and Institutional Effectiveness Presentation at California Community Colleges Chief Instructional Officers Workshop by Dorte Kristoffersen Associate Vice President, ACCJC 1

October 28, 2010 Presentation Outline Some Definitions Characteristics of Good Data and Data Sources Data-driven Decision-making Purposes of Accreditation Types of Data and Data Requirements Examples of Effective Use of Data Concluding Remarks 2

October 28, Definitions of Evidence and Data General definitions: Evidence includes everything that is used to determine or demonstrate an assertion Data refers to groups of information that represent the qualitative or quantitative attributes of a variable or set of variables.variable

October 28, Use of the Terms Evidence and Data in Accreditation Evidence is all information included in the accreditation process, i.e. Self Evaluation Reports of Educational Quality and Institutional Effectiveness, information supporting the claims and descriptions in the Self Evaluation Report, interviews during the site visit, and previous External Evaluation Reports. Data is information deliberately collected by the institution to provide it with clear information about the quality of its education and services.

October 28, 2010 Characteristics of Good Data For data to be a useful and reliable source of information and for decision-making, it needs to have the following characteristics. It should be: Accurate and tested for validity and significance Up-to-date and complete in terms of aspects of analysis it is meant to cover Consistently used From reliable sources Longitudinal and disaggregated as appropriate I 5

October 28, 2010 Data Sources There are several sources of data than an institution can draw information from. Data sources can be internal, external or both. Examples of data sources are: Institutional data, such as statistics and survey results Demographics at the local/district, regional or federal level Data reported to State Government and/or relevant associations, such as Cal-PASS Other sources relevant for the institution 6

October 28, 2010 Data-driven decision-making It is a sound principle when making judgments to act on what we know. Judgments should not be based on assumptions and feelings. Data is a necessary knowledge-base for decision-making The information that an institution collects, analyzes, and reflects upon should be designed to answer questions the institution has raised against its mission and institutional objectives, and the information will thus determine what action the institution takes to support the improvement of educational quality and support student success 7

October 28, 2010 Data-driven decision-making continued ‘No News; is normally ‘Good News’ But ‘No data’ does not mean ‘good news’. It means no sense of bearings and no direction for the future. 8

October 28, 2010 Purposes of Accreditation To provide assurance to the public that education provided by institutions meets acceptable levels of quality. To promote continuous institutional improvement. To maintain the high quality of higher education institutions in the region/nation. 9

October 28, 2010 Types of Data The principle of data-driven decision-making should apply to an institution’s reflections on its achievement of its mission and institutional objectives and its compliance with the Accreditation Standards The Commission requires various kinds of specific data Most of the data is required to support institutions’ claims of compliance with the Accreditation Standards and some of the data relate to the U.S.D.E requirements The Commission also expects that institutions collect data which is relevant for the institutions’ internal continuous quality improvement activities 10

October 28, 2010 Commission Data Requirements Data about incoming students, i.e.: Student demographics Student enrolment data across instructional programs Student educational goals Student development needs 11

October 28, 2010 Commission Data Requirements Continued Data about incoming students should include information about students studying in Distance Education or Correspondence Education mode, such as: Account of programs, courses, certificates where 50% or more is offered in distance education or correspondence education mode Annual growth in headcount enrollment 12

October 28, 2010 Commission Data Requirements Continued Data about Program Review, such as: Number of enrolled students Student learning outcomes at program and course level Assessment results 13

October 28, 2010 Commission Data Requirements Continued Data on student grievances and complaints, including follow-up Data on Internationalization, i.e.: Numbers of international campuses Numbers of programs for non-U.S. students Numbers of non-U.S. students enrolled 14

October 28, 2010 Commission Data Requirements Continued Two types of data are particularly important as they relate directly to student success, i.e. Student Achievement Data Student Learning Outcomes Data These two types of data will be explored in more detail as examples 15

October 28, 2010 Student Achievement Data Data on student achievement has been long required by the ACCJC. Institutions already use some common measures to examine student achievement, and some of these measures derive from federal regulatory language. 16

October 28, 2010 Student Achievement Data Continued Required Student Achievement Data: Retention rates from term to term Student progression to the next course/level of course, including from pre-collegiate to collegiate level Graduation rates (certificate or degree) Job placement rates Licensure pass rates Transfer rates to four-year institutions 17

October 28, 2010 Value of Student Achievement Data Student Achievement Data is end-point data providing an institution with key information about the achievement of its institutional mission and in terms of student success Such data will: If collected longitudinally and analyzed continuously provide the institution with indications as to whether improvements in pedagogy or services may be required to improve student progression and completion Keep institutions informed about fluctuations and serve as a warning if rates decrease and trends need to be reversed 18

October 28, 2010 Value of Student Achievement Data Continued Student achievement data will also: Provide information to the institutions about barriers to completion and transfer Present trends that identify the need for institutions to explore the barriers through the collection of additional data and/or conduct separate surveys If collected in disaggregated form inform institutions as to whether all students progress and complete their studies or if attention needs to be given to special groups 19

October 28, 2010 New Commission Requirement As of Fall 2012, Self Evaluation Reports of Educational Quality and Institutional Effectiveness should include data on key student achievement measures in disaggregated form by age, gender, race/ethnicity, socio-economic status and other measures that the institution considers relevant for its population 20

October 28, 2010 An Example of Student Achievement Data Research study by Institute for Higher Education Leadership and Policy (IHELP). Divided we Fail: Improving Completion and Closing Racial Gaps in California’s Community Colleges, October 2010 IHELP followed degree seeking Californian community college students over the course of six years The goal of the study was to use it to identify solutions to improve completion rates for all students and closing the disparities across racial/ethnic groups The study is conducted at a generic level, but the findings and how the can be used can serve as an example for individual colleges 21

October 28, 2010 An Example Continued Too few students reach the milestones: racial/ethnic disparities abound: Too many students fail to complete Six years after enrolling 70% of degree-seeking students had not completed a certificate or degree and had not transferred to a university (most had dropped out) Failure rates among black students (75%) and Latinos (80%) are higher 22

October 28, 2010 An Example Continued Critical milestone is missed: Only 40% of degree-seeking students had earned at least 30 college level credits at a CCC. 35% of Latino students and 28% of black students reached this milestone Fewer Latinos who reach the above milestone complete a certificate, degree or transfer (47%) as compared to white (60%), Asian-Pacific Islander (58%) and black (53%) students 23

October 28, 2010 Student Learning Outcomes Data The Commission also requires institutions to demonstrate educational effectiveness by collecting and analyzing student learning outcomes data Student Learning Outcomes data: Is end-point data Consists of learning outcomes at course, program and degree level Is derived from summative assessment 24

October 28, 2010 Value of Student Learning Outcomes Data Student Learning Outcomes data: Provides information about how well students are able to achieve institutional and programmatic missions and objectives, and Is therefore a key source of data for Program Review May help institutions identify barriers to student learning and student success and how these barriers can be addressed May identify whether improvements in curriculum and teaching and learning methods are warranted to increase the achievement of learning outcomes 25

October 28, 2010 Key Questions to Ask about Data Collection and Data Use What is the mission of the institution (Institutional, Programmatic)? What are the institutional objectives? What is the institution trying to achieve in terms of its institutional missions and educational quality, including possible goals and targets? What data is relevant to collect to gain information on institutional achievements? 26

October 28, 2010 Key Questions to Ask continued What are the data sources, i.e. internal and/or external? Who should be involved in the data analysis? How should the outcomes of the analysis be reported? Who is responsible for taking action and ensuring that improvements are made? 27

October 28, 2010 Concluding Remarks Data is collected intentionally and its meaning and relevance has been considered It is purposeful and designed to provide information on institutional achievements It is collected systematically and continuously Data is interpreted, reflected and acted upon It is used for improvement 28

October 28, 2010 ‘Learning without thought is labor lost; thought without learning is perilous.’ (Confucius BC) 29

October 28, 2010 Thank you ACCJC/WASC 10 Commercial Blvd, Suite 204 Novato, CA Tel: Fax: Website: