Basic Skills, Outcomes, and Fostering a Culture of Evidence and Inquiry | 2009 | RP Group

Basic Skills, Outcomes & Fostering a Culture of Evidence and Inquiry Sponsored by the William & Flora Hewlett Foundation Research conducted by the RP Group Rob Johnstone, Skyline College; Janet Fulks, Bakersfield College; Cathy Hasson, San Diego CCD; Ken Meehan, Fullerton College

Basic Skills, Outcomes, and Fostering a Culture of Evidence and Inquiry | 2009 | RP Group Agenda Opening Dialog: Why are we struggling? Challenges identified through the Basic Skills Outcomes Capacity (BSOC) Study Strategies for responding to these challenges Closing Dialog: What would help colleges strengthen cultures of inquiry? Basic Skills, Outcomes, and Fostering a Culture of Evidence and Inquiry | 2009 | RP Group

BSOC Finding There is a perception gap between administrators and faculty regarding the availability and use of assessment data. College administrators believe that information is more widely available and integrated into implementation planning than those who work more closely with students.

Basic Skills, Outcomes, and Fostering a Culture of Evidence and Inquiry | 2009 | RP Group The Information Gap No data needs being met All data needs being met Information Gap Gap

Basic Skills, Outcomes, and Fostering a Culture of Evidence and Inquiry | 2009 | RP Group Opening Dialog Findings from the BSOC survey suggest we are struggling to get the right information to the right people at the right time. What are your thoughts? Basic Skills, Outcomes, and Fostering a Culture of Evidence and Inquiry | 2009 | RP Group

BSOC Finding Colleges are data rich and information poor.

Basic Skills, Outcomes, and Fostering a Culture of Evidence and Inquiry | 2009 | RP Group The Data We Collect Data Not Being Used Data Frequently Being Used Unmet Unmet Data Needs

Basic Skills, Outcomes, and Fostering a Culture of Evidence and Inquiry | 2009 | RP Group The Action Gap “Our college is quite successful at gathering data and putting reports together based on that data. Where we struggle is with executing action plans as a result of this data. I don’t really think we do a good job of ‘doing’ anything with the data, or implementing programmatic changes or initiatives based on the data.”

Basic Skills, Outcomes, and Fostering a Culture of Evidence and Inquiry | 2009 | RP Group BSOC Finding There is a pattern of gaps in the use of data and evidence across community colleges – gaps that might be inhibiting institutional learning.

Basic Skills, Outcomes, and Fostering a Culture of Evidence and Inquiry | 2009 | RP Group Institutional Data Fills Two Important Gaps Elevated Level of Organizational Awareness Improved Processes & Performance Knowledge Gap Performance Gap Strategic Planning Function Institutional Effectiveness & Student Success Function (Poor Planning)(Good Planning)(Good Performance) Classroom and Service Area Assessment Standard Level of Organizational Awareness Institutional Outcomes and Benchmarks

Basic Skills, Outcomes, and Fostering a Culture of Evidence and Inquiry | 2009 | RP Group BSOC Finding Colleges may be focusing on the wrong data.

Basic Skills, Outcomes, and Fostering a Culture of Evidence and Inquiry | 2009 | RP Group Typical Twenty Year Trend for Common Institutional Outcome Measures Retention Rate Success Rate Consider the multitude of changes over this period: - faculty/staff turnover - program successes/failures - changing student demographics - budget contractions/expansion - evolving state & accreditation mandates - leadership turnover and yet these performance metrics remained amazingly stable. What does that tell us about the usefulness of these metrics in setting institutional strategies?

Basic Skills, Outcomes, and Fostering a Culture of Evidence and Inquiry | 2009 | RP Group Assessment and the Organizational Learning Map Desired Outcome (Pass transfer-level course) Program Level Assessment Assess at the end of the semester Modifications tend to be structural Change is slow & deliberate Assessment within the Classroom Assess both during and at the end of the semester Modifications are both structural & procedural Change is frequent & free form Entry Point (Three levels below transfer) Did the learning community for Math 100 work? Within Math 100, did a group assignment produce stronger learning than a lecture? Learning Pathways under Three Assessment Frameworks

Basic Skills, Outcomes, and Fostering a Culture of Evidence and Inquiry | 2009 | RP Group BSOC Recommendations 1.Basic skills success rates should be measured by analyzing intermediate measures of student progress, in addition to ultimate certificate/ transfer/associates degree attainment. 2.The 2009 ARCC supplemental report includes the “Gold Standard” - progress rate through basic skills courses to transfer level course by discipline. 1

Basic Skills, Outcomes, and Fostering a Culture of Evidence and Inquiry | 2009 | RP Group BSOC Finding Institutional research supports institutional metrics to a higher degree than classroom assessment.

Basic Skills, Outcomes, and Fostering a Culture of Evidence and Inquiry | 2009 | RP Group Level of Involvement in the Use of Data Chief Instructional Officer Classroom Faculty High Level of InvolvementLow Level or No Involvement

Basic Skills, Outcomes, and Fostering a Culture of Evidence and Inquiry | 2009 | RP Group BSOC Finding Much of what is need to support good decision making lies beyond the current grasp of our institutional databases.

Basic Skills, Outcomes, and Fostering a Culture of Evidence and Inquiry | 2009 | RP Group Student Progress in a Non-Linear Fashion

Basic Skills, Outcomes, and Fostering a Culture of Evidence and Inquiry | 2009 | RP Group BSOC Finding Over the last decade, most of the progress in gathering and utilizing data has been at the institutional level.

Basic Skills, Outcomes, and Fostering a Culture of Evidence and Inquiry | 2009 | RP Group Degree to Which Evidence is Used in Decision Making High Low Use of Data for Institutional Metrics Use of Data in Classroom/Service Area Assessment LowHigh Strong Sense of Direction but Poor Understanding of Program Performance Much Context- Specific Knowledge but Poor Integration Poor Planning & Weak Understanding of Programs Closing the Loop Robust Planning & High Levels of Innovation Outward Looking Inward Looking Forward Looking Cluster of California Community Colleges

Basic Skills, Outcomes, and Fostering a Culture of Evidence and Inquiry | 2009 | RP Group BSOC Recommendations Research should be brought to the intervention level, at the classroom. 2

Basic Skills, Outcomes, and Fostering a Culture of Evidence and Inquiry | 2009 | RP Group Expanded Definition of Research Capacity Production Dissemination Interpretation Translation Closing & Widening the Loop

Basic Skills, Outcomes, and Fostering a Culture of Evidence and Inquiry | 2009 | RP Group BSOC Recommendations Research offices should receive support that will enable them to address multiple domains of research capacity. Technical assistance should be provided to colleges that will help them to strengthen cultures of evidence and inquiry. 3 The Bridging Research, Information, and Culture (BRIC) Project  

Basic Skills, Outcomes, and Fostering a Culture of Evidence and Inquiry | 2009 | RP Group The BRIC Project Developing a Better Understanding of How to Close the Performance Gap Standard Level of Organizational Awareness Elevated Level of Organizational Awareness Improved Processes & Performance Knowledge Gap Performance Gap Strategic Planning Function Institutional Effectiveness & Student Success Function (Poor Planning) (Good Planning)(Good Performance) Institutional Outcomes and BenchmarksClassroom and Service Area Assessment

Basic Skills, Outcomes, and Fostering a Culture of Evidence and Inquiry | 2009 | RP Group Closing Dialog The BRIC project will allow us to provide tailored support at colleges. How could an external team of faculty, administrators, and researchers help deepen the culture of inquiry at your institution? Basic Skills, Outcomes, and Fostering a Culture of Evidence and Inquiry | 2009 | RP Group

Contact & Information RP Group Website BSOC/BRIC Project Contacts Robert Johnstone, Project Director Priyadarshini Chaplot, Project Coordinator Basic Skills, Outcomes, and Fostering a Culture of Evidence and Inquiry | 2009 | RP Group