1. The Creation of Two Comparison Groups to Facilitate Evidence-Based Decision Making Prepared and Presented by Keith Wurtz Dean, Institutional Effectiveness, Research and Planning Crafton Hills College

2. Session Objectives Provide a brief introduction to the Left Lane Program Explain where the idea for two comparison groups came from Describe the two comparison groups and the limitations of the study Describe the findings Review the process for Continuous Quality Improvement and how the results were used to inform decision-making

3. The Left Lane Program (LLP) Purpose of the LLP is to reduce the amount of time it takes students to earn an AA/AS Degree at Crafton from 5.3 years to 4 years Program funded by the San Bernardino Community College District Chancellor First Cohort in Fall 2012

4. The Left Lane Program (LLP) Students who participate in Left Lane receive Priority Registration B To keep the Priority Registration B status students are required to do the following – Attend a Summer Bridge Program – Participate in SOA 3 R – Student Orientation, Application, Assessment, Advisement, and Registration – Complete and closely follow their Student Education Plan – Enroll in math and English as stated in their SEP – Attend all tutoring meetings and check-in appointments – Maintain a 2.0 GPA – Enroll in 12 or more units per semester (depending on SEP)

5. The Left Lane Program (LLP) How are students selected for the LLP? – Need to place into one level below transfer English or one level below transfer level math – Need to apply

6. The Idea for Two Comparison Groups Collaborative Model of Institutional Research Faculty/Staff/Student Researcher Joint Activity Primary Responsibility Key Features: Dialogue-rich Jointly-driven processes Source: Dr. Rob Johnstone and Priya Chaplot, RP Group Conference Presentation Raju Hegde Dean of Math, English, Reading, and Instructional Support Kirsten Colvey Dean of Student Services, Counseling, and Matriculation

7. The Two Comparison Groups Left Lane Students – 258 out of 276 (93%) earned a GOR in Fall 2012 – 82% graduated high school in 2012 – 63% Female – 49% Hispanic First Comparison Group (Last Year High School was 2012) Second Comparison Group (Last Year High School was 2012 and did not participate in LLS)

8. The Two Comparison Groups Characteristic Left Lane Students 1 st Comparison Group 2012 High School Graduates 2 nd Comparison Group 2012 HS Graduate and Did not Participate in LL Strategy N258583357 Graduated HS82%100% Female63%48%49% Hispanic49%44%48% SOA 3 RYesVariedNo LCYesVariedNo SIVaried No

9. The Two Comparison Groups Characteristic Left Lane Students 1 st Comparison Group 2012 High School Graduates 2 nd Comparison Group 2012 HS Graduate and Did not Participate in LL Strategy N258583357 Placed Basic Skills Math22%26%30% Placed Developmental Math77%64%62% Placed Transfer Level Math1%10%8% Placed Basic Skills English36%37%40% Placed Developmental English40%34%32% Placed Transfer Level English24%29%28%

10. Limitations The success rate comparisons do not control for instructor or by when the section was offered The comparison groups do not control for skill level (i.e. math and English placements)

11. Percent of Students Earning a Grade on Record in English and Math Left Lane students were substantially (Effect Sizes were >.67) and statistically significantly (p values were <.001) more likely to earn GOR in English and Math.

12. Percent of Students Successfully Completing English and Math Left Lane students were slightly more likely to successfully complete English (ES =.10, p =.308) and math (ES =.11, p =.196) than students in the first comparison group Left Lane students were substantially and statistically significantly more likely to successfully complete English (ES =.32, p =.020) and math (ES =.27, p =.017) than students in the second comparison group

13. Percent of Students Successfully Completing English, Math, and/or Reading Left Lane female (ES =.20, p <.001, 81% to 73%), African American (ES =.43, p =.007, 81% to 61%), and Hispanic (ES =.19, p <.001, 79% to 70%) students were substantially and statistically significantly more likely to successfully complete courses than students in the first comparison group Left Lane female (ES =.31, p <.001, 81% to 68%), African American (ES =.77, p <.001, 81% to 45%), and Hispanic (ES =.27, p <.001, 79% to 66%) students were substantially and statistically significantly more likely to successfully complete courses than students in the second comparison group

14. English, Math, and Overall Retention (i.e. Persistence – earning GOR) from Fall to Spring Left Lane English (ES =.25, p =.007), Math(ES =.19, p =.034), and overall (ES =.19, p <.012) retention was substantially and statistically significantly higher for Left Lane students than students in the first comparison group Left Lane English (ES =.39, p =.002), Math(ES =.27, p =.018), and overall (ES =.31, p <.001) retention was substantially and statistically significantly higher for Left Lane students than students in the second comparison group

15. English and Math Improvement Rates from Fall to Spring Left Lane students did not perform as well in their subsequent English and/or math course after successfully completing an English and/or math course in their first semester

16. 1. Gather Data / Assess Process for Continuous Quality Improvement Continuous Quality Improvement 1. Gather Data / Assess 2. Review, Discuss, & Gather Feedback 1. Gather Data / Assess 2. Review, Discuss, & Gather Feedback 3. Identify Strategies for Improvement 1. Gather Data / Assess 2. Review, Discuss, & Gather Feedback 3. Identify Strategies for Improvement 4. Implement Improvements Communicate Results Communicate Feedback Communicate Strategies Communicate Improvements

17. Left Lane Program Summer Bridge Evaluation Surveys Initial Research after the first Fall semesterResearch More in-depth research after the first year research 1. Gather Data / Assess 2. Review, Discuss, & Gather Feedback 3. Identify Strategies for Improvement 4. Implement Improvements

18. Left Lane Program Summer Bridge Evaluation Surveys Initial Research after the first Fall semesterResearch More in-depth research after the first yearresearch Evaluation surveys and interviews indicated that students wanted both English and math during Summer Bridge 39% of non-Left Lane students participated in SOA 3 R, a learning community, or supplemental instruction. As a result, first comparison group had success rates 70% or higher. LLP students are more likely to enroll in math and English in the first term and they were less likely to enroll in and successfully complete math and English in subsequent spring semester. 1. Gather Data / Assess 2. Review, Discuss, & Gather Feedback 3. Identify Strategies for Improvement 4. Implement Improvements

19. A Note on Facilitating Continuous Quality Improvement Traditional approach to facilitating evidence-based decision making – Provide the research results to constituents only in reports, briefs, and dashboards – Discuss the possible implications and interpretations verbally Example – Left Lane students were substantially (ES = 1.10) and statistically significantly (p <.001) more likely to earn a GOR in a math course their first semester (89%) at Crafton than non-Left Lane students (34%) – Left Lane students were substantially (ES = 1.03) and statistically significantly (p <.001) more likely to earn a GOR in an English course their first semester (77%) at Crafton than non-Left Lane students (26%)

20. A Note on Facilitating Continuous Quality Improvement New approach to facilitating evidence-based decision making Provide the research results to constituents in reports, briefs, and dashboards, and… Provide the research results with an interpretation that focuses on the possible implications of the research Left Lane Comparison Groups and Possible Improvement Areas When preparing the comparison groups to examine the relationship between Left Lane Students and student outcomes one of the most interesting findings was the number of Fall 2012 first-time Crafton students who graduated from high school in 2012 who did not participate in Left Lane, but had participated in one of the Left Lane Strategies (LLS). Thirty-nine percent (n = 226) of these students participated in SOA 3 R, a learning community, or in supplemental instruction. As a result, the first comparison group of incoming students had success rates that were 70% or higher. Moreover, the Left Lane students often did not have a statistically significantly higher likelihood of achieving the outcomes examined than the first comparison group. Suggesting the possibility that one method for improving the LLP is to focus on ensuring that as many students as possible participate in the LLS rather than solely on LLP. One of the strategies for decreasing the average amount of time it takes students to complete an AA/AS Degree involves increasing how quickly students are able to successfully complete transfer level English and math. The initial results of the research illustrated here indicates that Left Lane students are more likely to enroll in math and English courses in the fall, but in some instances they were not as likely to enroll in math and English in the subsequent semester and/or successfully complete the subsequent English and/or math course in the summer semester. LLP may need to require students to participate in the LLS in the spring semester and devote additional resources to services in the spring semester.

21. Left Lane Program Summer Bridge Evaluation Surveys Initial Research after the first Fall semesterResearch More in-depth research after the first yearresearch Evaluation surveys and interviews indicated that students wanted both English and math during Summer Bridge 39% of non-Left Lane students participated in SOA 3 R, a learning community, or supplemental instruction. As a result, first comparison group had success rates 70% or higher. LLP students are more likely to enroll in math and English in the first term and they were less likely to enroll in and successfully complete math and English in subsequent spring semester. Combine Summer Bridge so that students are prepared for both English and math One method for improving the LLP is to focus on ensuring that as many students as possible participate in the LL strategies. Require students to participate in LL strategies in the spring along with enrolling in math and English in the spring. 1. Gather Data / Assess 2. Review, Discuss, & Gather Feedback 3. Identify Strategies for Improvement 4. Implement Improvements

22. Left Lane Program Summer Bridge Evaluation Surveys Initial Research after the first Fall semesterResearch More in-depth research after the first yearresearch Evaluation surveys and interviews indicated that students wanted both English and math during Summer Bridge 39% of non-Left Lane students participated in SOA 3 R, a learning community, or supplemental instruction. As a result, first comparison group had success rates 70% or higher. LLP students are more likely to enroll in math and English in the first term and they were less likely to enroll in and successfully complete math and English in subsequent spring semester. Combine Summer Bridge so that students are prepared for both English and math One method for improving the LLP is to focus on ensuring that as many students as possible participate in the LL strategies. Require students to participate in LL strategies in the spring along with enrolling in math and English in the spring. Combined the Summer Bridge Program to include both English and Math Required Students to complete a 2 semester SEP to stay enrolled in math and English New LLP proposal to SBCCD Chancellor emphasized increasing the capacity of the LLP and its strategies: counseling, SEPs, SOA3R and SI. In addition, also emphasized creation of DLAs in math and English, additional sections in math and English in both fall and spring. LL strategies are also being included in new Title COOP grant proposal. 1. Gather Data / Assess 2. Review, Discuss, & Gather Feedback 3. Identify Strategies for Improvement 4. Implement Improvements

23. Additional Questions