Qualitative & Quantitative Perspectives Research and Planning Group 40 th Annual Conference Pacific Grove, CA May 1-3, 2002 Esau Tovar, M.S. Retention Counselor Santa Monica College (310)

2 Need for Study Increased faculty concerns with current program High rates of unsuccessful grades awarded to “EA” students No clear evidence that EA works Need to systematically assess EA High probationary rates & low persistence rates at SMC

3 Why Early Alert?

4 Matriculation Components Admissions Orientation Assessment Counseling & Advisement Student Follow-Up Coordination & Training Research & Evaluation Prerequisites, Corequisites, Advisories

5 Student Follow-Up Post enrollment evaluation of non- exempt students Early intervention Appropriate referrals Positive and non-positive feedback

6 “Early Intervention” Model Early Alert Program Coordinated by Matriculation Office Instructors identify “at-risk” students through EA rosters Typically between 4 th & 6 th week All instructor responses merged for each student identified Notices sent to students informing them of status and possible referrals No additional follow-up

7 Goals of Matriculation Two primary goals Student success Institutional effectiveness Student services and instruction partnership Systemwide accountability Efficient use of resources Institutional research Increased participation of underrepresented students

8 Studying Early Alert Driven by limited research

9 Research Questions How effective is the EA program in the identification of “at-risk” students? To what extent has EA impacted student success? What do instructional and counseling faculty think of EA? What components should EA incorporate to improve its effectiveness?

10 EA Evaluation Approaches Institutional research Academic performance outcomes Student interviews Tutoring use Counseling/Instructional faculty interviews & survey Major findings

11 EA Academic Performance Outcomes Source: Geltner, P. (2001). The Characteristics of Early Alert Students: Fall Santa Monica College Office of Institutional Research. EA Grade Records 7,913 All Grade Records 72,912 EA students succeed to a lesser extent than All students (28% vs. 67%) Withdrawal rates are significantly higher for EA students 47% vs. 19% Despite differences in grades awarded, conclusions are hard to draw

12 Fall 2000 EA Messages Already contacted11.9% Absent or tardy24.5% Turn in assignments16.3% Tutoring needed19.1% Study skills needed13.0% Meet with instructor10.4% Meet with counselor 4.8% Source: Geltner, P. (2001). The Characteristics of Early Alert Students: Fall Santa Monica College Office of Institutional Research.

13 EA Faculty Interviews Faculty not satisfied with EA Use and Understanding of EA No clear link between EA & student success How is it helping students? Problems with Existing EA Program Timeliness is crucial if students are to succeed Exclusion of short-term & online classes The current EA program does not provide any feedback to instructors Once notices are sent out students do not respond Negative image of EA & increased load factor Suggestions for Improvement Use EA to identify at-risk students at any point during semester Letters should be customized by instructor; hand-delivered in class, or ed Feedback to instructors of services provided to students and outcomes Phone calls to students

14 “The Survey”

15 SMC EA Survey Online & Printed Form Background (8 items) EA Effectiveness (15 items X 2 scales) Student Academic Information (7 items X 2 scales) Revamped EA Scenario (9 items) Reporting Features (5 items) Open-ended questions (5 items)

16 Summary of Findings Participants N = 80 Position Held 61% Full-time teaching (n = 48) 32.5% Part-time teaching (n = 26) 4% Full-time counseling (n = 3) 1% Part-time counseling (n = 2) 1% Other (n = 1) FT Faculty Status 43% Tenured (n = 30) 57% Non-tenured (n = 21) Years Employed FT: M = 11.6 PT: M = 10.3 Department Participation Cross-sectional representation (20 departments) Frequency of EA Use 1% Never 1% Rarely 11% Sometimes 13% Often 74% Always Average Time Per Class 96% 1 hour or less Preference for EA Use 14% At any point, Ongoing 5% 1st to 2nd week 23% 3rd to 4th week 46% 5th to 6th week 11% 7th to 8th week 1% 9th week and beyond

17 Assessing Importance & Satisfaction with Early Alert Components

18 Importance of EA Components Referring students easily to college survival workshops (e.g., study skills, time management) Supporting increased student contact with instructional resources Making students aware of academic support resources: tutoring Supporting increased student contact with counselors Providing identified students with timely feedback and suggested interventions to improve performance Providing identified students with accurate feedback on identified problems Supporting increased student contact with instructors Supporting increased student contact with special programs (e.g., Latino Center, EOPS) Providing counselors with appropriate training on use of EA and follow-up 2 Monitoring individual and/or group student academic performance Providing or improving communication among instructors, counselors, and students Tracking students after initial identification Providing instructors with appropriate training on use of EA and follow-up 1 Aiding instructors to identify students at risk of performing poorly academically Assisting instructors by providing a means by which to evaluate student classroom performance n M SD Scale: Importance of Early Alert Components 1 = Not at all important, 5 = Very important Part-Time vs. Full-Time Faculty Member Mean Importance 1 PT = 3.77 (SD = 1.31), FT = 3.02 (SD = 1.47); t(67) = 2.05, p <.05 2 PT = 4.38 (SD = 0.89), FT = 3.42 (SD = 1.31); t(52) = 2.67, p <.01

19 Satisfaction with EA Components Making students aware of academic support resources: tutoring 1 Referring students easily to college survival workshops (e.g., study skills, time management) Supporting increased student contact with special programs (e.g., Latino Center, EOPS) 2 Supporting increased student contact with instructional resources Supporting increased student contact with instructors Providing identified students with timely feedback and suggested interventions to improve performance Aiding instructors to identify students at risk of performing poorly academically Providing counselors with appropriate training on use of EA and follow-up Supporting increased student contact with counselors Monitoring individual and/or group student academic performance Providing identified students with accurate feedback on identified problems Assisting instructors by providing a means by which to evaluate student classroom performance Providing or improving communication among instructors, counselors, and students Tracking students after initial identification Providing instructors with appropriate training on use of EA and follow-up n M SD Scale: Satisfaction of Early Alert Components 1 = Not at all satisfied, 5 = Very satisfied Part-Time vs. Full-Time Faculty Member Mean Satisfaction 1 PT = 4.00 (SD = 1.02), FT = 3.23 (SD = 1.20); t(66) = 2.67, p <.01 2 PT = 3.81 (SD = 0.91), FT = 2.85 (SD = 0.89); t(48) = 3.53, p <.001

20 EA Priorities Matrix* Very Important Not At All Important Very Satisfied Not At All Satisfied High Status Items Improvement Needed (large performance gaps) Greatest Strengths (smallest performance gaps) Low Status Items Opportunity to Study/Redirect (medium performance gaps) Low Status Items Opportunity to Redirect Efforts (medium performance gaps) * Priorities matrix based on Noel-Levitz’s Matrix for Prioritizing Action Importance – Satisfaction = Performance Gap

21 Performance Gaps in EA Components* Supporting increased student contact with counselors Providing identified students with accurate feedback on identified problems Providing identified students with timely feedback and suggested interventions to improve performance Providing or improving communication among instructors, counselors, and students Supporting increased student contact with instructional resources Referring students easily to college survival workshops (e.g., study skills, time management) Supporting increased student contact with instructors Tracking students after initial identification Monitoring individual and/or group student academic performance Providing counselors with appropriate training on use of EA and follow-up Making students aware of academic support resources: tutoring Supporting increased student contact with special programs (e.g., Latino Center, EOPS) Providing instructors with appropriate training on use of EA and follow-up Assisting instructors by providing a means by which to evaluate student classroom performance Aiding instructors to identify students at risk of performing poorly academically n M SD Importance – Satisfaction = Performance Gap

22 Additional Sources of Information Beneficial to Assist Students

23 Information Beneficial to Tailor Courses for All Students & to Identify At-Risk Students High school academic history (e.g., GPA, courses completed)* t(67) = 2.49, p <.01 English—writing—placement information English—reading—placement information Directory of campus referral sources College academic history (e.g., SMC or other college GPA, courses completed) t(69) = -2.71, p <.01 Mathematics placement information t(65) = -2.60, p <.01 Training on referral skills All At-Risk Students Students n M SD M SD Scale: Beneficial to assist ALL STUDENTS & Beneficial to IDENTIFY AT-RISK STUDENTS 1 = Definitely not beneficial, 5 = Definitely beneficial

24 Revamping Early Alert Program at SMC

25 Revamped EA: Scenario Revamped Early Alert Program to address shortcomings expressed by instructors and students. Among the potential solutions being explored are the following: Moving from a paper-and-pencil student evaluation roster to an online-based student evaluation allowing individual customization (for instructor and student); Allowing instructors to identify and assess only at-risk students when necessary; Allowing instructors to provide timely feedback directly to students by means of or printed letters that can be delivered to the student in class; Allowing instructors, counselors, and other campus programs to follow up with instructor recommendations and maintain contact with identified students; Tracking all sources of student contact (e.g., meetings with counselors, workshops attended, tutoring attendance) after initial identification to prepare and easily present reports on such things as the impact of early alert on student success indicators individually, by group, etc.

26 Level of Agreement to Revamped/Integrated EA Components Obtaining handouts at the beginning of the semester addressing interventions that are likely to help at-risk students (before they experience difficulty and are identified for EA purposes) I would be willing to hand out the EA letter directly to the student at a subsequent class meeting Reports pertaining to interventions suggested and those provided Reports pertaining to student success indicators (retention, persistence, grades) The college should design and implement an online-based Early Alert system Early alert evaluations should be completed only for students at risk of performing poorly or experiencing problem in classes I would be willing to have my name appear on the EA letter or sent to students Reports pertaining to how students are contacted and by whom I would be willing to use an online-based early alert system The college should continue to use a paper-and-pencil-based early alert system 1 Reports pertaining to the success of specific groups (gender, ethnicity, age, etc.) Early alert evaluations should be completed for every student in every class n M SD Scale: Level of Agreement Given Scenario 1 = Strongly Disagree, 5 = Strongly Agree Part-Time vs. Full-Time Faculty Member Mean Agreement 1 PT = 3.57 (SD = 1.26), FT = 2.30 (SD = 1.20); t(76) = 4.41, p <.01 2 PT = 2.97 (SD = 1.61), FT = 2.08 (SD = 1.46); t(75) = 2.47, p <.01

27 Open-Ended Questions

28 Additional Components/Feedback Desired: For Students Increase Student/Instructor/Counselor Contact Formalized Referrals/Resources Assessment/Prerequisites Completion & Accuracy Student Unpreparedness Lack Study Skills College Survival Workshops

29 Additional Components/Feedback Desired: For Faculty Implement Follow-Up Component Provide feedback on what happens with the identified student, & the services provided Timeliness of EA Notices Assessment/Prerequisites Completion & Accuracy Student Background Information Assessment Information Progress in Other Courses Increased Instructor/Counselor Collaboration Facilitate Student Learning Office Visits Classroom Management Faculty Training Formalized Referrals/Resources

30 Next Steps

31 Discussions Expanding EA Task Force Consulting with MIS Online EA System Reviewing Products: RetentionTRAX by Noel-Levitz Student Correspondence System by Precision Dynamics

32 Timeline Finalize discussionLate May Develop/Adopt ProductSummer ‘02 Pilot TestFall ’02 Assess EffectivenessFall ‘02 InstitutionalizeSpring ‘03