1. Assessing Student Engagement at the Course Level Rick D. Axelson, PhD, Univ. of Iowa, Off. Consult & Research in Medical Educ. Arend Flick, PhD, Norco College, Dept. of English, Norco, CA Marc A. Pizzimenti, PhD, Univ. of Iowa, Dept. of Anatomy & Cell Biology Tina Taylor-Ritzler, PhD, Dominican University, Dept. of Psychology, IL Annual Conference of the American Evaluation Association Anaheim, CA; November 2, 2011

2. Presentation Outline 1. Project Background 2. Engagement Survey Design 3. Overview of Pilot Test Results 4. Pilot Test Results by Site  Norco College, CA  University of Iowa, Carver College of Medicine  Dominican University

3. 1. Project Background Purpose: gather engagement information to aid instructor and student performance Timing – information available within the semester/quarter to the instructor and students Timing – information available within the semester/quarter to the instructor and students Unit of Analysis – collected at the course level Unit of Analysis – collected at the course level With greater detail about engagement: With greater detail about engagement:  Cognitive- interest, attention, meaning, effort  Affective –bonds with teachers & classmates that impact effort, integration, attachment, commitment to course  Behavioral – time-on-task, class attendance, and participation in class activities and assignments

4. 2. Engagement Survey Design Definition: ‘ “learning task engagement” …refer[s] to students’ cognitive investment, active participation, and emotional engagement with specific learning tasks.’ (Chapman 2003) Survey Item Development: Theory/research literature –Motivated Strategies for Learning Questionnaire (MSLQ) (Pintrich et al, 1993); & psychological conditions of engagement at work (e.g., Kahn 1990) for affective engagement Theory/research literature –Motivated Strategies for Learning Questionnaire (MSLQ) (Pintrich et al, 1993); & psychological conditions of engagement at work (e.g., Kahn 1990) for affective engagement Instructors/teaching evaluations – Review material with instructors to see what information would be useful for designing/delivering course; adopted items from standard course evaluation forms Instructors/teaching evaluations – Review material with instructors to see what information would be useful for designing/delivering course; adopted items from standard course evaluation forms Survey Items can be viewed at: http://www.surveymonkey.com/s/LHY2LQB http://www.surveymonkey.com/s/LHY2LQB http://www.surveymonkey.com/s/LHY2LQB - pilot test #1

5. 2. Survey Design - Content Background/Context for Course: Background/Context for Course:  Expected benefits from course  Demographics, First-generation college, Current work & family situation Perceptions of Classroom environment & course: Perceptions of Classroom environment & course:  Instructor  Course organization, reading materials, pace of course, etc.  Classmates Engagement: Engagement:  Motivations and attitudes (MSLQ subsets)  Learning strategies used by students (MSLQ subsets)  Time & study environment mgt (MSLQ subsets)  Affective engagement & identity (conditions for psychological engagement at work, bonds/fit with class & course) Open ended: student comments about the course & suggested improvements Open ended: student comments about the course & suggested improvements

6. 3. Overview of Pilot Test Results IRB Approvals obtained from all three institutions. IRB Approvals obtained from all three institutions.  Didn’t get permission at Norco to match students’ survey responses with their course grades/percentages. Samples: Samples:  Norco College – 3 courses (Business, English, Psychology), 10 instructors, ~ 200 students (n=85)  University of Iowa, College of Medicine – 1 class (Anatomy), 1 instructor, 150 students (n =66)  Dominican University – 3 courses ( Psychology), 1 instructor (n=27) and collecting more data this semester. Survey Administered via SurveyMonkey – provided students who requested reports with individualized reports on how their use of learning strategies compared with their classmates. Survey Administered via SurveyMonkey – provided students who requested reports with individualized reports on how their use of learning strategies compared with their classmates.

7. 3. Pilot Test Results (Reliabilities & Correlations with Final Course Score)  Perceptions of Classroom Environ. & Course  Course Characteristics (alpha =.87; r =.30)  Instructor Characteristics (alpha =.89; r =.20)  Expected Benefits from Course (alpha =.82; r =-.04)  MSLQ Scales  Motivations and attitudes (alphas:.59 -.89; r’s: -.26 -.33)  Students’ Learning strategies (alphas:.53 -.87; r’s: -.06 -.42)  Time & study environ mgt (alphas:.56 -.81; r’s: -.12 -.26)  New Affective Engagement & Identity items  Classmates (alpha =.80; r = -.07)  Belonging (alpha =.87; r =.23)  Discomfort (alpha =.87; r = -.21)

8. 3. Pilot Study Results – General Observations Reliabilities of measures were generally adequate for classroom improvement purposes. A number of the survey items can likely be eliminated without substantially reducing scale reliability. Revised survey: http://www.surveymonkey.com/s/STLFR3Jhttp://www.surveymonkey.com/s/STLFR3J Some reliabilities and correlations of measures with final course scores varied substantially by site. MSLQ items did not work as well in Community College or Medical School settings as they did for the Liberal Arts College students.

9. 4. Pilot Test Results by Site – Norco College Sample from eight sections of highest level pre- collegiate English composition course (N = 53) Demographics: 40% Latino; 33.8% White; 12.3% Asian; 3.1% African-American Mean age 26.3 (std dev = 10.3) 60% full-time; 40% part-time (25% work 40 hours a week or more) Graduated from college: Mothers 8.5% (6.4% DK); Fathers 23.5% (14.9% DK) Findings: Higher course rating (4.28* vs. 4.12 and 3.86*) (p <.01) 98% had missed two or fewer classes; Mean 5.6 study hrs (std dev =4.1) High expected benefits (4.32); Full range of learning strategies used Were not distracted by too much information (2.36), fatigue (2.04), or other students (1.89), but were distracted by fear of not getting material right (3.63) and interference from their own thoughts (3.04) Motivation questions had low alphas.

10. 4. Pilot Test Results by Site – University of Iowa, Medical School UI Anatomy Class (150 Students, respondents=66) Demographics: 6% Latino; 87% White; 2% Asian; 2% African-American; 3% Other Mean age 24.2 (std. dev. = 3.0) 100% full-time (only 4% work 1-10 hours outside of class) Graduated from college: Mothers 80%; Fathers 76% Findings: Average # of hours per week studying for this class = 9.2 (std dev =4.4) 52% had missed two or fewer classes Correlations with class performance: elaboration (r=.36), motiv (TA r=-.45, IGO r=.35, SEL r=.29) classes missed (r= -.28), course rating (r=.27) Reliabilities low on Rehearsal and Metacognitive Self-regulation

11. 4. Pilot Test Results by Site – Dominican University 3 Psychology courses (48 Students, respondents=27) Demographics: 83% White; 9% Latino; 4% Other Mean age 22.78 (std. dev. = 5.0) 100% full-time students 26% work 1-10 hrs, 26% 11-20 hrs, 22% 21-30 hrs, 13% more than 40 hrs (only 13% DO NOT work outside of class) Graduated from college: Mothers 39%; Fathers 26% Findings: Average # of hours per week studying for class = 4.3 (std dev =1.8) 93% had missed two or fewer classes Correlations with class performance: Motivation: Task value (r=.47), self efficacy for learning (.58*) Learning Strategies: Elaboration (.39), critical thinking (.51), metacognitive self regulation (.46) Resource Management Strategies: Time and study environment (.68*), effort regulation (.37) Instructor rating

12. Contact Information: Rick D. Axelson, PhD (rick-axelson@uiowa.edu, (319-384-4291); starting January 2012 (axelson@carnegiefoundation.org) rick-axelson@uiowa.eduaxelson@carnegiefoundation.orgrick-axelson@uiowa.eduaxelson@carnegiefoundation.org Arend Flick, PhD (arend.flick@rcc.edu, (951-372-7028) arend.flick@rcc.edu Marc A. Pizzimenti, PhD (marc-pizzimenti@uiowa.edu, (319-384- 4644) marc-pizzimenti@uiowa.edu Tina Taylor-Ritzler, PhD (tritzler@dom.edu) (708-524-6608) tritzler@dom.edu