1. Assessing Student Engagement in the Classroom Rick D. Axelson, PhD Assistant Professor, University of Iowa, Carver College of Medicine Arend Flick, PhD Professor of English, Norco College Annual Conference of the American Evaluation Association San Antonio, Texas; November 11, 2010

2. Presentation Outline 1. What is Engagement? 2. Accountability vs Educ Research 3. Classroom-based Survey 4. Pilot Testing of the Survey  Norco College, CA  University of Iowa, Carver College of Medicine

3. 1. What is Engagement? Increasingly Important Concept – student engagement a key to improving education (perception that students are less engaged) Increasingly Important Concept – student engagement a key to improving education (perception that students are less engaged) Everyday language: commitment, involvement, participation Everyday language: commitment, involvement, participation Educational Accountability: involvement of students in good educational practices at their institutions (NSSE, CSEQ) Educational Accountability: involvement of students in good educational practices at their institutions (NSSE, CSEQ) Educational Research Literature: connections between students and their learning environments: Educational Research Literature: connections between students and their learning environments:  Behavioral – time-on-task, physical involvement, participation in activities  Cognitive- interest, attention, meaning, effort  Affective –bonds with teachers, classmates – affecting effort, integration, attachment, commitment to institution Meta-construct involving all three components (Fredricks, Blumenfeld, Paris, 2004) Meta-construct involving all three components (Fredricks, Blumenfeld, Paris, 2004)

4. 2. Accountability vs. Educ Research To refine the definition of engagement, need to distinguish accountability from research usage of the term To refine the definition of engagement, need to distinguish accountability from research usage of the term Accountability vs.Educational research Measure:Aggregate indexVariable(s) in a learning process or task Level:Institutional-levelClassroom-level Focuses on:Overall amount Processes/mechanisms for of engagementbecoming engaged Use:CompareUnderstand/Diagnose Institutionsa learning process

5. 3. Classroom-based Survey Purposes: for instructors to use as a diagnostic tool (mid-, end-of-semester) for instructors to use as a diagnostic tool (mid-, end-of-semester) for researchers to study engagement processes for researchers to study engagement processes Definition: ‘ “learning task engagement” …refer[s] to students’ cognitive investment, active participation, and emotional engagement with specific learning tasks.’ (Chapman 2003) Survey Item Development: Theory/research literature –Motivated Strategies for Learning Questionnaire (MSLQ) (Pintrich et al, 1993); & psychological conditions of engagement at work (e.g., Kahn 1990) for affective engagement Theory/research literature –Motivated Strategies for Learning Questionnaire (MSLQ) (Pintrich et al, 1993); & psychological conditions of engagement at work (e.g., Kahn 1990) for affective engagement Instructors/teaching evaluations – Review material with instructors to see what information would be useful for designing/delivering course; adopted items from standard course evaluation forms Instructors/teaching evaluations – Review material with instructors to see what information would be useful for designing/delivering course; adopted items from standard course evaluation forms

6. 3. Survey Content Background/Context for Course: Background/Context for Course:  Expected benefits from course  Demographics, First-generation college, Current work & family situation Perceptions of Classroom environment & course: Perceptions of Classroom environment & course:  Instructor  Course organization, reading materials, pace of course, etc.  Classmates Engagement: Engagement:  Motivations and attitudes (MSLQ subsets)  Learning strategies used by students (MSLQ subsets)  Time & study environment mgt (MSLQ subsets)  Affective engagement & identity (conditions for psychological engagement at work, bonds/fit with class & course) Open ended: student comments about the course & suggested improvements Open ended: student comments about the course & suggested improvements

7. 4. Pilot-testing of Survey IRB Approval obtained from Norco College and UI IRB Approval obtained from Norco College and UI Sample: Sample:  Norco College – 3 courses (Business, English, Psychology), 10 instructors, ~ 200 students (77 respondents so far)  University of Iowa, College of Medicine – 1 class (anatomy), 1 instructor, 150 students (survey just distributed) Survey Administered via SurveyMonkey – offered to provide students with customized report on their learning strategies Survey Administered via SurveyMonkey – offered to provide students with customized report on their learning strategies Analysis: Analysis:  Descriptive results  Measurement (Rasch models to screen items/persons; CFA)  Relationships (Methods depend on the measurement analysis)

8. 4. Respondents from Norco College (n=77) Demographics Gender: 70% women, 30% men Gender: 70% women, 30% men Age: mean =26.5, standard dev. =10.5, (min=17, max=60) Age: mean =26.5, standard dev. =10.5, (min=17, max=60) Race/eth: 41% Latino, 12% Asian, 36% White, 3% African-American Race/eth: 41% Latino, 12% Asian, 36% White, 3% African-American Mother’s Ed: 53% HS grad or less, 40% some college or more Mother’s Ed: 53% HS grad or less, 40% some college or more Father’s Ed: 44% HS grad or less, 42% some college or more Father’s Ed: 44% HS grad or less, 42% some college or more Current Situation Outside work: 36% none, part-time 37%, full-time 27% Outside work: 36% none, part-time 37%, full-time 27% Class load: 50% part-time, 50% full-time students Class load: 50% part-time, 50% full-time students # of Children: 70% none, 20% 1-2 children, 10% 3+ children # of Children: 70% none, 20% 1-2 children, 10% 3+ children Responses by Course: English – 62%; Psychology – 20%; Business – 18% Responses by Course: English – 62%; Psychology – 20%; Business – 18%

9. 4. Initial Look at the Reliabilities of the Measures Background/Context for Course: Background/Context for Course:  Expected benefits from course (5 items, alpha =.84) Perceptions of Classroom Environment & Course: Perceptions of Classroom Environment & Course:  Instructor (11 items, alpha =.92)  Course organization (10 items, alpha =.92)  Classmates (8 items, alpha=.86) Engagement: Engagement:  Motivations and attitudes (5 MSLQ subscales)  Learning strategies used by students (5 MSLQ subscales)  Time & study environment mgt (4 MSLQ subscales)  Affective engagement & identity (9 items, alpha=.84)

10. 4. Reliabilities of the Measures (Cont’d) MSLQ Engagement Measures: MSLQ Engagement Measures:  Motivations and Attitudes: internal goal orientation, task value, control of learning beliefs, self-efficacy for learning, test anxiety  Learning Strategies: rehearsal, elaboration, organization, critical thinking, meta-cognitive self- regulation  Resource Management: time and study environment, effort regulation, peer learning, help- seeking Scales in gold font alpha >.73; Scales in white font alpha.73; Scales in white font alpha <.73 Underlined scales: alpha <.73 in MSLQ’s validation study

11. 4. Next Steps in Pilot Study Finish data collection Finish data collection Refine measures with Rasch model and Confirmatory Factor analyses: reduce number of items on new scales and examine the MSLQ scales (5 of 14 with low reliabilities) Refine measures with Rasch model and Confirmatory Factor analyses: reduce number of items on new scales and examine the MSLQ scales (5 of 14 with low reliabilities) (Does the same measurement model hold for Norco & UI students?) (Does the same measurement model hold for Norco & UI students?) Examine relationships between: 1) types of engagement and 2) engagement and other variables (e.g., students’ perception of classroom environment & course, expected benefits from the course) Examine relationships between: 1) types of engagement and 2) engagement and other variables (e.g., students’ perception of classroom environment & course, expected benefits from the course) Discuss results with instructors – useful information? Discuss results with instructors – useful information?

12. Contact Information: Rick D. Axelson, PhD (rick-axelson@uiowa.edu, (319-384-4291) rick-axelson@uiowa.edu Assistant Professor, University of Iowa, College of Medicine, Office of Consultation and Research in Medical Education Arend Flick, PhD (arend.flick@rcc.edu, (951-372-7028) arend.flick@rcc.edu Professor of English, Faculty Development Coordinator, Norco College