1. CAPHIA TEACHING AND LEARNING FORUM 18-19 September 2014, Perth Jane Taylor Public Health Program University of the Sunshine Coast Evaluating flipping: Development of the Flipped Classroom Student Engagement Questionnaire (FCSEQ)

2. Flipped team Jane Taylor Public Health Course Coordinator & Evaluator Rachel Cole Public Health Course Coordinator & Evaluator Mary Kynn Statistician & Movie Maker Julie-Anne Foster Paramedic Science Course Coordinator Bec Tretheway Project Research Assistant Kara Lilly Public Health Course Coordinator 18/09/2014Jane Taylor, University of the Sunshine Coast Exploratory Learning and Teaching Grant, 2013 Centre for Support and Advancement of Learning and Teaching, USC FCSEQ Movie

3. Overview A bit about flipping Research aim and process Flipped Classroom Students Engagement Questionnaire (FCSEQ) Some very preliminary (limited) findings Comments & questions 18/09/2014Jane Taylor, University of the Sunshine Coast

4. What is a flipped classroom? Pre-class online learning activities Face-to-face workshops Post class learning activities 18/09/2014Jane Taylor, University of the Sunshine Coast

5. http://ctl.utexas.edu/teaching/flipping_a_class/what_is_flipped Design foundations Bloom’s taxonomy SOLO taxonomy Introduction of concepts Knowledge Comprehensi on Application Pre- structural, Uni- structural, Multi- structural Clarification and exploration of concepts Application Analysis Synthesis Evaluation Multi- structural, Relational Consolidation of concepts through assessment Application Analysis Synthesis Evaluation Relational, Extended abstract 18/09/2014Jane Taylor, University of the Sunshine Coast

6. Why flip a classroom? Constructivist p edagogy Student centred Active and problem based learning strategies Emphasis on student responsibility for learning Facilitator role of teacher Students Learn more deeply Greater participation Increase in interaction and learning from one another Greater level of instructor feedback More time on higher levels of learning e.g., analysis, synthesis and evaluation 18/09/2014Jane Taylor, University of the Sunshine Coast

7. Flipped project overview 2013, S1: Academic team foci2013, S2: Test run in 3 classes, process feedback 2014, S1&2: Pilot in 10 classes (about 300 students) evaluation Evaluation: Informal in class feedback, survey end course, discussion groups end course, SETAC 18/09/2014 Jane Taylor, University of the Sunshine Coast

8. Why this research? Limited literature on assessing and monitoring impacts of flipped classroom approach on student engagement and learning Engagement = involvement in purposeful / meaningful academic activities (Kuh, 2001) Aim: To develop and pilot an impact evaluation questionnaire called the Flipped Classroom Student Engagement Questionnaire, to evaluate the impact of the flipped classroom on student engagement in learning 18/09/2014Jane Taylor, University of the Sunshine Coast

9. Flipped Classroom Student Engagement Questionnaire Aim To assess students’ overall level of engagement at the course level, and in pre, in class and post class flipped classroom components Development 1.Item generation – literature review, existing instruments and items, peer-review, pilot test 2.Validation (currently here) 3.Finalisation 18/09/2014Jane Taylor, University of the Sunshine Coast

10. 1. Item generation Literature review – Engagement measures at course level – Handelsman et al., 2005; Delialioglu, 2012 – National Survey of Student Engagement (NSSE), 2011 Student engagement constructs & indicators – Skills, Emotional, Participation and interaction, Performance (Handelsman et al) – Active and collaborative learning (Delialioglu, 2012; NSSE) – Time on task (Delialioglu, 2012) – Overall level of engagement (Handelsman, et al., 2005) – Confounding variables 18/09/2014Jane Taylor, University of the Sunshine Coast

11. Construct / Indicators Students …Examples Skills engagementStudy behavioursKeeping up to date with readings, taking notes in class, and coming to each scheduled class Participation/interac tion engagement Participation in class and interact ion with the lecturer/tutor and fellow students Ask questions if they don’t understand the lecturer/tutor; participate in small group discussions and overall class discussions and assisted fellow students Emotional engagement Connection with the courseChoosing to make the course material relevant to their lives; applying the course material to their lives; finding ways to make the course interesting; thinking about the course between classes; and really desiring to learn the material. Performance engagement Levels of performance towards getting a good grade Doing well in assessment tasks and asking questions, confidence that they can learn and do well in the course Global engagementLevels of engagement between courses Flipped compared to traditional Active and collaborative learning Frequency of particular learning strategies in and out of the classroom Asking questions, completing pre-class activities, assisted other students, asked for assistance from other students, and worked with other students on course material Total time on taskSum of time in hours doing specific learning activities Pre-class activities, watching e-lectures, studying, reading and writing Jane Taylor, University of the Sunshine Coast

12. Survey examples Skills engagement Participation/interaction engagement Emotional engagement Performance Active and collaborative learning 18/09/2014Jane Taylor, University of the Sunshine Coast

13. Pre-testing instrument Peer-review – Steering Committee – Two additional academics Pilot test – OT class using flipped – Interpretability, usability, timeliness… 18/09/2014Jane Taylor, University of the Sunshine Coast

14. 1. Item generation Confounders (Handelsman et al) – Entity theory of learning Learning can be extended and is not a fixed entity (Handelsman et al) – Incremental theory of learning Certain amount of intelligence and you can’t do much to change it (Handelsman et al) – Socio-demographic characteristics Age, sex, employment, level of education striving for; level of parent education; international/domestic student; living arrangements, access to internet at home, confidence in using technology (NSSE) Assessment of flipped approach compared with usual/traditional Perception of impact of flipped approach on academic performance, non-academic skills, working in groups, preferences 18/09/2014Jane Taylor, University of the Sunshine Coast

15. FCSEQ structure Four sections 1.Overall engagement 2.Participation in course 3.Achievement 4.Socio-demographic 19 closed-answer questions 2 open-ended questions 10 minutes Administration hard copy or electronic 18/09/2014Jane Taylor, University of the Sunshine Coast

16. 2. Validation Administration: four public health courses, one paramedic course Electronic and hard copy administration June-August 2014 18/09/2014Jane Taylor, University of the Sunshine Coast

17. 2. Validation Factor analysis may collapse one or more constructs – in progress How do these constructs then align with other student attributes and measures of performance? 18/09/2014Jane Taylor, University of the Sunshine Coast

18. 3. Finalisation & next steps Finalise instrument by end October 2014 Recruitment of more flipped classrooms by end January 2015 Full implementation 2015 Snippets from public health courses Sem 1 2014 18/09/2014Jane Taylor, University of the Sunshine Coast

19. Overall engagement in course across semester? n=51 18/09/2014Jane Taylor, University of the Sunshine Coast Most engaged

20. Overall flipped or standard classroom? n=32 18/09/2014Jane Taylor, University of the Sunshine Coast Most prefer flipped

21. Overall level of engagement? n=51 18/09/2014Jane Taylor, University of the Sunshine Coast Most more engaged

22. Overall academic performance? n=50 18/09/2014Jane Taylor, University of the Sunshine Coast Some felt it do, others not sure, some not

23. Pre class Doing pre class activities? (Active & collaborative learning) n=52 18/09/2014Jane Taylor, University of the Sunshine Coast Characteristic of most to do something

24. Pre class Hours per week preparing for workshops (Total time on task) n=51 18/09/2014Jane Taylor, University of the Sunshine Coast Majority 1-5 hours

25. In-class Workshop attendance? (Skills engagement) n=52 18/09/2014Jane Taylor, University of the Sunshine Coast Most characteristic to attend

26. In-class Workshops without pre-class activities? (Active & collaborative learning) n=49 18/09/2014Jane Taylor, University of the Sunshine Coast Some do, some don’t

27. In-class Having fun in workshops? (Participation / interaction engagement) n=52 18/09/2014Jane Taylor, University of the Sunshine Coast Majority having fun

28. In-class Active participation on small groups discussion? (Participation /interaction engagement) n=52 18/09/2014Jane Taylor, University of the Sunshine Coast Majority small group discussions characteristic

29. In-class Explaining materials to others? (Active & collaborative learning) n=50 18/09/2014Jane Taylor, University of the Sunshine Coast Majority sometime/ often explained course materials to others

30. Post class Making course material relevant to life? (Active & collaborative learning) n=52 18/09/2014Jane Taylor, University of the Sunshine Coast Majority found ways to make course material relevant to life

31. Post class Read additional materials (Active & collaborative learning) 18/09/2014Jane Taylor, University of the Sunshine Coast Majority read additional material related to course content

32. FCSEQ qualitative feedback Think more engaged? Interaction in class (n=9) Having a supportive class learning environment (n=7) Pre-class activities assisting in preparing for class (n=4) Supported engagement? Getting to know teachers (n=4) More in-depth learning and expanding on content (n=3) Ensuring on-track or same page (n=3) Getting to know/ learning from fellow students (n=3) Being responsible for/being in control of own learning (n=2) ×Not like pre-class activities (n=2) ×3 hour workshop too long (n=1) 18/09/2014Jane Taylor, University of the Sunshine Coast

33. Comments & Questions 18/09/2014Jane Taylor, University of the Sunshine Coast

34. Key References Australian Council for Educational Research. Australasian survey of student engagement: Student Engagement Questionnaire. Australia, 2011. Handelsman M, Briggs W, Sullivan N & Towler A. A measure of college student course engagement. The Journal of Educational Research 2005;98(3): 184-192. Delialioglu O. Student Engagement in Blended Learning Environments with Lecture-Based and Problem-Based Instructional Approaches. Journal of Educational Technology & Society 2012;15(3): 310. Biggs J. & Collis K. Evaluating the quality of learning: The SOLO Taxonomy. New York: Academic Press, 1982. Krathwohl D. A revision of Bloom's taxonomy: An overview. Theory into Practice. 2002;41(4): 212. 18/09/2014Jane Taylor, University of the Sunshine Coast