1. The University of SydneyPage 1 Smart Sparrow: Using learning analytics to measure student engagement in case based learning Presented by Dr Vanessa Hughes Sydney Nursing School

2. The University of SydneyPage 2 Smart Sparrow Case Based Studies Cohort: First Year Bachelor of Nursing Advanced Studies Students Nursing focus: Clinical notes, family/social history and diagnosis is provided Content: Focuses on connecting the signs/symptoms observed with the underlying pathophysiology

3. The University of SydneyPage 3 The Evolution of Online Case Based Learning in NURS1005 2012 201320142015 Passive PDF Online module + videos + PDF worksheet Smart Sparrow Adaptive Learning Smart Sparrow technology has made it possible to: embed videos into the lesson create lessons that adapt to the students responses create an active learning environment test student knowledge and understanding

4. The University of SydneyPage 4 Purpose of the Online Case Study Lesson – Replaces 1 hour of face to face teaching – Covers important content in greater detail – Connects bioscience theory with clinical practice – Students receive formative feedback that enables them to: a.Check their answers b.Understand the level of detail and knowledge expected c.Practice answering similar types of questions to the ones included in their Pathophysiology Workbook Assessment

5. The University of SydneyPage 5 1. Trigger Video 2. Clinical Notes 3. MCQ or Short Answer Question 3. MCQ or Short Answer Question Model Answer Guidance Correct Answer Incorrect Answer

6. The University of SydneyPage 6

7. The University of SydneyPage 7

8. The University of SydneyPage 8 Using Learning Analytics to Improve Learning and Teaching – How long did students spend watching each video? Did they watch all of it? – Were the questions too hard/easy? What needs to be clarified? – What are the concepts that all students struggle with? What can I do to facilitate their understanding of the content? – How long did each exercise take? Are they disengaging at some point? – I would never have been able to easily answer any of these questions without Smart Sparrow Analytics.

9. The University of SydneyPage 9 Lesson Completion Semester 2 Week 1Week 2 96% of students attempted the lesson 85% of students attempted the lesson

10. The University of SydneyPage 10 Simple overview of how students engaged with the activity

11. The University of SydneyPage 11 Anonymous Participation Students Enrolled in Smart Sparrow 49% completed the activity 94% completed the activity

12. The University of SydneyPage 12 Ability to Track Learning Pathways

13. The University of SydneyPage 13 Short Answer Responses

14. The University of SydneyPage 14 Student Feedback (1= Strongly disagree, 5= Strongly Agree) Week 1 (n=179)Week 2 (n=154) The videos in this module helped me to better understand the content. 4.11 I feel that completing this module will help me complete my pathophysiology workbook. 4.064.05 I feel that the feedback provided encouraged me or helped me find the answer when I made a mistake. 4.274.28 The model answers provided helped me understand what is expected of me and helped me understand what I needed to know about this topic. 4.544.49

15. The University of SydneyPage 15 Using Analytics to Track Student Performance Using the analytics students can be categorised as: –No risk –Low risk –High risk This data can be used to: –send personalised emails to students –identify students for follow up using the Track & Connect program

16. The University of SydneyPage 16 Using analytics to provide personalised student feedback Students are sent a personalised email (weekly) (via SRES) based on their risk category.  No risk – congratulations on completing all activities  Low risk – encouragement to improve  High risk – encouragement to improve, support and advice Students at risk are emailed with their advisors details as well as the contact details for other university support services. The email also contains links to additional resources that address common misconceptions found in the short answer responses. This email feedback system has lead to several student problems being identified by the end of week 3. It is hoped that this system will lead to a reduction in absent fails and enable students who are falling behind to get the help they need earlier in the semester.

17. The University of SydneyPage 17 Important Points – Numbers only tell half the story: a.Use a combination of MCQ and short answer questions. b.By quickly scanning short answer responses you can identify common misconceptions. – Start small and build on the lesson. – Keep it simple, too many trap states and pathways can lead to errors. – Guidance to correct mistakes can be simple e.g. a recommended reading, video etc. If you have not used SRES (Student Relationship and Engagement System), you will need to contact Adam Bridgeman (adam.bridgeman@sydney.edu.au) to be registered as a user.