Measuring Engagement: Learning Analytics in Online Learning Dr. Griff Richards Thompson Rivers University Open Learning Electronic Kazan, 19 April 2011

60 o N 49 o N Distance Learning In Canada - Is growing, especially in K-12 public schools -15% of 2008 high school graduates in BC took at least one on-line course -Increase in “Blended Learning”, online activities for F2F courses. Kamloops, British Columbia

Learning Engagement The more a learner interacts with the content and with their peers about the content, the more they are likely to internalize it, and remember it... Richard Snow, 1980

Interaction Equivalency Theory (Garrison & Anderson) The quality of the interaction is more important than the source Course Content Instructor Other learners

Learning Engagement (Kuh) Learning engagement promotes student retention and academic success. (George Kuh) Engagement can be broadly measured for a campus using National Engagement School Survey Instrument (NESSI).

Learning Engagement (Kuh) Five Factors: 1. active and collaborative learning, 2. student-faculty interaction, 3. supportive campus environment, 4. enriching educational experiences, and 5. the level of academic challenge.

Engagement Online Same Five Factors: 1. active and collaborative learning, 2. student-faculty interaction, 3. supportive campus environment, 4. enriching educational experiences, and 5. the level of academic challenge. on line

Promoting Engagement= promoting learning 1.Collaborative learning activities: Small groups Positive Interdependence Individual responsibility to the group Group reporting Individual reflection Peer formative evaluation => no place to hide.

Online Engagement 2. Student – Faculty Interaction -> weekly pacing messages -> rapid response to questions -> re-broadcast of questions to all the class -> flexibility for individual needs -> online “presence” in forums

Online Engagement 3. Supportive Online Environment -> LMS – e.g. Moodle or Blackboard -> Clear expectations and timelines -> Facilitates interactions -> Provides data on student interactions with LMS -> Synchronous conferencing tools

Online Engagement 4. Enriching educational experiences -> Variety of learning activities -> Project based learning -> Authentic assessment -> Discovery and sharing -> Challenge in Zone of Proximal Development (Vygotsky)

Promoting Engagement 5. the level of academic challenge -> Clear but demanding standards -> Clear but specific instructions -> Peer expectations -> Building social capital in a learning community

Measuring Engagement Engagement is “action based on internal motivation” Can not directly measure engagement Must look for external traces, actions that indicate interest, and involvement Kuh – NESSI – sample every year of 100,000s of students -> statistical trends, but not individual as results come too late to correct problems!

Measuring Online Engagement Engagement is inferred from activity. Student interaction with Online systems leaves an “exhaust trail” of their activity Learning Analytics is not a statistical sample, it is all the data for all learners Questions: What patterns to look for? and How to interpret them?

LMS interaction data e.g. Moodle discussion Extract linked list Plot interactions in star map Example: Snapp

SNAPP Visualization

SNAPP shot of Welcome Students “engaged”

SNAPP shot of Welcome Individuals with lower engagement, 3 or less messages

Does Activity = Engagement ? Beer (2010) plotted total LMS interactions with academic grades. Does activity = engagement? Is Blackboard more engaging than Moodle? 

Activity Outside LMS As learners become engaged, they move to more engaging channels: , Elluminate, Skype This activity is not tracked by LMS. No data available.

Interpretation of Analytics Data patterns require verification Quantitative data requires interpretation --> make and test hypotheses --> create useful models When we measure something we risk changing it. – e.g. If learners know we count hits they may make meaningless hits to fool the system.

Analytics for Learners! The same analytics should be able to provide easy to understand information dashboards for students.

How can analytics improve online learning? Measuring something is the first step “Better to measure something than to measure nothing” (Scrivens) Need more data than page hits. We also need to ask learners about their experience, what worked, what needs improvement.

Confused feedback systems Over time multiple fix-it systems have crept in Steady enrolments Credible learning Happy learners

Analytics can help build Quality Systems Multi-layered system, Timely, Qualitative & Quantitative data Steady enrolments Credible learning Happy learners Strategic course redesign

Dynamic Evaluation Model (Richards & Devries) » PreparationConductReflection Design Facilitation Learning Analytics at the activity level

Dynamic Evaluation Model » PreparationConductReflection Design Facilitation Learning Timely feedback enables quick fixes

The Goldilocks Principle of Learning Analytics Too little Too much Just right Masha & Medved We need to find balance.

Thank you! Questions?

Measuring Engagement: Learning Analytics in Online Learning Dr. Griff Richards Thompson Rivers University Open Learning Electronic Kazan, 19 April 2011

4 places to improve on-line learning Course Content Instructor Other learners 1. Clearer design 2. Collaborative activities 3. Less lecture, more mentoring 4. More study skills