From Incidental Learning Analytics Research to a Sustainable Learning Analytics Program Stefan T. Mol University of Amsterdam

UvA is currently here UvA Siemens et al. 2014

The UvAInform Project - History Initiated as a proposal from the ICTS Department Expertise group Education (EGO) Reserved budget of 150K EGO dislike of (bottom-up) tender procedure with limited strategic vision Focus Group Learning Analytics Established Late 2012 UvAInform proposal approved in June 2013 Central infrastructure (LRS) (De?)centralized pilots

UvAInform Process Focus Group: evaluation criteria Definition of initial pilots Discussion with an external expert (Erik Duval) Challenge 1 – how to agree on the separate pilot(s) Dilemma – To LRS or not to LRS Resolution - Definition of LRS as separate project All faculties present their research ideas Challenge 2 – How to agree on pilot requirements and budgets Resolution: 7 small pilots in 3 pilot clusters Challenge 3 – How to manage ethics & privacy Resolution – Faculty level ethics approval / Data Governance center

Learning Record Store Community sourced, secure, scalable repository/infrastructure Store and retrieve statement data reliably and ensures a good scalable storage layer for various types of data and data streams Scales above 100 billion records. These data can be made available in a secure and consistent way for further analysis. Upon this LRS infrastructure dashboards can be built or developed for the delivery of (analysed) data to students, educators, and researchers

Cluster 1: Mirroring of traditional and non- traditional study performance to students COACH2 - FNWI Visualize the position of individuals in the context of the group using BB data Using positioning of individual student to determine support from teaching staff. Cluster Exam Feedback (qDNA - FMG) More fine grained mirroring of exam results to provide students and teachers with insight in the development on four competencies (interpretation, analysis, evaluation, inference) and knowledge goals

Cluster 1: Mirroring of traditional and non- traditional study performance to students Identifying Types of Effective Comparative Feedback and Relevant Mediators (FMG)

Cluster 1: Mirroring of traditional and non- traditional study performance to students Goal setting in education (FEB) Students are instructed/ to formulate goals in a concrete manner such that they are Specific, Measurable, Attainable, Relevant and Time-bound (SMART). Dashboard facilitates individual students to choose from, and set their own goals (and deadlines) against specific course events Dashboard shows students how they are scoring/succeeding in attaining the goals compared to their fellows

Cluster 2: Using specific student data to provide feedback for teachers Learning Analytics Pilot “Turnitin” – Assignments Criteria Validation (FMG) Visualization of Turnitin scores for students and teachers over (partial) assignments or courses Do the grading schemes used for the evaluation of written assignments hold any validity, i.e. do they correlate with overall study results? Teacher insight into Weblectures (FMG) Visualization of use of video’s/weblectures to find potential problems, difficult material to improve teaching, provide better remedial material

Cluster 3: Using other people’s data to provide recommendation system to students Validating Learning Analytics in Higher Ed. (FGW) Determine the predictive validity of demographic data, learning styles, motivation, and behavior to optimize prediction of study success Developing interventions (recommendations based) based on evidence

Positioning a learning analytics project Centralized vs. decentralized Research vs. practice Imposed vs. desired Technology vs. Pedagogy Make or buy (resource oriented) Generic infrastructure vs. pilot specific infrastructure in relation to UvA learning analytics lifecycle Adoption of best practices or local identification thereof

Lessons Learned UvAInform Top down approach currently bridge too far Yet… Need for generic infrastructure/standards Need for access to data Need for large scale experimentation International collaboration Data governance

Towards a Center for Data Governance and Innovation Supporting learning analytics (and other big data initiatives) Evaluating/approving specific learning analytics projects, ensuring ethical and legal compliance, Centralization of knowledge Streamlining policies (e.g. user agreements) so as to facilitate Learning analytics (and other big data Initiatives) Facilitating communications amongst key stakeholder Managing public relations Later: Establishing decision trees

Working scenario – LA Sandbox Realize cheap centralized data warehouse for one educational program with full ethics clearance and student informed consent Experiment with Open dashboard and different requirements Data governance Randomized controlled trials (dashboards/LMS’s/interventions, etc.) Continuous delivery (adding data sources) Intersections with blended learning/digital testing…

Q&A