Issues in Categorising Learning Designs HELEN WALMSLEY-SMITH STAFFORDSHIRE UNIVERSITY

Overview Reasons for categorising learning activities Taxonomies of learning activity types Using taxonomies to categorise learning activities Development of eDAT based on eDesign Template Content analysis and Inter-rater reliability Updated eDAT and consultation activity Discussion

Why Categorise? Representation, comparison, evaluation, sharing…

Existing Taxonomies Ulster: receives, debates, experiments, creates, explores, practices, imitates and meta-learns Laurillard: Acquisition, Discussion, Investigation, Practice, Collaboration, Production, JISC Pedagogical Vocabularies Project reports identified that there is a ‘lack of common understanding of, and shared vocabularies for, pedagogical practice amongst teachers, system developers, learning technologists and e-learning researchers.’ Mod4L project identified issues with representation of learning design

Using taxonomies to categorise Examples of issues: ◦Exercise using the Learning Designer to map activities to teaching and learning activity types illustrated difficulties ◦AMP (Assessing MOOC Pedagogies) involved extensive group discussion to reach agreement on categorisation of MOOC ◦Open University programme of mapping activities to types also found the process challenging

Initial eDAT Based on principles in eDesign Template: ◦Open / closed activities ◦Tutor managed / student managed ◦Variety of interaction ◦Variety of assessment

Content Analysis Process aims to increase validity: ◦Codebook or categories created - terminology ◦Text analysis units determined – ‘activity segment’ ◦Raters trained in use of categories ◦Each rater looks at all learning activities (or sufficient overlap) ◦Raters work independently ◦Inter-rater reliability measured

Inter-rater Reliability Analysis using Krippendorff’s Alpha: Unlike percent agreement and Cohen’s Kappa, Krippendorff’s alpha is a measure of the disagreements in a set of data Where Do = observed disagreement and De = expected disagreement This takes chance into account as well as the magnitude of the misses. In addition, it applies to any number of observers, any number of categories, any metric or level of measurement, incomplete or missing data and large and small sample sizes alike. Acceptable levels of IRR =.6 to.8

eDAT in development… Updated following pilots Activity: Look at eDAT and sample activities Review categories Try categorising independently Compare and discuss

Bibliography Charlton, P., Magoulas, G., & Laurillard, D. (2012). Enabling creative learning design through semantic technologies. Technology, Pedagogy and Education, 21(2), 231–253. Currier, S., Macneill, S., Corley, L., Campbell, L. M., & Beetham, H. (2006). Vocabularies for Describing Pedagogical Approach in e-Learning: A Scoping Study. In DCMI ’06 Proceedings of the 2006 international conference on Dublin Core and Metadata Applications: metadata for knowledge and learning. Falconer, I., Beetham, H., Oliver, R., Lockyer, L., & Littlejohn, A. (2007). Mod4L Final Report: Representing Learning Designs. Retrieved February 15, 2016, from Krippendorff, K. (2011). Computing Krippendorff’s Alpha-Reliability. Philadelphia: Annenberg School for Communication Departmental Papers. Laurillard, D. (2012). Teaching as a Design Science: Building Pedagogical Patterns for Learning and Technology. Abingdon, UK: Routledge, Taylor & Francis Group.

Rienties, B., Toetenel, L., & Bryan, A. (2015). “Scaling up” learning design: impact of learning design activities on LMS behavior and performance. In Proceedings of the Fifth International Conference on Learning Analytics And Knowledge - LAK ’15. Swan, K., Day, S., Bogle, L., & Prooyen, T. van. (2015). Metaphors for Learning and the Pedagogies of MOOCs. In 2015 Meeting of the American Educational Research Association, Chicago, IL. University of Ulster. (2008). Hybrid Learning Model. Retrieved March 20, 2014, from Walmsley, H. (2015). The e-Design Template: A Pedagogic Guide for e-Learning Designers. In M. Maina, B. Craft, & Y. Mor (Eds.), The Art & Science of Learning Design. Rotterdam: Sense Publishers.