The Open University's Institute of Educational Technology Evaluation of mobile learning Agnes Kukulska-Hulme, 4 July 2009, Brest source:

The Open University's Institute of Educational Technology Session overview 1.Introduction and sources 2.Approaches to evaluation: why and how to evaluate 3.Common problems and weaknesses – drawing on experience of PhD thesis examinations and reviewing 4.Your aspirations and issues 5.Usability evaluation 6.Wrap-up/ summary of key points

The Open University's Institute of Educational Technology Introduction and sources Tradition of CAL/TEL evaluation at The Open University, UK – supporting distance learners –Questionnaires –Telephone interviews – interviews –Focus groups –Observations captured on video –Eye tracking –Accessibility for learners with disabilities

The Open University's Institute of Educational Technology Mobile learning evaluation sources Traxler, J. & Kukulska-Hulme, A. (2005) Evaluating Mobile Learning: Reflections on Current Practice. In Proceedings of mLearn 2005: The future of learning in your hands, October Traxler, J. & Kukulska-Hulme, A. (2006) The evaluation of next generation learning technologies: the case of mobile learning. In ALT-C 2006: The Next Generation, Research Proceedings, 5-7 Sept Section in Big Issues in Mobile Learning, 2006 (ed Sharples): “What are appropriate methods for evaluating learning in mobile environments”? ES_REPORT_PUBLISHED.pdf ES_REPORT_PUBLISHED.pdf

The Open University's Institute of Educational Technology Mobile learning evaluation sources Chapter on evaluation methods by Mike Sharples in the book Researching Mobile Learning, Peter Lang Verlag, 2009 (eds Vavoula, Pachler & Kukulska-Hulme) Recent article by Vavoula, G. & Sharples, M. (2009) Meeting the Challenges in Evaluating Mobile Learning: A 3-level Evaluation Framework. IJMBL, 1(2).

The Open University's Institute of Educational Technology Evaluation of mobile usability Book chapter: Kukulska-Hulme, A. (2005) Mobile Usability and User Experience In: Kukulska-Hulme, Agnes and Traxler, John eds. Mobile Learning: A handbook for educators and trainers. London, Routledge, pp. 45–56. Journal article in IRRODL: Kukulska-Hulme, A. (2007) Mobile Usability in Educational Contexts: What have we learnt? International Review of Research in Open and Distance Learning, 8(2), pp. 1– Book chapter: Kukulska-Hulme, A. (2008) Human Factors and Innovation with Mobile Devices. In: Hansson, T. ed. Handbook of Research on Digital Information Technologies. IGI Global, pp. 392–

The Open University's Institute of Educational Technology Meanings/purposes of ‘evaluation’ Assessment –tests, grading achievement Comparison –with traditional means/methods of learning Development –formative evaluation, piloting Demonstrating the value of an innovation –Value for money, educational value

The Open University's Institute of Educational Technology What are you going to evaluate? “An analysis of 12 international case studies in Kukulska-Hulme & Traxler (2005) reveals that reasons given for using mobile technologies in teaching and learning relate principally to: improving access exploring changes in teaching and learning alignment with institutional or business aims” From: Traxler, J & Kukulska-Hulme, A. (2006) The Evaluation of Next Generation Learning Technologies: the Case of Mobile Learning

The Open University's Institute of Educational Technology Vavoula, G. & Sharples, M. (2009) Meeting the Challenges in Evaluating Mobile Learning: A 3-level Evaluation Framework. IJMBL. Propose a three-level framework for evaluating mobile learning: –a micro level concerned with usability –a meso level concerned with the learning experience –a macro level concerned with integration within existing educational and organisational contexts.

The Open University's Institute of Educational Technology What do you want to know? (chapter by Sharples, 2009, in Researching Mobile Learning) Usability: will it work? Effectiveness: is it enhancing learning? Satisfaction: is it liked? Case studies: Mobile learning organiser (higher education), MyArtSpace (school/museum learning), Personal Inquiry (school/indoor and outdoor learning)

The Open University's Institute of Educational Technology MyArtSpace case study Micro level: examined individual activities, such as making notes, recording audio, viewing the collection online, and producing presentations of the visit. Meso level: examined the learning experience as a whole, exploring whether the classroom-museum-classroom continuity worked. Macro level: examined the impact of MyArtSpace on educational practice for school museum visits. For each level, the evaluation covered three stages: Stage 1: what was supposed to happen, based on pre-interviews with stakeholders and documentation Stage 2: what actually happened, based on observer logs, focus groups, and post-analysis of video diaries Stage 3: the gaps between findings from stages 1 and 2, based on reflective interviews with stakeholders and critical incident analysis of the findings from stages 1 and 2

The Open University's Institute of Educational Technology What is the value of mobile learning? How does mobility change learning? How and why is the technology used outside of the main activity? How is it used in conjunction with other technologies? How does it fit in with the user’s work life or lifestyle? Does its use evolve over time? Does it have a lasting impact? Who is involved in the learning experience?

The Open University's Institute of Educational Technology Taylor (2006) in Big Issues in Mobile Learning “Users… may find themselves fascinated by the new devices in a way which they may find interesting, and even fun, but which produces no lasting valuable impact on their work practices” “Our potential subjects of study may be wandering around studying things that interest them, at times that suit themselves, with little or no concern for consistency”

The Open University's Institute of Educational Technology Taylor (2006) in Big Issues in Mobile Learning “The current emphasis on ‘activity’ has stimulated a great deal of evaluation effort focused around Activity Theory, where, for example, emphasis is placed on historical perspectives (learners’ past experiences) and how these relate to the technology to be evaluated” “The key issue is to realise that evaluation may entail the peeling back of multiple layers of activity, and the need to keep track of what has been peeled and what hasn’t is essential”

The Open University's Institute of Educational Technology Taylor (2006) in Big Issues in Mobile Learning “It is essential to understand how structured learning activities blend with other more social or informal activities” “Traditionally, evaluators might relate the success of a design to the success with which learners can achieve pre-identified learning outcomes. The nature of learning outcomes in the mobile age needs to be adaptive. For example, they may relate to the extent to which someone has assimilated information into their own experience and development”

The Open University's Institute of Educational Technology How will you evaluate? Questionnaires were often used; interviews, focus groups and observations were used less frequently. Accounts of interviews and focus groups were brief and suggested that the sessions were usually short and probably informal. Statistical analysis and system data were both used infrequently. In a relative small number of cases, evaluation used and combined several of these different techniques. From: Traxler, J & Kukulska-Hulme, A. (2006) The Evaluation of Next Generation Learning Technologies: the Case of Mobile Learning

The Open University's Institute of Educational Technology Technologies for data capture Audio logs Questions by sms or Discussion forum Automatic logging of interactions Video recording Eye tracking Pen and paper

The Open University's Institute of Educational Technology Participants as Researchers; ethical issues (Vavoula, G., mLearn’08 workshop) Automatic logging of interactions with technology –System logs enabled on personal device Carrying or wearing data-capture equipment –Wear mobile eye-trackers, camcorders, carry microphones Manually record learning experience –Keep diaries, blogging involvement

The Open University's Institute of Educational Technology Mobile Learning Research Ethics (Vavoula, G., mLearn’08 workshop) Mobile technology translates (most often) to personal technology –Are learners willing to be monitored? How much of their privacy will they unveil? What if they’re under-age? –Is it OK to monitor everything? How much do we really need to know? –Even if they agree, is it easy to safeguard personal data? What are best dissemination practices? –Will users cooperate in practice? E.g. synchronise as and when needed? Informed consent –Can we really inform accurately? –Can they really consent unreservedly? –How do we deal with late withdrawals? Ethical mobile learning practice –Participants as co-researchers

The Open University's Institute of Educational Technology Common problems and weaknesses (based on examinations and reviews) 1.Lack of clear definitions ‘practices’, ‘contexts’, ‘collaboration’, ‘community’… 2.Evidence from literature comes from very different cultural settings/contexts Japan, Finland… 3.Convenience samples used badly Target users poorly defined; skewed samples 4.Ignoring minority views ‘Most people were happy’

The Open University's Institute of Educational Technology Vavoula, G. & Sharples, M. (2009) Meeting the Challenges in Evaluating Mobile Learning: A 3- level Evaluation Framework. IJMBL. Propose six challenges in evaluating mobile learning: 1.capturing and analysing learning in context/across contexts 2.measuring mobile learning processes and outcomes 3.respecting learner/participant privacy 4.assessing mobile technology utility and usability 5.considering the wider organisational and socio-cultural context of learning 6.assessing informality

The Open University's Institute of Educational Technology Your aspirations and issues STEP 1 What are you intending to evaluate? What challenges will you face? What methods will you use?

The Open University's Institute of Educational Technology Your aspirations and issues STEP 2 Grand Designs for Mobile Learning (pre-conference workshop at mLearn’09 in October) Consider how your plans for introducing or extending mobile learning relate to broader social and educational agendas and global challenges, e.g.: fostering creativity and innovation, social inclusion, cross-cultural understanding, reading literacy, lifelong learning, bridging the digital divide…

The Open University's Institute of Educational Technology Evaluation of mobile usability A synthesis of usability issues across a range of mobile learning projects (Kukulska-Hulme, 2007) identified four main categories: the physical attributes of mobile devices, e.g. size, weight, memory, battery life content and software applications network speed and reliability the physical environments of use

The Open University's Institute of Educational Technology Evaluation of mobile usability The 4 categories relate to six aspects of mobile learning: the learner other people tasks engaged in the device being used networks or connectivity the locations where learning happens

Factors impacting on the usability of mobile devices in education (Kukulska-Hulme, 2007)

The Open University's Institute of Educational Technology Planning your evaluation The user (learner): How self-motivated are the learners? How familiar are learners with all features of their device? What reward comes from mobile learning? Can the device be adapted and personalised to suit specific needs? Will it fit with lifestyle? Other people: Who can support the learners on the go? What spontaneous or pre-planned collaboration can take place? What communities can learners be part of? What do others interacting with the mobile learner need? Tasks engaged in: Are there tasks set by instructors/tutors or learner-generated tasks? Does speed of network access have an impact on the tasks? Are the tasks confined to the mobile device or do they connect with other environments and tools? Do interruptions matter?

The Open University's Institute of Educational Technology Planning your evaluation Device being used: What input devices and other accessories are available? How long will learners keep the device (if not owned)? Is there compatibility or conflict with other tools being used? Connectivity/networks used: Are wireless networks reliable? Can learners manage to get connected? Are they dependent on connectivity? What are the costs involved, if any? Locations of use: Are they suitable for the type of learning envisaged? Is quiet or privacy available if required? Is there continuity of use across different locations? Can location-based context be incorporated into the learning experience?

The Open University's Institute of Educational Technology Wrap-up/ questions/ summary of key points

Institute of Educational Technology The Open University Milton Keynes United Kingdom Thank you!