1. Moving into the mainstream: Researching the institutional introduction of EVS, from the realm of the enthusiasts to supporting the later adopters www.herts.ac.uk Amanda Jefferies School of Computer Science Marija Cubric Business School

2. Contents Background Literature review Methodology Preliminary findings Focus on staff views on technology adoption and use of EVS Conclusions

3. Background Major institutional investment by the university in 2010-11 (Assessment and Feedback project) 3,845 students form 8 academic schools issued with a personal handset from Jan 2011 JISC EEVS project (stand B), alongside iTEAM Strand A, started in September 2011 EEVS = Evaluating Electronic Voting Systems for Enhancing Student Experience EEVS Objectives To provide an up-to-date view of the student and staff experiences of EVS To identify a set of critical success factors for introducing and maintaining the use of EVS in support of an institutional assessment and feedback strategy. Focus on large-scale deployment and differences across multiple disciplines.

4. Literature Review: Use of EVS in HE Survey of literature on the introduction of EVS in HE 300+ papers on published between 1998 and 2011 A wide variety of writers : – Brown, Davis, Draper, d’Inverno, Kennedy, Nichol, Oliver, White et al – Our own early adopters :- Lorimer, Hilliard, Thornton and Willis (University of Hertfordshire) Most authors reported on local use in classes: to engage students, to encourage interactivity, to support constructivist approaches to teaching and learning in line with e.g. Chickering and Gamson’s guidelines. Lack of literature on institutional deployment, with the exception of: Twetten et al (2007) Successful Clicker Standardisation, Educause Quarterly 4 – Experience from EVS standardisation in four North American universities

5. Methodology: Data Exploratory pragmatic approach: Gaining understanding of the meanings from various stakeholders Focus on practical applied research, integrating different perspectives to help interpreting data (Sounders et al 2009) Mixed-method data collection: Qualitative data: students’ blogs, staff interviews, staff reports Quantitative data: students’ questionnaire, staff questionnaire Time horizon: Mainly cross-sectional (a ‘snapshot’ taken in 2011/12) with some elements of longitudinal study (students’ blogs over a period of 2-4 weeks). Sampling: Self-selection (voluntary student participants for questionnaires) Self-selection & purposive (voluntary student participants for blogs; further selection made to enable balanced representation across different schools) Purposive (heterogeneous): staff involved with or leading EVS adoption in different Schools

6. Methodology: Students’ survey design Survey launched in February 2012 Includes 27 questions spread across 9 sections Total number of responses N= 590 (across 11 different Schools) Response rate: 14.4%. (based on 4093 EVS handsets registered in June 2012 ) SectionSource Background Information Standard demographic questions Usage(Venkatesh et al, 2003 Ease of useBangor et al, 2008 Impact on LearningDraper & Brown 2002 Performance Expectancy Venkatesh et al, 2003 EngagementJISC Evaluation guidelines SatisfactionJISC Evaluation guidelines Operational procedures and funding iTEAM questions Free text comments

7. Methodology: Staff survey design Survey launched in June 2012 Includes 18 questions spread across 5 sections Total number of responses N= 88 (across 13 different Schools) Response rate: 5.9%. (based on 1500 academic staff registered on staffq list in June 2012) SectionSource Background Information Standard demographic questions Technology adoption(Venkatesh et al, 2003) Usage (Venkatesh et al, 2003) + iTEAM questions Effect on teaching practice JISC Evaluation guidelines Impact on LearningDraper & Brown 2002 Operational procedures and funding iTEAM questions Free text comments

8. Methodology: Students’ blogs: Participants Student blogs were completed in a variety of ways according to individual preferences : Word Files (46), online blog entries (14), Short video clips (11), Audio files (1) and submitted via a restricted access group area on StudyNet (UH MLE) 65 students registered for the web/ written blog of whom 27 were chosen to ensure a spread of gender and programme. One failed to complete their blog 7 Academic Schools: Business (6), Computer Science (7), Humanities (6), Law (4), Education (1), Sports management (1), Mathematics (1) Equal male and female numbers applied but ratios varied according to subjects 15M:11F All the bloggers had previously used EVS since at least January 2011 The reflection period varied between 2 and 4 weeks in the period Nov 11- Feb 12

9. Methodology: Student blogs: Original and Emerging Categories and Themes All CategoriesSub-categories (Themes)Total Pedagogy Examples of use,Interesting lectures, Subject suitability, Subject-specific EVS use, Improvements in use, Potential use, Learning 292 Formative use, Inadequate use, Misaligned expectations, Variety of assessment, Edutainment and more … UsagePrevious experience with EVS,Use by modules and levels,Frequency of use78 Technology Type of handset,Technical issues,EVS functions, 99 Changing channels Institut’l benefitsAttendance,Institutional motivation47 ExpectationsInitial thoughts,Institutional use of data37 Effort ExpectancyEase of use, Accessibility40 Facilitating conditions Training,Handsets distribution Technical competence of tutors, Tutor's experience with EVS use/question writing 56 Institutional deployment Handset ownership 32 Handset cost, Inconsistent application, Too ambitious implementation Social aspectsAnonymity28 Student experienceSatisfaction and Dissatisfaction21 Personal emotionsAnxiety, and extra pressure, privacy7

10. Methodology: Staff interviews: Participants and Developing Categories and Themes Original Categories Sub-categories (Themes) Technology Adoption Drivers for change, Institutional Support for technology adoption, Positive factors of EVS Negative factors of technology adoption Negative factors of EVS usage Embedding EVS successfully Personal Competence Training & Support at institutional level Support at local level Champions for EVS Benefits of Training Changing styles of Pedagogy Use of MCQs for learning and teaching Developing Question Banks Benefits of EVS for Student support Semi-structured interviews with 12 staff members incl: Local School Champions (2) Associate Heads and Heads of School (3) Lecturers (6) The iTEAM Project Director (1)

11. Preliminary Findings : Summary of students’ views (survey and blogs) Usability or ease of use of the EVS was rated very highly by students Very positive impact on students’ learning and satisfaction: Responding to questions makes me think more about the course material (80%+) EVS provides them with an immediate check of their understanding (80%+) Viewing responses give them an idea of how they are doing in relation to the rest of the class (80%+) Less positive impact on perceived performance and engagement : use of EVS will increase their chances of getting a good grade (49.9%) Spend more time discussing the subject with colleagues (47.5%) Thinks about the subject outside of the classroom (34%) Ask more questions in classes that use EVS (21%) Time given for answering questions is not always sufficient Formative use of EVS has been seen as adding a value to students’ learning experience, however summative use has in some cases created “an unfortunate tension between the administrators, the tutors and the students” (MC12) All students find the EVS very easy to use, with little or no training; however, accessibility might be an issue for students with sight or dexterity impairments. Majority agree that EVS is a great tool for answering QUESTIONS, but inadequate as a TEST tool

12. Preliminary Findings : Staff Survey There is an overall (UH-wide) highly positive attitude to technology (EVS) acceptance with 83% of surveyed staff intending to use EVS in the next 6-12 months. A more positive attitude in schools from Social Sciences and Arts subject area than in other subject groups There is an overall (UH-wide) high agreement on handing out EVS for free upon registration and disagreement that the use of EVS has led to immediate operational efficiencies

13. Preliminary Findings : Staff interviews The Head of School ‘Our biggest challenge was getting staff on board with it. I would think when I started, 25% of staff were very enthusiastic and 75% were not, now we’re probably running at 50/50. Everybody who’s used EVS and got it to work comes back and says it’s great and they’re going to use it again… the technology has got to be easy so that people can get in the swing of it’ Head of School interview June 2012

14. Preliminary Findings : Staff interviews The Lecturers ‘Once it’s embedded in the workload for staff then that should be greatly reduced in terms of marking, writing assignments, it’s really refreshing the bank of multiple choice questions [they’ve created.]’ …it was so obviously simple to generate questions and I think that has always been the main barrier in people’s minds, that because it’s something new and that it’s a technology that it has to be difficult as well

15. Conclusions High student enthusiasm for using EVS in class especially for formative and low-stakes summative assessments Plenty of top-down support and leadership and local support is important to embed the technology Academics using EVS are not yet seeing all the expected efficiencies in time saved that they had hoped for. BUT 93% of those responding to the survey and using it for the first time in 2011 said they plan to use it in the next 6 -12 months Patterns of previous technology adoption at UH are mirrored – – The example of the MLE saw early student enthusiasm but academics took more time to feel fully at ease with the change in practice and pedagogy.

16. References Bangor, A., Kortum, P., & Miller, J.A. (2008). The System Usability Scale (SUS): An Empirical Evaluation, International Journal of Human-Computer Interaction, 24(6). Chickering and Gamson http://www.uis.edu/liberalstudies/students/documents/sevenprinciples.pdfhttp://www.uis.edu/liberalstudies/students/documents/sevenprinciples.pdf D’Inverno, R., Davis, H., & White S. (2003), Using a personal response system for promoting student interaction, Teaching Mathematics and its Applications, Vol. 22 (No. 4): 163-169. Draper, S. W., & Brown, M. I.(2002).Use of the PRS handsets at Glasgow University, Interim Evaluation Report: March 2002 http://www.psy.gla.ac.uk/~steve/evs/interim.html accessed April 2012 http://www.psy.gla.ac.uk/~steve/evs/interim.html Draper, S. W., & Brown, M. I. (2004). Increasing interactivity in lectures using an electronic voting system. Journal of Computer Assisted Learning, 20(2), 81-94. JISC (2004) http://www.jisc.ac.uk/media/documents/publications/effectivepracticeelearning.pdfhttp://www.jisc.ac.uk/media/documents/publications/effectivepracticeelearning.pdf Kanter, R.M., Stein, B.A. & Jick, T.D. (1992) The Challenge of Organizational Change. Free Press. New York Kennedy, G. E., & Cutts, Q. I. (2005). The association between students' use of an electronic voting system and their learning outcomes. Journal of Computer Assisted Learning, 21(4), 260-268 Lorimer, J., & Hilliard, A. (2009). Use of a Electronic Voting System (EVS) to Facilitate Teaching and Assessment of Decision Making Skills in Undergraduate Radiography Education. Paper presented at the Proceedings of 8th European Conference on e-Learning, Bari, Italy Moore, G. A. (1991). Crossing the Chasm. New York: Harper Business. Nicol, D., & Draper, S. (June 2009). A blueprint for transformational organisational change in higher education: REAP as a case study. In J. T. Mayes (Ed.), Transforming Higher Education through Technology-Enhanced Learning. Oliver, M. (2006). New pedagogies for e-learning? ALT-J: Research in Learning Technology, 14(2), 133 - 134. Robins, K., (2011) EVS in the Business School, University of Hertfordshire Internal Report Thornton, H. A. (2009). Undergraduate Physiotherapy students’ choice and use of technology in undertaking collaborative tasks. Open University, UK, Milton Keynes Twetten, J., Smith, M.K., Julius, J. and Murphy-Boyer, L. (2007) Successful Clicker Standardization EDUCAUSE QUARTERLY Number 4 2007 Venkatesh, V., Morris, M. G., Davis, G. B., & Davis, F. B. (2003). User acceptance of information technology: toward a unified view. MIS Quarterly, 27(3), 425–478. Willis, J. (2009) Using EVS in the School of Life Sciences, University of Hertfordshire Internal Report