Usability evaluation in a multiphase, exploratory design-based research study of an online community for the practice of special education in Bulgaria Rob Peterson, PhD University of Wollongong Faculty of Education robpeterson@gmx.com www.robertrpeterson.com Ian Olney, EdD University of Western Sydney Teaching Development Unit i.olney@uws.edu.au June 2009, ED-MEDIA, Hawaii

Overview Background of study Website development cycle Types of usability evaluations Participant selection Usability tasks and scenarios Data collection Equipment Data analysis Results

Background Establishment of the Special Education Bulgaria online community 3-year PhD project Design-based research Exploratory research design Design principles for virtual communities of practice Moodle and Mahara www.SpecialEducationBulgaria.com

Data Sources Research Participants…………………..76 Personal interviews 19 Usability interviews...……………………….18 Expert consultations 4 Web-based questionnaires 18 Email-based questionnaires 17 Website Participants…………………….242 Registered website users Discussion forum and other posts Website Usage Logs

Website Development Cycle by Research Phase Аdapted from “Product development life cycle” (Rubin,1994)

Website Versions Prototype Alpha Beta/Final

Usability Evaluations Inquiries, inspections, and tests Exploratory, assessment, and validation usability evaluations Phase 1, Phase 2, Phase 3 Qualitative Quantitative

Participant Selection Purposeful-quota sampling Purposeful-intensity sampling

Tasks and Scenarios Task: Find a definition for the word dyslexia. Task: Send a message to the website administrator. Scenario 1: Registration Scenario 2: Chat Scenario 3: Discussion forums Scenario 4: Lost password and username Scenario 5: Upload Post-Test Questions

Data Collection Observation Sheet Design Element Importance (A-C) Repairability (1-3) Problem / Recommendation

Setup and Equipment for Usability Evaluation at Remote Location

Problem/Solution Matrix for Analysis of Usability Data

Usability Findings by Importance (A-C) and Repairability (1-3)

Common Findings (Repaired) Superfluous or confusing data required in registration process. Confusion regarding the type of characters to use, Cyrillic or Latin. Cannot see other participant’s full profile information such as email address or city.

Common Findings (Not Repaired) Cyrillic characters not supported by Moodle, e.g., Cyrillic characters do not alphabetize correctly in lists. Only 50% of the RSS feed’s Bulgarian news items returned actually had to do with education and Bulgaria.

Key Points Special Education Bulgaria project Exploratory and design-based research Usability inquires versus tests Usability evaluations at remote locations Data analysis in terms of importance and repairability Challenges and solutions … see the paper

17 Authors Rob Peterson, PhD University of Wollongong Faculty of Education www.robertrpeterson.com Ian Olney, EdD University of Western Sydney Teaching Development Unit i.olney@uws.edu.au Supervisors Prof Jan Herrington Murdoch University School of Education A/Prof Mira Tzvetkova-Arsova Sofia University, Faculty of Primary and Preschool Education A/Prof Deslea Konza Edith Cowan University Faculty Education and Arts A/Prof Krassen Stefanov Sofia University, Faculty of Mathematics and Informatics Building the European Network for Lifelong Competence Development Ian Olney, PhD University of Western Sydney Teaching Development Unit

Key References Barnum, C. M. (2002). Usability testing and research. New York, NY: Longman Publishers. Barnum, C. M., Mukherjee, A., Mielke, N., Porter, M., Schumann, M., Speaks, B., et al. (1999). Hotmail usability test plan. Retrieved January 12, 2006, from http://www.ablongman.com/barnum Creswell, J. W., & Clark, V. L. P. (2007). Designing and conducting mixed methods research. Thousand Oaks, CA: Sage Publications. Reeves, T. C. (2006). Design research from a technology perspective. In J. van den Akker, K. Gravemeijer, S. McKenney & N. Nieveen (Eds.), Educational design research (pp. 52-66). London: Routledge. Rubin, J. (1994). Handbook of usability testing: How to plan, design, and conduct effective tests. New York, NY: John Wiley & Sons, Inc. Rubin, J., Chisnell, D., & Spool, J. (2008). Handbook of usability testing: How to plan, design, and conduct effective tests (2nd ed.). Indianapolis, IN: Wiley Publishing, Inc.

Addendum Multiphase Design Based Research Adapted from “Predictive Stage 1 Analysis of practical problems by researchers in collaboration Stage 2 Development of solutions informed by existing design principles and technological innovations Addendum Stage 3 Iterative cycles of testing and refinement of solutions in practice Stage 4 Reflection to produce “design principles” and enhance solution implementation Multiphase Design Based Research Adapted from “Predictive and design research approaches in educational technology research” (Reeves, 2006) Stage 1 Analysis of practical problems by researchers in collaboration Stage 2 Development of solutions informed by existing design principles and technological innovations Stage 3 Reflection to produce “design principles” and enhance solution implementation Phase 1 Needs Assessment Stage 1 Iterative cycle of testing and refinement of solutions in practice Stage 2 Development of solutions informed by existing design principles and technological innovations Stage 3 Reflection to produce “design principles” and enhance solution implementation Phase 2 Formative Evaluation Stage 1 Iterative cycle of testing and refinement of solutions in practice Stage 2 Development of solutions informed by existing design principles and technological innovations Stage 3 Reflection to produce “design principles” and enhance solution implementation Phase 3 Effectiveness Evaluation Future impact and maintenance evaluations