1. Usability evaluation in a multiphase, exploratory design-based research study of an online community for the practice of special education in Bulgaria
Rob Peterson, PhD University of Wollongong Faculty of Education
Ian Olney, EdD University of Western Sydney Teaching Development Unit
June 2009, ED-MEDIA, Hawaii

2. Overview Background of study Website development cycle
Types of usability evaluations
Participant selection
Usability tasks and scenarios
Data collection
Equipment
Data analysis
Results

3. Background
Establishment of the Special Education Bulgaria online community
3-year PhD project
Design-based research
Exploratory research design
Design principles for virtual communities of practice
Moodle and Mahara

4. Data Sources Research Participants…………………..76
Personal interviews
Usability interviews...……………………….18
Expert consultations
Web-based questionnaires
-based questionnaires
Website Participants…………………….242
Registered website users
Discussion forum and other posts
Website Usage Logs

5. Website Development Cycle by Research Phase Аdapted from “Product
development life cycle”
(Rubin,1994)

6. Website Versions
Prototype
Alpha
Beta/Final

7. Usability Evaluations
Inquiries, inspections, and tests
Exploratory, assessment, and validation usability evaluations
Phase 1, Phase 2, Phase 3
Qualitative Quantitative

8. Participant Selection
Purposeful-quota sampling
Purposeful-intensity sampling

9. Tasks and Scenarios Task: Find a definition for the word dyslexia.
Task: Send a message to the website administrator.
Scenario 1: Registration
Scenario 2: Chat
Scenario 3: Discussion forums
Scenario 4: Lost password and username
Scenario 5: Upload
Post-Test Questions

10. Data Collection Observation Sheet Design Element
Importance (A-C) Repairability (1-3)
Problem / Recommendation

11. Setup and Equipment for Usability Evaluation at Remote Location

12. Problem/Solution Matrix for Analysis of Usability Data

13. Usability Findings by Importance (A-C) and Repairability (1-3)

14. Common Findings (Repaired)
Superfluous or confusing data required in registration process.
Confusion regarding the type of characters to use, Cyrillic or Latin.
Cannot see other participant’s full profile information such as address or city.

15. Common Findings (Not Repaired)
Cyrillic characters not supported by Moodle, e.g., Cyrillic characters do not alphabetize correctly in lists.
Only 50% of the RSS feed’s Bulgarian news items returned actually had to do with education and Bulgaria.

16. Key Points Special Education Bulgaria project
Exploratory and design-based research
Usability inquires versus tests
Usability evaluations at remote locations
Data analysis in terms of importance and repairability
Challenges and solutions … see the paper

17. 17
Authors
Rob Peterson, PhD University of Wollongong Faculty of Education
Ian Olney, EdD University of Western Sydney Teaching Development Unit
Supervisors
Prof Jan Herrington Murdoch University School of Education
A/Prof Mira Tzvetkova-Arsova Sofia University, Faculty of Primary and Preschool Education
A/Prof Deslea Konza Edith Cowan University Faculty Education and Arts
A/Prof Krassen Stefanov Sofia University, Faculty of Mathematics and Informatics
Building the European Network for Lifelong Competence Development
Ian Olney, PhD University of Western Sydney Teaching Development Unit

18. Key References
Barnum, C. M. (2002). Usability testing and research. New York, NY: Longman Publishers.
Barnum, C. M., Mukherjee, A., Mielke, N., Porter, M., Schumann, M., Speaks, B., et al. (1999). Hotmail usability test plan. Retrieved January 12, 2006, from
Creswell, J. W., & Clark, V. L. P. (2007). Designing and conducting mixed methods research. Thousand Oaks, CA: Sage Publications.
Reeves, T. C. (2006). Design research from a technology perspective. In J. van den Akker, K. Gravemeijer, S. McKenney & N. Nieveen (Eds.), Educational design research (pp ). London: Routledge.
Rubin, J. (1994). Handbook of usability testing: How to plan, design, and conduct effective tests. New York, NY: John Wiley & Sons, Inc.
Rubin, J., Chisnell, D., & Spool, J. (2008). Handbook of usability testing: How to plan, design, and conduct effective tests (2nd ed.). Indianapolis, IN: Wiley Publishing, Inc.

19. Addendum Multiphase Design Based Research Adapted from “Predictive
Stage 1 Analysis of practical problems by researchers in collaboration
Stage 2 Development of solutions informed by existing design principles and technological innovations
Addendum
Stage 3 Iterative cycles of testing and refinement of solutions in practice
Stage 4 Reflection to produce “design principles” and enhance solution implementation
Multiphase
Design
Based
Research
Adapted from “Predictive
and design research
approaches in educational
technology research”
(Reeves, 2006)
Stage 1 Analysis of practical problems by researchers in collaboration
Stage 2 Development of solutions informed by existing design principles and technological innovations
Stage 3 Reflection to produce “design principles” and enhance solution implementation
Phase 1 Needs Assessment
Stage 1 Iterative cycle of testing and refinement of solutions in practice
Stage 2 Development of solutions informed by existing design principles and technological innovations
Stage 3 Reflection to produce “design principles” and enhance solution implementation
Phase 2 Formative Evaluation
Stage 1 Iterative cycle of testing and refinement of solutions in practice
Stage 2 Development of solutions informed by existing design principles and technological innovations
Stage 3 Reflection to produce “design principles” and enhance solution implementation
Phase 3 Effectiveness Evaluation
Future impact
and maintenance evaluations