1. DIGITAL MEDIA INSTITUTE hypermedia laboratory Finnish Virtual University TAMPERE UNIVERSITY OF TECHNOLOGY A Multidisciplinary evaluation framework and tool for the evaluation of usability, pedagogical usability, accessibility and informational quality of web-based courses Kirsi Silius kirsi.silius@tut.fi Anne-Maritta Tervakari anne.tervakari@tut.fi Seppo Pohjolainen seppo.pohjolainen@tut.fi Tampere University of Technology (TUT) Finland

2. DIGITAL MEDIA INSTITUTE hypermedia laboratory Finnish Virtual University TAMPERE UNIVERSITY OF TECHNOLOGY 2 Viewpoints of evaluation user group usability accessibility presentation of information presentation of information reliability of information reliability of information pedagogical usability pedagogical usability value added quality of information fluency of use instructional quality benefits experts

3. DIGITAL MEDIA INSTITUTE hypermedia laboratory Finnish Virtual University TAMPERE UNIVERSITY OF TECHNOLOGY 3 Usefulness of web-based learning environments based on Nielsen (1993), modified by Tervakari & Silius 2002-03 Utility Easy to learn Efficient to use Usability Easy to remember Few errors Subjectively pleasing Value added Support for organisation of the teaching process Support for learning, tutoring and scaffolding process and support for the achievement of learning objectives Support for the development of learning skills (interaction with other actors and growth of learners` autonomy Organisation of teaching process Development of quality of teaching Development of learners’ learning skills Testing and development of educational ICT Pedagogical usability Usefulness Social acceptability System acceptability Practical acceptability Cost Compatibility Reliability Etc. Nielsen, J. 1993. Usability Engineerings. Boston (MS.): Academic press.

4. DIGITAL MEDIA INSTITUTE hypermedia laboratory Finnish Virtual University TAMPERE UNIVERSITY OF TECHNOLOGY 4 The main phases of survey at TUT –Context analysis: to find significant background factors, which have strong impact to the instructional value of the certain web-based learning environment. –Assessment of implementation of web courses using multidisciplinary framework based on usability, pedagogical usability, accessibility and informational quality –Advantages and disadvantages of web-based learning in various contexts. What are the critical factors of quality in web-based learning? –Value added of web-based courses in various contexts.

5. DIGITAL MEDIA INSTITUTE hypermedia laboratory Finnish Virtual University TAMPERE UNIVERSITY OF TECHNOLOGY 5 web-based course common usefulness of web-based course usefulness from teacher’s viewpoint learners their personal contexts Usefulness in certain teaching and learning context organisation, science, subjects, actors etc.. teachers their personal contexts experts or evaluators evaluation of web-based course with the tool teaching and designing of web-based course studying usefulness from learners’ viewpoint value-added? advantages? disadvantages? usefulness of certain web-based course in certain teaching and learning context Common usefulness society, culture etc. context independent factors

6. DIGITAL MEDIA INSTITUTE hypermedia laboratory Finnish Virtual University TAMPERE UNIVERSITY OF TECHNOLOGY 6 Evaluation sections

7. DIGITAL MEDIA INSTITUTE hypermedia laboratory Finnish Virtual University TAMPERE UNIVERSITY OF TECHNOLOGY 7 design and/or development of web-based course development of web-based course feedback to teachers and designers resources implementation of web-based course experts’ evaluation and questionnaires for real users analysis of data by experts web-based course training of designers and teachers for evaluating their web- based courses with the tool Iterative evaluation evaluation by teachers and designers as well as questionnaires for real users with the tool analysis of data by teachers and designers

8. DIGITAL MEDIA INSTITUTE hypermedia laboratory Finnish Virtual University TAMPERE UNIVERSITY OF TECHNOLOGY 8 Purpose of this evaluation tool This tool is not –for making ranking lists of web-based courses –for evaluate the validity or substantiveness of the content This tool is to –help teachers to pay attention to properties of their web pages which might need some improvement

9. DIGITAL MEDIA INSTITUTE hypermedia laboratory Finnish Virtual University TAMPERE UNIVERSITY OF TECHNOLOGY 9 Start of evaluation 1. 2. 3.

10. DIGITAL MEDIA INSTITUTE hypermedia laboratory Finnish Virtual University TAMPERE UNIVERSITY OF TECHNOLOGY 10 Choose sections

11. DIGITAL MEDIA INSTITUTE hypermedia laboratory Finnish Virtual University TAMPERE UNIVERSITY OF TECHNOLOGY 11 Questions to be answered

12. DIGITAL MEDIA INSTITUTE hypermedia laboratory Finnish Virtual University TAMPERE UNIVERSITY OF TECHNOLOGY 12 Report 1: Results of the evaluation

13. DIGITAL MEDIA INSTITUTE hypermedia laboratory Finnish Virtual University TAMPERE UNIVERSITY OF TECHNOLOGY 13 Report 2: Guides and tips

14. DIGITAL MEDIA INSTITUTE hypermedia laboratory Finnish Virtual University TAMPERE UNIVERSITY OF TECHNOLOGY 14 Report 3: Own answers

15. DIGITAL MEDIA INSTITUTE hypermedia laboratory Finnish Virtual University TAMPERE UNIVERSITY OF TECHNOLOGY 15 Filter out unrelevant questions

16. DIGITAL MEDIA INSTITUTE hypermedia laboratory Finnish Virtual University TAMPERE UNIVERSITY OF TECHNOLOGY 16 Browse through guides and tips

17. DIGITAL MEDIA INSTITUTE hypermedia laboratory Finnish Virtual University TAMPERE UNIVERSITY OF TECHNOLOGY 17 Guides and tips

18. DIGITAL MEDIA INSTITUTE hypermedia laboratory Finnish Virtual University TAMPERE UNIVERSITY OF TECHNOLOGY 18 Futher development More criteria and guidelines into pedagogical usability section More references into all sections Questionnaires for real users The tool will help to match data collected from evaluators and real users