1. Quality Evaluation methodologies for e-Learning systems (in the frame of the EC Project UNITE) Tatiana Rikure Researcher, Riga Technical University (RTU), Latvia

2. UNITE “Unified e-Learning Environment for the School” EC 6 th FP IST, STREP started: February 2006, duration: 30 months 13 partners from 11 countries (coordinator: Fraunhofer Institute for Computer Graphics) The strategic goal: is to contribute to the improvement of European wide education in schools based on common, innovative principles in technology, pedagogy, scenarios as well as validation of success The strategic objective: is the development of a set of frameworks to support Europe-wide eLearning in schools and to adapt these frameworks to national needs by "instantiation".

3. will be implemented in 14 secondary schools in 11 countries UNITE “Unified e-Learning Environment for the School” Results UNITE e-Learning concept (new e- Learning scenarios based on innovative principles of e-Learning and a sound pedagogical model) UNITE system (a flexible e-Learning ICT platform based on an innovative e-Learning platform & pedagogical methodology) Validation & evaluation methodology (the quality of the UNITE system & the success of the e-Learning scenarios) Socio-economic evaluation Europe-wide network of schools Europe-wide repository of the re-usable e- Learning content etc.

4. UNITE “Unified e-Learning Environment for the School” System

5. UNITE “Unified e-Learning Environment for the School” Validation framework Test the impact and viability of the technology and the e-Learning activities in a range of school settings Show the appropriateness, meaningfulness and usefulness of the UNITE system and the e-Learning scenarios

6. Riga Technical University (RTU) in UNITE “Unified e-Learning Environment for the School” RTU (Division of Applied Systems Software) main tasks: Designing the quantitative assessment methodologies for the validations Leading the validation activities in Latvia Quality evaluation will be developed and executed on 2 levels: Quality Assessment of the system: at the design phase, of the final product, Measuring Efficiency (success \ impact) of the system: using qualitative evaluation, using quantitative evaluation.

7. Quality approaches any methods, policies, procedures, rules, criteria, tools, checklists or any other verification instruments or measures that have the purpose of enhancing the quality of e-Learning products or services, can be classified for the purposes of the UNITE systems’ validation. Design phase Efficiency of the system Software verification approaches: Static: - Consistency techniques - Measurement techniques Dynamic: - White-box (structural) testing - black-box (functional) testing - Random testing Software testing Formative evaluation Critical commenting Developmental testing Revising Reviews (of literature, learners, learning context, existing solutions, etc.) Walk-through Storyboards with potential users & experts Observations, video-taping, user-tracking, interviews with users & experts Peer reviews Final product Quality assurance guidelines Quality evaluation criteria Quality marks Best Practices Benchmarking Bloom’s Taxonomy Expert evaluation Quality assurance frameworks

8. Quality approaches any methods, policies, procedures, rules, criteria, tools, checklists or any other verification instruments or measures that have the purpose of enhancing the quality of e-Learning products or services, can be classified for the purposes of the UNITE systems’ validation. Design phase Efficiency of the system Software validation approaches: Formal methods Fault injections: - Hardware - Software Dependability analysis: - Hazard analysis - Risk analysis Software testing Final product Summative evaluation Expert evaluation Evaluation of the learning content Quality assurance guidelines Quality evaluation criteria Quality marks Best Practices Benchmarking Usability testing Feedback from users (questionnaires, interviews, pre and post tests, etc.) Revising Expert / Peer review

9. Quality approaches any methods, policies, procedures, rules, criteria, tools, checklists or any other verification instruments or measures that have the purpose of enhancing the quality of e-Learning products or services, can be classified for the purposes of the UNITE systems’ validation. Design phase Efficiency of the system Final product Pedagogical experiments Quality assurance guidelines Quality evaluation criteria Quality marks Best Practices Benchmarking Expert evaluation Kirkpatrick's 4 Levels of Evaluation ROI Learners performance testing Feedback from learners and users Interviews (oral guided, written, telephone, etc.) Questionnaires Lists of criteria (checklists) Concept-mapping exercise BLOGs Observations Case Studies Collaborative assessment techniques WATTS (web-assessment techniques and tools) Quality management activities Quality audits Knowledge surveys

10. Conclusions The development of e-Learning products and the provision of e-learning opportunities is one of the most rapidly expanding areas of education and training. E- Learning is also one of the areas that attracts the most research and development funding. If this investment is to be maximised, it is our challenge to generate robust models for the systematic evaluation of e-learning and produce tools which are flexible in use but consistent in results. Measuring the effectiveness of e-Learning is an important issue in the development and use of any e-Learning system or strategy, which not only shows the value of their use, but also allows improving in the future. Measuring the quality of the e-Learning systems should be done with the use of multiple-method evaluation strategy that considers both qualitative and quantitative criteria. Evaluation of e-Learning is fundamentally the same as the evaluation of any other learning but with particular groups of variables playing a more prominent role and the impact of others differs significantly from their impact in traditional learning.

11. Acknowledgements UNITE (Unified e-Learning Environment for the School) IST4Balt (Information Society Technologies Promotion in Baltic States) http://www.balticit.com/ist4balt/ eLOGMAR-M (Web-based and Mobile Solutions for Collaborative Work Environment with Logistics and Maritime Applications) http://www.elogmar-m.org

12. Contacts Tatiana Rikure Riga Technical University (RTU) Faculty of Computer Science & Information Technology Division of Applied Systems Software Meza str. ¼, Riga, Latvia +371 7089096; fax: +371 7089572 rikure@cs.rtu.lv