1. University of Maribor, Faculty of Electrical Engineering and Computer Science, ines.kozuh@uni-mb.si, matjaz.debevc@uni-mb.si Review of the evaluation methods for OP4L project Ines Kožuh prof. dr. Matjaž Debevc Project meeting OP4L, Salzburg 5.-6. December 2011

2. University of Maribor, Faculty of Electrical Engineering and Computer Science, ines.kozuh@uni-mb.si, matjaz.debevc@uni-mb.si Why evaluation? Summative evaluation –product evaluation, –quantitive, –numeric scores. Formative evaluation –more complex, –„test run“.

3. University of Maribor, Faculty of Electrical Engineering and Computer Science, ines.kozuh@uni-mb.si, matjaz.debevc@uni-mb.si What will we evaluate?

4. University of Maribor, Faculty of Electrical Engineering and Computer Science, ines.kozuh@uni-mb.si, matjaz.debevc@uni-mb.si OP4L learning environment

5. University of Maribor, Faculty of Electrical Engineering and Computer Science, ines.kozuh@uni-mb.si, matjaz.debevc@uni-mb.si How will we evaluate?

6. University of Maribor, Faculty of Electrical Engineering and Computer Science, ines.kozuh@uni-mb.si, matjaz.debevc@uni-mb.si Evaluation methods MethodEvaluationAuthor SUSusabilityBangor et. al., 2008 SUMIusabilityKirakowski & Corbett, 1993 Usabiltiy heuristicsusabilityNielsen, Molich, 1990 Adapted heuristicsusability with socio- constructivism tenets Squires, Preece, 1999 Multidisciplinary tool for evaluation usability, pedagogical usability Silious & Tervakari, 2003 Pedagogical rating of online courses pedagogySonwalkar, 2002 MSLQ (Motivated strategies for learning questionnaire) pedagogy, motivatonPintrich & De Groot, 1990

7. University of Maribor, Faculty of Electrical Engineering and Computer Science, ines.kozuh@uni-mb.si, matjaz.debevc@uni-mb.si SUMI – Software Usability Measurement Inventory measuring usability, mentioned in the ISO 9241 standard as a recognised method of testing user satisfaction, 50 statements to which the users have to reply that they either Agree, Don't know, or Disagree, 3 minutes to fill out the questionnaire, at least 12 users.

8. University of Maribor, Faculty of Electrical Engineering and Computer Science, ines.kozuh@uni-mb.si, matjaz.debevc@uni-mb.si SUMI – Software Usability Measurement Inventory

9. University of Maribor, Faculty of Electrical Engineering and Computer Science, ines.kozuh@uni-mb.si, matjaz.debevc@uni-mb.si An example of SUMI results

10. University of Maribor, Faculty of Electrical Engineering and Computer Science, ines.kozuh@uni-mb.si, matjaz.debevc@uni-mb.si Motivated Strategies for Learning Questionnaire (MSLQ) 81-item instrument, 7-point Likert Scale, 6 motivation scales (31 items measuring value, expectancy, and affective component), 9 learning strategies (50 items measuring cognitive and metacognitive strategies, and resource management strategies).

11. University of Maribor, Faculty of Electrical Engineering and Computer Science, ines.kozuh@uni-mb.si, matjaz.debevc@uni-mb.si Evaluation Evaluation will take place in Belgrade, Maribor and Skopje. In Belgrade already started - first impressions?

12. University of Maribor, Faculty of Electrical Engineering and Computer Science, ines.kozuh@uni-mb.si, matjaz.debevc@uni-mb.si Contact University of Maribor, Faculty of Electrical Engineering and Computer Science, Smetanova 17, 2000 Maribor www.feri.uni-mb.si Ines Kožuh e-mail: ines.kozuh@uni-mb.si prof. dr. Matjaž Debevc e-mail: matjaz.debevc@uni-mb.si