S519: Evaluation of Information Systems Meta-evaluation D-Ch11

Meta-evaluation It is evaluation of an evaluation (Scriven, 1991) to determin the quality and/or value of an evaluation http://en.wikipedia.org/wiki/Evaluation#cite_note-7

Five criteria Validity Utility Conduct Credibility Costs

Validity To what extent, the conclusions of the evaluation are justified Check carefully the facts they use and values they assign Question everything Constraints (budget, time, etc.) Using KEC checkpoints Using standards (http://www.eval.org/Publications/GuidingPrinciples.asp)

Validity Covers all relevant sources of value Comprehensively covers process, outcome and cost Include no irrelevant or illicit criteria Data used to directly address the criteria Include analyses that are appropriate for the data Clearly states how data are interpreted Is clear about where evaluative conclusions come from Include valid recommendation

Utility Are the findings Relevant to the questions or decisions being faced by the audience Timely Clearly communicated Cost effective

Conduct Legal Ethical Professional standards Cultrual appropriateness Unobtrusiveness Minimal disruptive to the evaluand

Credibility Familiar with the context Independence, impartiality, and/or lack of conflict of interest Expertise in evaluation and in the evaluation subject field

Cost How reasonable the costs of an evaluation

Meta-evaluation rating table

Exercise D-p217, exercise 2 Form a pair, do the evaluation Form a group to select the best result, Design a poster Present them in a poster Using three sentenses to summarize your evaluation What you learn What you feel How to improve