1. S519: Evaluation of Information Systems
Result
D-Ch10

2. Last week What is QWS? What are the steps?
What are lessons-learned during the exercise?

3. Putting them together Now we are ready to write our evaluation report.
Basically we are going to fill our content to the checklist boxes we learned in lec2.

4. The Key Evaluation Checklist (D-p. 6-7)
I. Executive
Summary
II. Preface
III. Methodology
1. Background
& Context
2. Descriptions
& Definitions 3.
Consumers 4.
Resources
5. Values
6. Process
Evaluation
7. Outcome
Evaluation
10.
Exportability
8 & 9. Comparative
Cost-Effectiveness
11. Overall Significance
14.
Reporting
&
Follow-up
12.
Recommendations
& Explanations
13. Responsibilities
15. Meta-evaluation

5. KEC Preliminary checkpoints
To orient readers to the basics of :
what the evaluation is and
why the evaluation was done
What the main approach was

6. I. Executive summary
Shows a short overview (1-2 pages) of what was evaluated and what the main findings were.
You can consider to include the followings:
Very short description of the program
The overall conclusion about the quality or value of the evaluation
The graphical profile of the evaluand‘s performance (see Exhibit7.6)
Several (<7) bullet points about the most important strengths and weaknesses.

7. II. Preface Address the following questions:
Who asked for the evaluation and why?
What are the main evaluation questions?
Is this a formative or summative evaluation?
Who are the main audiences for this evaluation report?
Mainly based on your work in Ch2
Please always provide justifications for your conclusions

8. III. Methodology The main methodologies used in the evaluation
How you set up your experiments
How you collect data
How you analyze your data?
How you draw conclusion
And each with why.

9. KEC foundations checkpoints
To represent all the initial ingredients
Background
Context
What it is
Whom it serves
The nature and limitation of any resources
Where the values come from

10. 1. Background and Context
Incude just enough information for readers to be able to understand the basic retionale for the program
Contents are:
Why did this program or product come into existence in the first place (needs assessment)
How is the evaluation supposed to address the needs and problems?
What are the main context of evaluands (e.g. Physical, economic, political) to facilitate or constrain the evaluation?

11. 2. Description and definitions
Describe your evaluand in enough details
Definition of important terms

12. 3. Consumers
Identify the actual and potential recipients or users of the evaluand
Describe consumers in geographic locatoin, demographics
Downstream impactees

13. 4. Resources
Describe the available and important resources (which helps to understand the conclusion of the evaluation)
Funds/budget
Physical space
Experience
networks

14. 5. Values
To convey to the audience how the evaluation team determinded what should be considered as „valuable“ or „high quality“ for the evaluand.
Justify the validity

15. KEC Sub-evaluation checkpoints
Getting evaluation explicitly

16. 6. Process evaluation
The issues of evaluation, content and implementation (see Ch4)
Ethics, the priniciple of equity and fairness
Consistency with professional and scientific standards
Efficiency (i.e, minimal wasted effort or resource)
Needs of consumers (e.g., timeliness, learning style, current knowledge)
Needs of staff (e.g., performance on tasks and activities)

17. 7. Outcome evaluation
The outcomes are what happened to the consumers as a result of coming into contact with the evaluand
Intended and unintended
Short-term and long-term effects
A good way to do
List the main outcome dimensions with a brief explanation of how the list was generated.
Rate each dimension on importance
Explain the used methods (ch7)
Rate the each outcome dimension (ch8)
Provide the justification (evidence, logic, methods) for your rate

18. 8&9. Comparative cost-effectiveness
It is important to know whether the evaluand is cost-effective
See D-p62-63

19. 10. Exportability
Whether your evaluation could also have value outside its current context.
It is always worth considering the possibility

20. KEC conclusions checkpoints
Combining Checkpoints6-10 to draw an overall conclusion

21. 11. Overall significance
Using synthesis methodology to draw a summary covering Checkpoints 6-10
Graphical representation (e.g., Exhibit7.6)
Summary of strengths and weaknesses

22. 12. Recommendations and explanations (possible)
Recommendations for furture improvement

23. 13. Responsibilities (possible)
Who is repsonsible for good or bad results
You have to be highly skilled

24. 14. Report and support
Who get the copies of the evaluation reports and in what forms
What are follow-ups to ensure that the findings are used

25. 15. Meta-evaluation Self review of the evaluation itself
The validity of their conclusions
Utility to relevant stakesholders
The methodology used for evaluations
Credibility
Cost
Lessoned learned during this evaluation.

26. Exercise Go back to your evaluation group
Discuss all these checkpoints and how are you going to put all your result together

27. Exercise Go through these checkpoints
Write down 3 checkpoints which are most important to you and state why?
Discuss with your partner
Form the group and select 3 most important checkpoints, why and
Present your group conclusion and your suggestions in poster
Hang your group poster and people can shop around poster
Vote for the best poster