1. Support the spread of “good practice” in generating, managing, analysing and communicating spatial information Training Evaluation By: Rainer Zachmann Unit: M03U05

2. Content Introduction Relevance of evaluation Monitoring course climate Pre- and post-tests Partial evaluation

3. Content Continuous activity evaluation Evaluation of course coordination Evaluation of course materials Final course evaluation

4. Introduction Evaluation is an essential part of the training curriculum. Evaluation helps in many aspects of training. Evaluation shows whether objectives have been accomplished.

5. Introduction Evaluation tools may range from objective tests to subjective assessments. None of the evaluation procedures described are intended to grade trainees or resource people.

6. Support the spread of “good practice” in generating, managing, analysing and communicating spatial information Relevance of Evaluation

7. Relevance of evaluation Kirkpatrick’s levels of training evaluation: –Level 1 = reaction –Level 2 = learning –Level 3 = behaviour –Level 4 = results

8. Relevance of evaluation Level 1 (= reaction) measures participants’ satisfaction –lowest level of measurement, nonetheless important –includes aspects before and during the course

9. Relevance of evaluation Level 2 (= learning) measures: –acquisition of new knowledge; –improvement of skills; –change in attitude.

10. Relevance of evaluation Level 3 (= behaviour) follows the application after training. Level 4 (= results) assesses impact at trainees' organisations.

11. Relevance of evaluation This Unit addresses Levels 1 and 2. Levels 3 and 4 are follow-up activities.

12. Relevance of evaluation Evaluation is an essential and integral part of the training curriculum, not just the end of training.

13. Relevance of evaluation Evaluation helps to: determine training needs; specify training objectives; adjust training methods and materials; delete unnecessary content.

14. Relevance of evaluation Evaluation helps to: execute training; verify accomplishments; follow-up with trainees; reduce training costs.

15. Relevance of evaluation Evaluating whether objectives have been accomplished is as important as specifying training objectives.

16. Relevance of evaluation Two types of evaluation: formative evaluation to monitor training summative evaluation to measure accomplishments

17. Relevance of evaluation Teachers and organisers often evaluate informally and unconsciously.

18. Relevance of evaluation Objective tests are based on specific questions to be answered. Response tests give freedom for subjective expression. Practical skills testing is important for PGIS.

19. Relevance of evaluation Formal evaluations include: monitoring course climate; pre- and post-tests; partial tests; continued activity evaluation; evaluation of course coordination and course materials; final course evaluation.

20. Relevance of evaluation Evaluations are not intended to grade people, but to monitor course progress.

21. Support the spread of “good practice” in generating, managing, analysing and communicating spatial information Monitoring Course Climate

22. Monitoring course climate Many methods exist: “Democracy Wall” “Evaluation Dartboard” is useful for monitoring participants’ feelings

23. Multilingual democracy wall deployed at the 2007 Web2forDev Conference in Rome, Italy Photo: Anja Barth, CTA (From Unit M10U01)

24. Taylor, B. and Beniest, J. 2003. Training in agroforestry - A toolkit for trainers Page 2.176; ISBN 92 9059 151 x The World Agroforestry Centre United Nations Avenue PO Box 30677, GPO 00100 Nairobi, Kenya www.worldagroforestrycentre.org

25. Support the spread of “good practice” in generating, managing, analysing and communicating spatial information Pre- and Post-Tests

26. Pre- and post-tests The pre-test assesses trainees' initial knowledge. Multiple-choice questionnaires may be used. Multiple-choice questions are difficult to design, but analysis is easy and objective.

27. Pre- and post-tests Multiple-choice questions consist of a "stem" and several options for answers. The stem should be a concise statement that leads directly into the options.

28. Pre- and post-tests Include in the stem as much information as possible about the item. Formulate the stem concisely. Each option should be a grammatically correct completion of the stem.

29. Pre- and post-tests Multiple-choice questions consist of a stem and: –a definitely wrong distracter; –one correct answer; –(an)other definitely wrong distracter(s).

30. Pre- and post-tests Example: Multiple-choice questions are: a) easy to developdistracter b) easy to answerdistracter c) relatively objective answer d) not widely applicable distracter

31. Pre- and post-tests Only one option should be the correct answer. Correct answers must not be self-evident. All other options, i.e. distracters, should be plausible but definitely wrong. The length of options should not provide a clue to the answer.

32. Pre- and post-tests Coordinators should ask only relevant questions. Avoid negatively formulated questions. Arrange the answers in random order. Coordinators should not use multiple- choice questions when other evaluations are more appropriate.

33. Pre- and post-tests Coordinators should present the pre-test at the beginning of the course. Ask trainees to be honest and not to guess. Analyse the test immediately. Discuss results with trainees and resource people. Coordinators should not give the questionnaires back.

34. Pre- and post-tests Use the same questionnaire for the post- test. Present the post-test one day before the course ends. Analyse the test overnight. Compare pre- and post-test results and present them to trainees and resource people.

35. Support the spread of “good practice” in generating, managing, analysing and communicating spatial information Partial Tests

36. Partial tests Partial tests help ensure that resource people cover subject matter appropriately. Coordinators should not use partial tests to grade trainees.

37. Partial tests Coordinators should use the first 15 to 20 minutes of each day for the partial tests. Ask four or five questions from topics of the previous day. Ask open essay questions.

38. Partial tests Coordinators should design questions and expected answers (both!) in collaboration with resource people. The partial tests should only cover relevant course content.

39. Partial tests Allow trainees to use open books, training documents, etc. to place them in a real life situation where they may consult references.

40. Partial tests Coordinators should analyse answers immediately with instructors. Return tests to trainees and discuss results.

41. Support the spread of “good practice” in generating, managing, analysing and communicating spatial information Continuous Activity Evaluation

42. Continuous activity evaluation Trainees evaluate the course. At the beginning of the course, explain how to use the evaluation. Ask trainees to be critical.

43. Continuous activity evaluation Every morning, coordinators should distribute new evaluation forms. Ask trainees to evaluate immediately. At the end of each activity, write the title of the activity on a board to remind trainees to evaluate.

44. Continuous activity evaluation At the end of the day, coordinators should collect the evaluation forms. Summarise results on the summary forms.

45. Continuous activity evaluation Coordinators should include trainees in processing the evaluations. Ask trainees to draw conclusions and report them back to the class the next morning.

46. Continuous activity evaluation Coordinators should give the summaries to the resource people and discuss the results. Keep copies in the files for future planning.

50. Support the spread of “good practice” in generating, managing, analysing and communicating spatial information Evaluation of Course Coordination and Course Materials

51. Evaluation of course coordination Resource people evaluate the support given by course coordinators. Coordinators should adapt the sample to specific needs. Ask resource people to return the evaluation after the course.

53. Evaluation of course materials Resource people evaluate materials included in the PGIS Training Kit. Whenever you use some of the PGIS materials (e.g. Unit Trainer Notes, Exercises, Handouts, PowerPoint Presentations, etc.), return your evaluation to the course coordinator.

55. Support the spread of “good practice” in generating, managing, analysing and communicating spatial information Final Course Evaluation

56. Final course evaluation Ask trainees to assess the course’s overall acceptability, strengths and weaknesses and to provide suggestions. Coordinators should adapt the sample to their needs.

57. Final course evaluation The first question - general opinion on the course - is arranged so that ideal answers result in a diagonal pattern from top left to bottom right.

58. First question

59. Final course evaluation An important aspect is the balance between theory and practice. Trainees usually criticise having too much theory and not enough practice.

60. Question 7

61. Final course evaluation Questions like “What topics should have been extended?” are not conclusive: trainees wish to extend almost everything. A better question would be “What topics should have been reduced or omitted?”

62. Final course evaluation Coordinators should not ask “What topics have been most useful?”: trainees find almost everything most useful. Instead, ask “What topics have been least useful?” Coordinators may ask “What new topics should be included in the future?”

63. Final course evaluation Coordinators should give questionnaires to trainees several days before the course ends. Collect questionnaires the day before the course ends. Extract the most relevant conclusions overnight. Discuss the conclusions with trainees the next morning.

64. Final course evaluation After the course, coordinators should summarise the questionnaires. Include the summary in the final course report. Refer to the results when planning the next course.

65. Last question