1. Assessing Learning Outcomes Dr. Deb Gebeke, Facilitator

2. Apps’ Textbook O Original Extension Evaluation Textbook written in 1991…over 25 years ago! O Page 121-131…only 10 pages devoted to assessment/evaluation O Happiness Index vs. Outcomes and Impacts O How will you know you are making a difference? Spending resources wisely?

3. Apps’ Summary O Evaluation: Determine something’s value O Program Evaluation: Determine to what extent the objectives are realized O If objectives are realized, does that mean the class was a success? O That participants learned something of value?

4. Textbook: Apps’ Summary O Evaluation can be both formal and informal. O Evaluation can be conducted at different times and for different reasons: O Before: Needs Assessment O During: To make changes based on feedback O After: Assessing Outcomes O App’s approach is not vigorous enough for today

5. What are the Four Stages of Learning? O Stage 1: Ignorance O Stage 2: Confusion O Stage 3: Confidence O Stage 4: Mastery O’Connor and Seymour, 1993

6. Stage 1: Ignorance O Unconscious Incompetence O We are unaware of a skill. O Unaware that one cannot do a Task. “I don’t know that I don’t know.”

7. Stage 2: Confusion O Conscious Incompetence O Conscious attention is on the task and the results are variable. This is the stage when the learning rate is the greatest. O Aware of the task, but cannot do it. “I know that I don’t know.”

8. Stage 3: Confidence O Conscious Competence O Full conscious attention is still needed to carry out an activity. The skill is not yet fully integrated and habitual. O Able to think through a task step by step and do it “I know that I know. I get it”

9. Stage 4: Mastery O Unconscious Competence O Skill has been fully integrated and is habitual O Can do the task without thinking about intermediate steps “I can just do it. I have integrated what I learned into practice”

10. Four Stages of Learning UnconsciousConscious CompetentStage 4: MasteryStage 3: Confidence IncompetentStage 1: IgnoranceStage 2: Confusion

11. Four Levels of Evaluation O Dr. Donald Kirkpatrick: Model began over 50 years ago as a sales and training evaluation tool O Kirkpatrick’s Four Levels of Evaluation 1. Reaction 2. Learning 3. Behavior Change 4. Results O Print and Post Summary card- http://www.ag.ndsu.edu/evaluation/documents/Eva luation%20Cards.pdf http://www.ag.ndsu.edu/evaluation/documents/Eva luation%20Cards.pdf

12. Reaction to web-based summary clips (Dr. Myron Eighmy) O Why Evaluate (.51 min) http://youtu.be/glH79s5U9cg http://youtu.be/glH79s5U9cg O Four Levels (1.35 min) http://youtu.be/uN6fMLN3Z0M http://youtu.be/uN6fMLN3Z0M O Reaction or Satisfaction (3.22 min) http://youtu.be/9qvjPtt5Bwk http://youtu.be/9qvjPtt5Bwk O Learning (6.47 min) http://youtu.be/dukDR9UsQ8Q http://youtu.be/dukDR9UsQ8Q O Behavior Change (5.19 min) http://youtu.be/WX_SaKrPlfM http://youtu.be/WX_SaKrPlfM O Impact (4.25 min) http://youtu.be/OnvZ0DQyYBk http://youtu.be/OnvZ0DQyYBk O The Ripple Effect (3.19 min) http://youtu.be/wqmBaVbhRiU http://youtu.be/wqmBaVbhRiU Detailed Chart for Four Levels of Evaluation (business model) http://www.ag.ndsu.edu/evaluation/documents/kirkparicks-four-levels-of-training- evaluation-in-detail

13. What resources are available to assist in overall planning and evaluation for programs? O NDSU: Worksheet and guidelines for completing program planning worksheet http://www.ag.ndsu.edu/programplanning/program- planning-resources O Evaluation Tool Kit (samples): http://www.ag.ndsu.edu/ext-emp/evaluation/

14. How do you know when your program is successful? O Objectives are met O Sample Basic Tool for evaluating programs O http://www.ag.ndsu.edu/evaluation/evaluation- instruments/learningeval.docx http://www.ag.ndsu.edu/evaluation/evaluation- instruments/learningeval.docx

15. Consider adding other items O Was the information helpful? O How will you use the information? O How effective were each of the strategies? O Demographic information O Open ‐ ended questions to allow for feedback on items that learner feels are important? O What behavior do you intend to change? Be ready for follow up survey.

16. Methods and Timing O Survey Monkey, Qualtrics and other electronic resources O During program, post then pre or retrospective, mail, phone, etc.

17. Resources O American Evaluation Association - http://www.eval.org/ http://www.eval.org/ O Bloom’s Taxonomy of Action Verbs - http://www.ag.ndsu.edu/evaluation/doc uments/blooms-taxonomy-1 http://www.ag.ndsu.edu/evaluation/doc uments/blooms-taxonomy-1