1. Key Performance Measures, Evaluation Plans, and Work Plan
Hello. I’m Shelley Kuklish, and I’m the epidemiologist for the Arizona Nutrition Network. In this portion of the training I will be going over the Key performance measures, evaluation plans and work plan portions of your Fiscal Year 2010 application.
Shelley Kuklish
Epidemiologist
Arizona Nutrition Network

2. Session Objectives
By the end of this session, participants will be able to:
Complete the Key Performance Measures, Evaluation Plans, and Work Plan Sections of the application
Write S.M.A.R.T. objectives
Write succinct action steps
Identify an evaluation plan for each objective
By the end of this training session, you will be able to complete the key performance measures, evaluation plans, and work plan sections of the application. You will be able to write SMART objectives. (We’ll get into what a smart objective is in a few minutes). You will also be able to write succinct action steps. Finally, you will be able to identify an evaluation plan for each objective that you write in your workplan.

3. Application It is recommended that you:
Complete Work Plan (Appendix B) before completing Key Performance Measures and Evaluation Plans on the “Project Overview” sheet
So to start out, we recommend that you complete your work plan, or appendix B of your application before completing the key perfromance measures and evaluation plans sections on the project overview sheet.

4. Here is just a screen shot of the project overview tab of the work plan. This section you’ll skip until after you’ve completed your workplan. This section is much easier to complete when you have completed your work plan first.

5. Housekeeping
We will be working on a ‘sample’ work plan one step at a time.
Goal is that you leave here with at least one objective, one action step and one evaluation plan for that objective.
Can be used as an example for writing the rest of your FY10 work plan
During the practice, Network staff will walk around to help as necessary.
Be polite and tolerant of other’s opinions.
The goal of this training is to teach you HOW to write your work plan, not WHAT you are doing to teach SNAP-ED.
So just a few housekeeping rules before we get started.
We’re going to be working on a sample work plan one step at a time.
Our goal is that you leave this training with at least 1 objective, 1 action step and 1 evaluation plan for that objective. You can then use these as an example when writing the rest of your fiscal year 2010 workplan.
During the practice sections, Network staff will be walking around to help you as necessary.
Please be polite and tolerant of other’s opinions. We’re interested in teaching you how to write your work plan, and are not focusing on What you are doing to teach nutrition education.

6. Work Plan
So let’s get started by taking a look at a sample workplan.

7. To start, on the project overview sheet you will need to select the goals that you will be focusing on for fiscal year The goals that you check will then unlock so that you can type in your work plan.

8. Here’s what the workplan, or appendix B looks like once you’ve selected your goals. As you see here, the only goal I selected was the goal related to fruit and vegetable consumption. So, the only goal that I can type in on the workplan is the fruit and vegetable goal. As you can see down at the bottom, the rest of the boxes are greyed out, meaning I can’t type anything into them until I select the goal on the project overview sheet.

9. Writing Objectives
Let’s move on to writing objectives.

10. The objectives will go in the area I have circled
The objectives will go in the area I have circled. There is only room for four objectives per state goal, so please select your objectives carefully. In a few minutes I will show you that objectives really only fit into three areas, so you will probably not even use all four objective areas.

11. Goals vs. Objectives A GOAL is: A broad statement of intent
Non-specific, non-measurable
Example:
Increase the proportion of SNAP recipients and eligibles aged two years and older who meet dietary recommendations for fruit and vegetable consumption.
Lots of people tend to get confused when it comes to goals and of objectives. A goal is a broad statement of intent. It’s non-specific and non-measurable. Here is an example of a goal. This goal is simply to increase the proportion of SNAP recipients and eligibles who meet recommendations for fruit and vegetable consumptions. It is not specific and not time-bound. It is a broad statement of an intent.

12. Goals vs. Objectives An OBJECTIVE is:
A realistic target for the program
Answers: Who will do how much of what by when?
Example:
By the end of the class series, 50% of low income 3rd grade students participating in the FVMM classes will report an increase in fruit and vegetable consumption.
So if that was a goal and not an objective, than what is an objective you might say? Well, an objective is a realistic target for a program. It answers the question ‘who will do how much of what by when?’. Here’s an example of an objective. This goal says that by the end of the class series, half of low income 3rd graders who participate in the Fruits and veggies more matters classes will report an increase in fruit and vegetable consumption. The objective meets the SMART criteria, which I will explain on the next slide.

13. S.M.A.R.T. Objectives S.M.A.R.T. objectives are: Specific Measurable
Achievable
Relevant
Time Bound
As I said before, when you write an objective, it should be SMART. SMART is an Acronym for: Specific, Measurable, Achievable, Relevant and Time Bound. I will explain each one in more detail.

14. Specific What are you going to do, with or for whom? Measurable
Achievable
Relevant
Time Bound
What are you going to do, with or for whom?
Focus on results, not how they are achieved.
So when I say a objective should be specific, I mean that the objective should answer the question of ‘What are you going to do, with or for whom? Objectives are focused on results not how they are achieved. We’ll get into how they are achieved when we look at action steps.

15. Measurable Specific Is the objective measurable? Quality Achievable
Relevant
Time Bound
Is the objective measurable?
Quality
Resources
Quantity
Time
An objective should be measurable. This means that you should be able to measure progress against the objective. This includes quality, resources, quantity or time. Examples would be increasing the number of students you reach from 1200 to 1300, or increasing the percentage of students who correctly answered questions after the class from 40% to 80%.

16. Achievable Specific Measurable Relevant Time Bound
Can you achieve this objective?
Set objective after considering the baseline
Keep objectives within reach
Is this objective realistic?
Can we do this?
Why are we doing this?
A objective should be achievable. When you write an objective, ask yourself the question, is this achievable? For example: Stating that 100% of clients will exercise daily is not achievable. How much can you REALLY expect to increase in 1 year? You should consider your baseline before writing your objective, and keep the objective in reach. You should also ask if the objective you have written is realistic. Ask yourself if you can do this, and why you are doing this.

17. Relevant Specific Measurable Achievable
Time Bound
Is this objective logical as it relates to SNAP-Ed goals?
Ask yourself if the objective is relevant. Is it logical as it relates to SNAP-Ed goals? Is it allowable? Will this objective lead to the desired results?

18. Time Bound Specific Measurable Achievable Relevant
What is the timeframe for this objective?
For the Network purposes, the time frame generally is by September 30, 2010
Finally, ask yourself if your objective is time bound. An objective should always have a timeframe. For general network purposes, the time frame is September 30, It could also be by the end of the class series, after attending a nutrition workshop, etc.

19. Objectives Apply to 3 Areas
Process (Operational)
Impact (Knowledge)
Outcome (Behaviors)
So as I said before, objectives fit in to three areas. Process, impact and outcome. I’ll give you some more detail about each of these on the following slides.

20. Process Objectives (Operational)
Tells what you are doing and how you will do it
Tracks participants, interactions and activities
Example:
By September 30, 2010 at least 5,000 Fun Food News will be distributed to SNAP-Ed participants at community events.
Process objectives tell what you are doing and how you will do it. It tracks participants, interactions and activities. Most partners objectives will fall into this type of objective. I’ve provided an example of a process objective. The example says that by September 30, 2010, at least 5,000 FFN will be distributed to SNAP-Ed participants at community events.

21. Impact Objectives (Knowledge)
Tells how you will change attitudes and knowledge
Example:
By the end of the class series, 80% of low income 3rd grade students participating in the BBB classes will increase their knowledge of activities that help build strong bones.
Impact objectives tell how you will change attitudes and knowledge.

22. Outcome Objectives (Behavior)
Tells what the long term implications of your program will be.
Example:
By the end of the class series, 50% of low income 3rd grade students participating in the FVMM classes will report an increase in fruit and vegetable consumption.
Outcome objectives tell what the long term implications of your program will be. Objectives related to behavior change fall under this category.

23. Easy Way to Write a SMART Objective
Answer the following question:
Who will do
How much
Of What
By When?
Example:
By the end of the class series, 50% of low income 3rd grade students participating in the FVMM classes will report an increase in fruit and vegetable consumption.
An easy way to write a SMART objective is to ask yourselves the following questions when you are writing an objective. “Who will do how much of what by when?” The objective does not have to be written in the order of the question, so long as it answers the question. Here is an example of a outcome objective. To answer the ‘who will do’ portion of the question, we have low income 3rd grade students participating in FVMM classes. To answer the ‘how much’ part of the question, we have 50%. To answer the ‘of what’ portion of the question, we have ‘will report an increase in fruit and vegetable consumption. To answer the ‘by when’ portion of the question, we have ‘by the end of the class series’.

24. Your Turn! Write An Objective Statement
Make sure that it meets the SMART requirements
Specific
Measurable
Achievable
Relevant
Time Bound
Answer the following questions:
Who will do
How much
Of What
By When?
Now it’s your turn to write an objective. Pick one of the state goals that your program will be selecting for their FY 2010 application, and write one or more objectives for that goal. You have three worksheets in your packet. The first sheet has the state goals for your reference. The second sheet is a worksheet to help you write your objective. The third sheet is a copy of the work plan. Right now we’re just going to use the first two. Once you have settled on your objective, you can write it on the work plan sheet if you wish. Remember as you are writing your objective to make sure that it meets the smart requirements (which are specific, measurable, achievable, relevant, and time bound). Also, make sure the objective answers the following questions. We’ll be walking around to assist you. If you’re having trouble, please grab one of us to help you.
Allow 5 minutes for writing goals.
Ok. Now you all have a objective that you’ve written. Can I have a couple of volunteers share the objectives they’ve written? Please tell us which state goal you’ve selected, and the objective you’ve written for that goal.
<Write objective on board as they are reading them>
Great! Thank you for sharing. Ok. Let’s look at the goals that have been shared and see if they meet the SMART criteria.
<Go through the objective with the group to see if it meets SMART and if it answers Who will do how much of what by when?>

25. By the end of the class series, 50% of low income 3rd grade students
participating in the FVMM classes
will report an increase in fruit
and vegetable consumption.
(FVMM survey baseline 48%)
By September 30, 2010, teach the
importance of eating fruits and
vegetables through the FVMM classes
to at least 500 low-income 3rd grade
students.
So here are a few objectives that I came up with for the fruit and vegetable consumption state goal. My first objective is a outcome objective (remember outcome objectives look at behavior) It says “By the end of the class series, 50% of low income 3rd grade students participating in the FVMM classes will report an increase in fruit and vegetable consumption.” As you can see, the baseline from last year was 48%, so a increase of 2% in a year is a realistic and achievable objective. The second objective is a process objective (remember process objectives tell what you are doing and how you will do it) The objective reads: “By September 30, 2010, teach the importance of eating fruits and vegetables through the FVMM classes to at least 500 low-income 3rd grade students”. The 500 would have been set based on whatever was done last year + an additional amount that you anticipated to reach this year. This keeps it achievable.

26. Writing Action Steps
Before we move on to writing action steps, does anyone have any questions regarding writing objectives? Keep in mind if you’re having trouble writing your objectives as you’re doing your application, or come up with a question, my contact information is on the slideshow, and I’m always willing to help!

27. Moving on to action steps
Moving on to action steps. The area I’ve circled is where you will write your action steps for the objective that you wrote a few minutes ago.

28. Action Steps vs Objectives
Focus on the activities
Provide the ‘How’
Objective
Focuses on the results
Provide the ‘What’
So what’s the difference between an action step and an objective? Well, an action step focuses on the activities you will do to achieve your objective. Action steps provide ‘how’ you will achieve your objective. Objectives focus on the results and provide the ‘what’ your activities will achieve. So like the picture of the archer, the action steps are the man pulling back the bow, aiming for the target, releasing the bow and the arrow flying to the target. The objective is the arrow hitting the bulls eye, the final target, or result of all the action steps that lead up to it. The archer’s main objective wasn’t to aim the bow, it was to hit the target.

29. Action Steps Should work towards the objective.
Should provide how you will attain your objective.
Be brief and succinct.
Objectives
Action Steps
As I said on the previous slide, action steps work towards your objective. They provide information on how you will attain your objective.

30. Example:
Objective:
By the end of the class series, 50% of low income 3rd grade students participating in the FVMM classes will report an increase in fruit and vegetable consumption.
Action Step:
1.Teach FVMM classes to low income 3rd grade students in 18 schools.
2. Conduct pre-/post-surveys
So here’s an example of some action steps. Here’s the outcome objective that I wrote in section 1. By the end of the class series, half of low income 3rd grade students participating in the FVMM classes will report an increase in fruit and vegetable consumption. So, my action steps to reach this objective is to teach the FVMM classes to low income 3rd grade students in 18 schools, and to conduct pre/post-surveys.

31. Your Turn! Write an Action Step
Using the objective statement you wrote in section 1, write at least 1 action step for this objective.
Remember:
Action steps should work towards the objective.
Should provide how you will attain your objective.
Be brief and succinct.
So now it’s your turn. Using the objective statement you write in section 1, write at least 1 action step for this objective. Remember that the action steps work towards the objective, should provide how you will attain your objective and be brief and succinct. Again we’re going to walk around the room to help, so if you need help just ask!
<give 5 mins to write action steps>
Ok. Now you all have some action steps that you’ve written for your objectives. Can I have a couple of volunteers share the action steps they’ve written? Please tell us what your objective was, and what your action steps for that objective is.
<Write on board as they are reading them>
Great! Thank you for sharing. Ok. Let’s look at these action steps. Do you feel that they all should fall under this objective or should there be two objectives (one process and one outcome/impact).

32. By the end of the class series, 50% of low income 3rd grade students
Teach FVMM classes to low income 3rd grade
students in 18 schools.
By the end of the class series, 50%
of low income 3rd grade students
participating in the FVMM classes
will report an increase in fruit
and vegetable consumption.
(FVMM survey baseline 48%)
Conduct pre/post-test evaluations.
Coordinate with 18 schools to conduct
FVMM classes.
By September 30, 2011, teach the
importance of eating fruits and vegetables
through the FVMM classes to at
least 500 low-income 3rd grade
students.
Meet with teachers to schedule FVMM classes.
Teach FVMM classes to low income 3rd grade
students in 18 schools.
So here’s an example of some action steps I have written for the outcome and process objectives that I wrote in section 1. So, for the outcome objective related to increasing fruit and vegetable consumption, my action steps are teach Fvmm classes to low income 3rd grade students in 18 schools, and to conduct pre/post evaluations. For the process objective, where I’m reaching at least 500 students with the FVMM curriculum, my action steps are to coordinate with the 18 schools to conduct the classes, meet with the teachers to schedule the classes, and teach the classes at the 18 schools.

33. Writing Evaluation Plans
Before moving on to evaluation plans, does anyone have any questions regarding writing action steps?

34. We’re finally at evaluation measures
We’re finally at evaluation measures. The area circled will be where we enter in our evaluation measures.

35. Evaluation Measures Formative Process Impact/Outcome
Collects data for improvement of program/processes
Process
Evaluates that the program is being delivered as planned
Impact/Outcome
Measure change in knowledge, attitudes or behaviors
There are three types of evaluation measures. Formative, which collects data for improvement of program and processes. Process, which evaluates that the program is being delivered as planned, and Impact/outcome, which measure change in knowledge attitudes or behaviors.

36. Formative Evaluation
Formative research is conducted before a program is designed and implemented, or while a program is being conducted.
It helps you:
Decide on and describe your target audience,
Understand the needs of the audience and the factors which influence their behavior
Helps you to design programs, services, and products to address those needs.
Let’s look at formative evaluation a little closer. Formative research is conducted before a program is designed and implemented, or while a program is being conducted. It helps you decide on and describe your target audience, understand the needs of the audience and the factors that influcence their behavior, and helps you to design programs, services and products to address those needs.

37. Examples of Formative Evaluation
Needs assessments
Focus group discussions
In-depth interviews
Market assessments
Teacher evaluations
Pilot programs where content of nutrition education is tested and evaluated, etc.
Here are some examples of formative evaluation. They include needs assessments, focus groups, interviews, market assessments, teacher evaluations, and pilot programs. All of the research that we do prior to our social marketing campaigns are formative research. The annual needs assessment that we report to USDA is formative research. Anything that you collect that you use to adapt or tailor your program to the needs of your population, such as teacher evaluations, are formative research. An important thing to note about formative evaluation is that there are not formative objectives. You shouldn’t be writing an objective that says ‘change program as indicated by teacher evaluations’ (because it’s not measurable!) so, there won’t be an evaluation measure that is ‘teacher evaluations’. While formative evaluation is an important part of your program, for your work plan, you aren’t going to have any formative evaluations to evaluate your objectives. Your formative evaluations would fit under ‘action steps’. You can include these type of evaluations, but they should not be the evaluations that you are using to evaluate your objective.

38. Process Evaluation
Process evaluation focuses on how the program operates, the services it delivers, and the function it carries out.
It addresses whether the program was implemented and is providing services as intended.
Let’s look at process evaluation. Process evaluation focuses on how the program operates, the services it delivers and the functions it carries out. It addresses whether the program was implemented and is providing services as intended.

39. Examples of Process Evaluation
Tracking:
Number of materials distributed
Number of clients viewing a food demonstration
Number of classes held
Number of participants reached
Examples of process evaluation include tracking the number of materials distributed, the number of clients viewing a food demo, the number of classes held, or the number of participants reached. The majority of you already are doing process evaluation and might not even know it!

40. Outcome/Impact Evaluation
An outcome or impact evaluation assess the change in knowledge or behaviors as a result of a particular program or intervention.
The goal of an outcome or impact evaluation is to learn how effective the intervention was in changing the target populations' attitudes, awareness, and behaviors.
Outcome and impact evaluations assess the change in knowledge or behaviors as a result of a particular program or intervention. The goal of this type of evaluation is to learn how effective the intervention was in changing the target population’s attitudes, awareness and behaviors.

41. Examples of Outcome/Impact Evaluation
Pre-/post-test showing an increase in knowledge
Outcome
Pre-/post-surveys showing an improvement in self-reported behaviors
Point-in-time surveys, tracked over time
An example of an impact evaluation include pre/post tests showing an increase in knowledge. An example of an outcome evaluation include pre/post surveys showing an improvement of behaviors. Another example of either impact/outcome evaluations, (or both) are point in time surveys that are tracked over time, like the network survey that they are doing now. It is collecting both knowledge and behavior information and will be tracked over time.

42. Example: Objective: Action Steps: Evaluation Plan:
By the end of the class series, 50% of low income 3rd grade students participating in the FVMM classes will report an increase in fruit and vegetable consumption.
Action Steps:
1. Teach FVMM classes to low income 3rd grade students in 18 schools.
2. Conduct pre/post-test evaluations.
Evaluation Plan:
Conduct pre- and post-surveys using FVMM consumption survey. (Baseline 48%)
Here is the objective we’ve been working with throughout this session, the action steps and the evaluation plan for the objective. The objective is to have half of the students increasing fruit and vegetable consumption by the end of the class series. Our action steps again are to teach the classes and to conduct the pre/post surveys. So, the evaluation plan is to conduct the pre/post surveys using the FVMM consumption survey. The baseline for this objective is 48%. I’ll explain baselines on the next slide.

43. Baselines
Clearly defined starting point from where implementation begins, improvement is measured, or comparison is made
The first data point for your evaluation
Allows you to create achievable objectives by providing a starting point.
Allows for tracking trends over time.
So you might be asking what is a baseline. It’s a clearly defined starting point from where implementation begins, improvement is measured or a comparison is made. It’s the first data point for your evaluation. It allows you to create achievable objectives by providing you a starting point, and it allows for tracking trends over time.

44. Evaluation Plans Should Tie Directly to The Objective
Process objective=process evaluation
Example:
Objective: Teach FVMM curriculum to 1,000 students in FY 2010
Evaluation: Track number of students participating via attendance records
Impact objective=impact evaluation
Objective: 5th graders increasing knowledge of osteoporosis prevention
Evaluation: Pre/Post-test measuring knowledge
Outcome objective=outcome evaluation
Objective: 3rd graders increasing fruit and vegetable consumption
Evaluation: Pre/Post-test measuring consumption
An important point to note regarding your work plan is that evaluation plans should tie directly to the objective. So, if you have a process objective, you will have a process evaluation to evaluate that objective. An example of this would be that your objective is to teach the FVMM curriculum to 1000 students in FY 2010, your evaluation plan would be to track the number of students participating using attendance records. If you have an impact objective, you should have an impact evaluation plan. An example of this would be an objective regarding 5th graders increasing knowledge of osteoporosis prevention. Your evaluation plan would be to conduct pre/post tests to measure knowledge change. If you have an outcome objective, you should have an outcome evaluation. An example of this would be 3rd graders increasing fruit and vegetable consumption after FVMM classes. The evaluation plan would be to conduct a per/post survey to measure fruit and vegetable consumption.

45. Your Turn! Write An Evaluation Plan
Remember:
Evaluation plans tie to the OBJECTIVE
Process objective=process evaluation
Impact objective=impact evaluation
Outcome objective=outcome evaluation
Now it’s your turn. Using the objective you wrote in section 1 and the action steps you wrote in section 2, write an evaluation plan to evaluate your objective. Remember that the evaluation plan ties directly to the objective. Also, remember that if you have a process objective, you’ll need a process evaluation plan. If you have an impact objective, your evaluation plan will be an impact evaluation, and if you have an outcome objective, you’ll write an outcome evaluation plan. We’ll be walking around in case you need help.
<give 5 minutes>
Ok. Lets see what you guys came up with. Please tell us the objective you selected, and the evaluation plan you created to evaluate that objective.

46. By the end of the class series, 50% of low income 3rd grade students
Teach FVMM classes to low income 3rd grade
students in 18 schools.
By the end of the class series, 50%
of low income 3rd grade students
participating in the FVMM classes
will report an increase in fruit
and vegetable consumption.
(FVMM survey baseline 48%)
Conduct pre/post-test evaluations.
Conduct pre- and post- surveys
using FVMM consumption survey
Coordinate with 18 schools to conduct
FVMM classes.
By September 30, 2011, teach the
importance of eating fruits and vegetables
through the FVMM classes to at
least 500 low-income 3rd grade
students.
Meet with teachers to schedule FVMM classes.
Teach FVMM classes to low income 3rd grade
students in 18 schools.
Track number of students using
attendance records
Here is the example I came up with for an outcome and a process objective. For my outcome objective (regarding increasing fv consumption) my evaluation plan is to conduct pre/post surveys using the fvmm consumption survey. For my process objective (regarding reaching 500 students), my evaluation plan is to track the number of students using attendance records.

47. Things to Note
You can include evaluation plans for each action step, however at least ONE evaluation plan must directly apply to the objective.
Just a side-note, you can include evaluation plans for each action step, 1 evaluation plan must tie directly to the objective. This is where you could put your formative evaluations if you have any.

48. Project Overview
Now that you’ve finished your work plan, you will be able to complete the key performance measures and evaluation plans section of your application.

49. For the Key Performance Measure section, you will check the tools you stated that you will use to evaluate your objectives in your work plan. You will then provide detail on how you will use those tools to evaluate your objectives. So, for the example, we used pre/post surveys and attendance records, so I have provided a few sentances about what tool I will be using and how I will use it. For pre/post surveys (behavior) I put that the fvmm consumption survey will be used as a pre/post survey in conjunction with the FVMM classes to measure fruit and vegetable consumption in low-income third grade participants. For attendance records, I put that the number of students participating in the FVMM classes will be tracked using attendance sheets.

50. For the evaluation plan section, you’ll include narrative regarding your evaluation plans. For example, under my process evaluation plans, I’m simply stating that the number of students reached will be tracked using attendance sheets, and at least 500 students will be reached. For outcome/impact, I put that the fvmm consumption survey will be administered prior to and at the completion of the FVMM classes to measure fruit and vegetable consumption for participants, to assess if the class is effective in increasing fruit and vegetable consumption for participants. For Fiscal year 2009, 48% of students increased fruit and vegetable consumption after the FVMM classes. A goal of 50% has been set for FY 2010.

51. Appendix F
Send in samples of each evaluation tool used for impact/outcome evaluations
Stephanie will give you more information in her presentation.
So now you’ve got all of your evaluation plans written. In appendix F you will need to send in samples of each evaluation tool you will be using for your impact/outcome evaluations. Stephanie will provide more detail regarding this appendix in her presentation.

52. Thank You! Shelley Kuklish 602-364-1736 Shelley.Kuklish@azdhs.gov
Questions? Comments?
Thank You!
Shelley Kuklish
Does anyone have any questions or comments? As always if you need assistance regarding this part of the application, here’s my contact information. I’m always more than happy to help you!