1. Welcome to the Allegany Franciscan Ministries 2016 Major Grant Evaluation & Success Measures Webinar We will begin PROMPTLY at 11:00 a.m. ~~ Please MUTE your phone ~~ Please do not use a headset as they cause feedback that is disruptive to others on the call. March 14, 2016

2. Reflection As We Gather Creator and Source of All Being, We give you thanks for the beauty of this day, for the gift of this world, and for the gift of each other. We thank you for the ways in which you are present to us in our service of the sick and the suffering, the poor, and the outcast. As we gather today, we ask you to be with us. Open our eyes — that we may see what you wish us to see. Open our ears — that we may hear what you want us to hear. And, open our hearts — that we may feel as you feel toward those in greatest need of our services. Bless all that we do here in our gathering; may it be a holy work done in your name. Amen.

3. Agenda  Welcome and Reflection  The Application  Evaluation – our philosophy  Success Measures  Definitions  Examples  Resources  Q & A Presenters Jessica Gonzalez, Program Officer Shelley Robertson, EdD, Evaluation Consultant

4. Our goals for this webinar To help you and your staff team…  Understand our evaluation philosophy, process & expectations.  Better understand the difference between program outcomes & indicators; and the difference between those outcomes that are measureable & meaningful and those that are not.  Begin to conceptualize program success measures before your initial meetings with our Program Officer.  Have the tools and knowledge to submit a successful application.

5. Major Grant Timeline Hillsborough, Pinellas, Palm Beach, Martin & St. Lucie Counties Miami Dade County Notes LOI Due September 3, 2015 January 29, 2016 Online application must be submitted by 12 pm (noon). Notification of our decision; decline or invite to submit full application November 9, 2015 February 29, 2016 We will e-mail and send a letter to you. Major Grant Full Application Technical Assistance Webinar Week of November 16, 2015 Week of March 1, 2016 For those selected to submit a full application; please hold date (date subject to change). Deadline to submit full application January 14, 2016 April 14, 2016 Online application must be submitted by 12 pm (noon). Site Visits & Meetings March 2016 TBD Decisions Announced April 2016 June 2016 We will work with applicants, after decisions are finalized, on budgets & success measures. Grant Period Begins May 2016 July 2016

6. 2016 Allegany Franciscan Ministries Focus

7. The Application – How to Apply  Read the entire 2016 Major Grant Application Instructions document.  Go to our web site, www.afmfl.org.  Click on the “Current Partner” button.  Log on to the online application system.  Confirm or edit all contact information.  Under “LOI” you will see “Application”.  Click on “Edit Application”.

8. The Online Application – Some Basics  Complete your responses in Word© and then paste into the appropriate field.  Pay attention to character limits. Document will not save if you are over the character count.  Don’t use acronyms or jargon.  Save your work frequently.  When uploading documents, click “save as draft” to finalize each upload.  Before submitting, click “Application Packet” to print your application and read/have someone else read the application.

9. The Application

11. Budget

12. Evaluation Philosophy As a good steward, Allegany Franciscan Ministries is dedicated to assuring that our investments move us toward our mission. Our evaluation process is clear, simple, and focused on outcomes. The evaluation process is respectful of organizations with different size, sophistication, values, beliefs, and traditions. It balances qualitative and quantitative methods to inform and guide our decisions while improving services provided by our grant partners.

13. Getting to measures  An organization should articulate how the program leads to long-term change for people.  Success measures document progress being made towards that long-term change.

14. Allegany Franciscan Ministries SUCCESS MEASURES FORM Organization Name:Project Title: Year: From (month/year): To (month/year): Outcomes Please state one outcome per row. Indicators What is the specific statistic you will track to determine your success on this outcome? What data tool will you use to collect this data? Activities What are the key activities that will lead to each outcome? Measurement plan What is your measurement plan? For example, how often will you administer your data tool? How will you assess whether you were successful? Participants increase knowledge about how to cook healthy meals. 85 percent of participants are able to identify four healthy cooking methods and up to 10 healthy food choices. Provide 10 hours of training to 20 participants in 5 two- hour sessions. We will administer a paper pre and post test to participants and track the results in Excel. Coalition members will increase their advocacy activities 90 percent of members will take at least two advocacy actions during the six months after training. Recruit and train a coalition of up to 20 advocates. Members will report actions at monthly coalition meetings; staff will document efforts. complete one page – and one page only - per year of grant

15. Definitions  Outcomes: improvement and/or change in knowledge, skills, attitudes, or behavior. Outcomes may also refer to desired changes in a community or in an organization.  Indicators: the specific items used to track a program’s success. Indicators describe observable, measureable characteristics or changes. An indicator should include the specific characteristic measured, what data tool is used, when it is measured, and who collects the data.

16. Definitions  Activities: what a program does that will lead to the outcomes.  Measurement plan: states any sampling strategies, how the data is entered, how you will analyze the characteristics to assess whether you were successful, and how data will be reported.

17. Program outcomes Program outcomes state the changes in clients’ knowledge, skill, attitude, or behavior.

18. Example: Program outcomes  Clients increase their access to health care.  An indicator could be that 80% of an estimated 100 clients who did not see a primary care physician in the last year see a primary care physician while in the program.  Clients have increased access to healthy food.  Clients are compliant with their plan of care.  Clients increase positive health behaviors (e.g., not smoking, exercise, eating 5 fruits and veggies, maintaining a healthy weight).

19. Organizational capacity outcomes Organizational outcomes state changes in capacity areas such as revenue, fiscal management, board development, operations, etc.  The organization should be able to explain – although probably not measure – how the change in capacity will impact those they serve, such as improved service quality or more clients served.

20. Example: Organizational capacity outcomes  The organization increases the capacity of its board of directors.  An indicator would be that the organization meets the 10 benchmarks on the board development section of the Capacity Benchmarking tool (see resources).  The organization increases financial stability.  The organization increases efficiency of operations.  The organization increases capacity through partnerships.

21. 21 Creating Outcome Statements: Step one: describe your client (i.e., age, gender, status) Examples: at-risk middle school youth, adults with mental illness, frail seniors, Hispanic teens, adults with disabilities, low- income parents The Outcome Formula

22. 22 Step two: identify the issue or topic you are addressing. Examples: healthy lifestyle choices, employment, substance abuse, grades, sports, self-esteem, coping skills, health status. Creating Outcome Statements: The Outcome Formula

23. 23 ClientVerbAdjectiveTypeExample/topic WillIncreaseKnowledge about DecreaseSkill(s) in MaintainAttitude about Behaviors such as Examples: 1. At -risk middle school youth will decrease behaviors such as participating in gangs, getting in fights, and using drugs. 2. Hispanic teens will increase skills in job seeking. Creating Outcome Statements: The Outcome Formula Demonstrate

24. 24 Characteristics:  Observable and measurable.  Client-focused.  Unambiguous.  State a time frame.  Match.  Understandable. Indicators

25. 25 The Indicator Formula Tells what you will measure (see table below), how, when, and who is measuring. For this type of outcome Tell us….Examples: Knowledge What three main things the clients learn SkillWhat, specifically, they can demonstrate Attitude What they will or will not feel Behavior What they will or will not do

26. 26 Indicator Examples Tool: Pick one: survey, pretest and posttest, posttest only, interview, case records, observation form, or official statistics. Tell us the name of the tool and who created it. Examples: curriculum pretest and posttest, online survey created by USF, case records kept by the agency, report cards from the school, ADL checklist provided by the State Department of Elder Affairs, survey provided by DJJ. Target: Put the percentage of your clients that you think will achieve the outcome. The target is NOT the client population. Examples: 20%, 30%, 40%, 50%, 60%, 70%, 80%, 90%.

27. 27 The Indicator Formula How examples: how the tool is administered. When examples: at the end of the program, beginning, every 30 days, 90 days after completing the program, etc. Who examples: program staff, non-program staff, outside agency personnel, other agencies.

28. 28 Indicator Examples Knowledge:  [Clients] will learn three elements of sportsmanship as measured by an end of the program posttest administered by staff. Skill:  [Clients] will demonstrate that they can prepare a resume, find an open job through the paper or online, and answer interview questions as measured by non-program staff observation during the job fair at the end of the program. Attitude:  [Clients] will state that they feel hopeful about a positive future as demonstrated by exit interviews conducted by program staff prior to leaving the shelter. Behavior:  [Clients] will not be adjudicated for gang activity, fights, or drugs while they are in the program as measured by DJJ statistics collected by the School Resource Officer.

29. Common errors  Lack of clarity about what you want to achieve.  Confuse activities or outputs with outcomes.  Confusion over number served per year, % achieving outcomes, or increases over time.  Limited measurement plan.

30. Hints  Outcomes: State one outcome per row.  Indicators and measurement: What is the specific statistic will you track to determine your success on this outcome? What data tool will you use to collect this data?  Activities: What are the key activities that will lead to each outcome?  Measurement plan: What is your measurement plan? For example, how often will you administer your data tool? How will you assess whether you were successful?

31. Next steps  Log in to the grant administration system and begin the application.  Contact Jessica with application questions earlier rather than later.  Complete the Full Application by April 14, 2016, 12 pm (noon).  Communicate with your team and discuss possible outcomes with your team.

32. Contact Information  For questions: contact Jessica Gonzalez at 727-507- 9668 or jgonzalez@afmfl.orgjgonzalez@afmfl.org  For technical support: contact Kathie Hardy at 727- 507-9668 or khardy@afmfl.orgkhardy@afmfl.org

33. Questions?