1. Welcome to the Allegany Franciscan Ministries 2014 Major Grant Evaluation & Success Measures Webinar We will begin PROMPTLY at 11:00 a.m. ~~ Please MUTE your phone ~~ Please do not use a headset as they cause feedback that is disruptive to others on the call. April 29 th, 2014

2. Reflection Blessed are they who give without expecting even thanks in return, for they shall be abundantly rewarded. Blessed are they who translate every good thing they know into action, for ever higher truths shall be revealed unto them. Blessed are they who do God's will without asking to see results, for great shall be their recompense. Blessed are they who love and trust their fellow beings, for they shall reach the good in people and receive a loving response. Blessed are they who after dedicating their lives and thereby receiving a blessing, have the courage and faith to surmount the difficulties of the path ahead, for they shall receive a second blessing. Adapted from from the book "Peace Pilgrim"

3. Agenda  Welcome and Reflection  The Application  Evaluation – our philosophy  Success Measures & Outcomes  Definitions  Examples  Resources  Q & A Presenters Jessica Gonzalez, Program Officer Erin Baird, Director, Grants and Technology Shelley Robertson, Evaluation Consultant

4. Our goals for this webinar  To help you and your staff team…  Understand our evaluation philosophy, process & expectations.  Better understand the difference between program outcomes & indicators; and the difference between those outcomes that are measureable & meaningful and those that are not.  Begin to conceptualize program success measures before your initial meetings with the Program Officer.  Have the tools and knowledge to submit a successful application.

5. 2014 Major Grants  In each region, 4 to 6 organizations are likely to be funded.  Complete Major Grant Applications are due by May 29, 2014 at 12 pm (noon).  Site Visits will occur in late July and early August.  Board will finalize decisions in September.  Grant agreements will then be developed and signed; initial payments will be made between. November and January, depending on the project.

6. Major Grant Timeline Hillsborough, Pinellas, Palm Beach, Martin & St. Lucie Counties Notes LOI DueMarch 20, 2014Online application must be submitted by 12 pm (noon). Notification of our decision; decline or invite to submit full application By April 25We will e-mail and send a letter to you. Major Grant Full Application Technical Assistance Webinar April 30 For those selected to submit a full application; please hold date (date subject to change). Deadline to submit full applicationMay 29, 2014Online application must be submitted by 12 pm (noon). Site Visits & MeetingsJuly & August Decisions Announcedlate September We will work with applicants between September & December to finalize budgets & success measures. Grant Period BeginsNovember 1, 2014 or later

7. 2014 Allegany Franciscan Ministries Strategies  Community education and outreach designed to increase knowledge and support behavioral changes.  Healthcare services and basic needs.  Increased capacity and coordination of the healthcare delivery system.  Advocacy and community efforts that educate, engage, raise awareness, or mobilize communities.

8. The Application – How to Apply  Read the entire 2014 Major Grant Application Instructions document.  Go to our web site, www.afmfl.org.  Click on the “Current Partner” button.  Log on to the online application system.  Confirm or edit all contact information.  Under “LOI” you will see “Application”.  Click on “Edit Application”.

9. The Online Application – Some Basics  Complete your responses in Word© and then paste into the appropriate field.  Pay attention to character limits. Document will not save if you are over the character count.  Don’t use acronyms or jargon.  Save your work frequently.  When uploading documents, click “save as draft” to finalize each upload.  Before submitting, “Print Packet” and read/have someone else read.

10. The Application

12. Budget

13. Evaluation Philosophy As a good steward, Allegany Franciscan Ministries is dedicated to assuring that our investments move us toward our mission. Our evaluation process is clear, simple, and focused on outcomes. The evaluation process is respectful of organizations with different size, sophistication, values, beliefs, and traditions. It balances qualitative and quantitative methods to inform and guide our decisions while improving services provided by our grant partners.

14. Getting to measures  An organization should articulate how the program leads to long-term change for people.  Success measures document progress being made towards that long-term change.

15. Allegany Franciscan Ministries SUCCESS MEASURES FORM Organization Name:Project Title: Year: From (month/year): To (month/year): Outcomes Please state one outcome per row. Indicators What is the specific statistic you will track to determine your success on this outcome? What data tool will you use to collect this data? Activities What are the key activities that will lead to each outcome? Measurement plan What is your measurement plan? For example, how often will you administer your data tool? How will you assess whether you were successful? Participants increase knowledge about how to cook healthy meals. 85 percent of participants are able to identify four healthy cooking methods and up to 10 healthy food choices. Provide 10 hours of training to 20 participants in 5 two- hour sessions. We will administer a paper pre and post test to participants. Coalition members will increase their advocacy activities 90 percent of members will take at least two advocacy actions during the six months after training. Recruit and train a coalition of up to 20 advocates. Members will report actions at monthly coalition meetings; staff will document efforts. complete one page – and one page only - per year of grant

16. Definitions  Outcomes: improvement and/or change in knowledge, skills, attitudes, or behavior. Outcomes may also refer to desired changes in a community in response to community-level and system-wide activities.  Indicators: the specific items used to track a program’s success. Indicators describe observable, measureable characteristics or changes. An indicator should include the specific characteristic measured, what data tool is used, when it is measured, and who collects the data.

17. Definitions  Activities: what a program does with its resources to fulfill its mission.  Measurement plan: states any sampling strategies, how the data is entered, how you will analyze the characteristics to assess whether you were successful, and how data reported. For example, what counts as an increase?

18. Program outcomes Program outcomes state the changes in clients’ knowledge, skill, attitude, or behavior.

19. Example: Program outcomes  Clients increase their knowledge about how to reduce risk behaviors.  An indicator could be that 80% of clients increase their knowledge about what are risk behaviors, consequences of risk behaviors, and how to reduce risk behaviors as measured by comparing post test to pretest scores on the curriculum-embedded tool.  Clients increase their skills in preparing healthy food.  Clients demonstrate improved confidence toward changing behavior.  Clients experience decreased health risk behaviors.

20. Organizational capacity outcomes Organizational outcomes state changes in capacity areas such as revenue, fiscal management, board development, operations, etc.  The organization should be able to explain – although probably not measure – how the change in capacity will impact those they serve, such as improved service quality or more clients served.

21. Example: Organizational capacity outcomes  The organization increases the capacity of its board of directors.  An indicator would be that the organization meets current needs on 10 benchmarks on the board development section of the Capacity Benchmarking tool (see resources).  The organization increases its fundraising capacity.  An indicator could be that the organization achieves benchmarks on a capacity tool or that the organization (a) develops at least two new revenue streams, (b) reduces expenses by 5 percent in comparing 2010 to 2009, and (c) that 90% of staff report that the new system saves them time.

22. Choosing outcomes: meaningful and measurable  What we want to see:  Fewer youth exhibit at risk behaviors due to undiagnosed mental health problems.  What we don’t want to see:  Fewer youth abuse substances and show increased stability and life skill.

23. Choosing outcomes: meaningful and measurable  What we want to see:  75% of recipients increase their intake of healthy food daily.  What we don’t want to see:  75% of recipients report eating more fruits, vegetables and protein after starting our food delivery program.

24. Choosing outcomes: meaningful and measurable  What we want to see:  “Organization A” will provide primary medical services to individuals and families with no health insurance.  What we don’t want to see:  Provide safety net medical services for individual and families without health insurance or who cannot afford or are not eligible for conventional health care insurance.

25. 25 Creating Outcome Statements: Step one: describe your client (i.e., age, gender, status) Examples: at-risk middle school youth, adults with mental illness, frail seniors, Hispanic teens, adults with disabilities, low- income parents The Outcome Formula

26. 26 Step two: identify the issue or topic you are addressing. Examples: healthy lifestyle choices, employment, substance abuse, grades, sports, self-esteem, coping skills, health status. Creating Outcome Statements: The Outcome Formula

27. 27 ClientVerbAdjectiveTypeExample/topic WillIncreaseKnowledge about DecreaseSkill(s) in MaintainAttitude about Behaviors such as Examples: 1. At -risk middle school youth will decrease behaviors such as participating in gangs, getting in fights, and using drugs. 2. Hispanic teens will increase skills in job seeking. Creating Outcome Statements: The Outcome Formula Demonstrate

28. Outputs vs. Outcomes  Outputs answer the question “how much”.  Outputs include number of clients, service units, hours, and appointments.  Outcomes are client-focused.  Outcomes relate to knowledge, skills, attitudes, or behaviors.

29. 29 Characteristics:  Observable and measurable.  Client-focused.  Unambiguous.  State a time frame.  Match.  Understandable. Indicators

30. 30 The Indicator Formula Tells what you will measure (see table below), how, when, and who is measuring. For this type of outcome Tell us….Examples: Knowledge What three main things the clients learn SkillWhat, specifically, they can demonstrate Attitude What they will or will not feel Behavior What they will or will not do

31. 31 Indicator Examples Tool: Pick one: survey, pretest and posttest, posttest only, interview, case records, observation form, or official statistics. Tell us the name of the tool and who created it. Examples: curriculum pretest and posttest, online survey created by USF, case records kept by the agency, report cards from the school, ADL checklist provided by the State Department of Elder Affairs, survey provided by DJJ. Target: Put the percentage of your clients that you think will achieve the outcome. The target is NOT the client population. Examples: 20%, 30%, 40%, 50%, 60%, 70%, 80%, 90%.

32. 32 The Indicator Formula How examples: how the tool is administered. When examples: at the end of the program, beginning, every 30 days, 90 days after completing the program, etc. Who examples: program staff, non-program staff, outside agency personnel, other agencies.

33. 33 Indicator Examples Knowledge:  [Clients] will learn three elements of sportsmanship as measured by an end of the program posttest administered by staff. Skill:  [Clients] will demonstrate that they can prepare a resume, find an open job through the paper or online, and answer interview questions as measured by non-program staff observation during the job fair at the end of the program. Attitude:  [Clients] will state that they feel hopeful about a positive future as demonstrated by exit interviews conducted by program staff prior to leaving the shelter. Behavior:  [Clients] will not be adjudicated for gang activity, fights, or drugs while they are in the program as measured by DJJ statistics collected by the School Resource Officer.

34. Making It Measurable  Direction of change.  Area of change.  Target population.  Degree of change.  Time frame.  As measured by  Baseline.  Example: To decrease the percentage of 18 to 20 year old college students reporting binge drinking from 13 percent to 12 percent by 6/30/2015 as measured by the CORE survey.

35. Hints  Outcomes: State one outcome per row.  Indicators and measurement: What is the specific statistic will you track to determine your success on this outcome? What data tool will you use to collect this data?  Activities: What are the key activities that will lead to each outcome?  Measurement plan: What is your measurement plan? For example, how often will you administer your data tool? How will you assess whether you were successful?

36. Evaluation Resources In addition, Dr. Shelley Robertson is available at no charge to you for up to two hours. Dr. Robertson can assist you in identifying which of these resources might be most helpful for your specific project. She can also help as you develop or refine the outcomes to be tracked and reported, and determine appropriate indicators and measurement tools.

37. Next steps  Log in to the grant administration system and begin the application.  Contact Jessica with application questions earlier rather than later.  Complete the Full Application by May 29, 2014, 12 pm (noon).  Communicate with your team and discuss possible outcomes with your team.

38. Contact Information  For all questions and technical support: contact Jessica Gonzalez at 727-507-9668 or jgonzalez@afmfl.org jgonzalez@afmfl.org  For help with outcomes and success measures, first contact Jessica Gonzalez and then our evaluation consultant, Shelley Robertson, 239-633-3241, info@snrobertson.com info@snrobertson.com

39. Questions?