1. If we build a partnership will they come? Integrating Needs Assessment, Process Evaluation, and Impact Evaluation Ronald Jester and Robert Wilson, University of Delaware rcjester@udel.edurcjester@udel.edu, rwilson@udel.edurwilson@udel.edu

2. A Program Evaluation may include Process Indicators or Impact Indicators or A combination of Process and Impact Indicators

3. A Process Evaluation Analyzes Program Inputs, such as Outreach Activity Services Delivered Adherence to Grant Proposal Day-by-day Staff Activity Costs Materials

4. An Impact Evaluation Analyzes Program Outputs, such as Productivity Farm output Individual farmer or waterman output Efficiency Benefits/Costs Client Satisfaction with services Improvements in Quality of Life

5. AgrAbility evaluations may include both process and impact indicators

6. The Delaware Maryland Program (DMAP) employs a systems model for program evaluation This model starts with the target population - -2400 farmers with disabilities The model tracks the target population through a service delivery system The system includes a number of filters that regulate the flow of AgrAbility clients The model includes a number of efficiency measures that identify impediments to efficient client flow

7. The AgrAbility Process includes 5 steps: 1. Identify farmers with disabilities. 2. Diagnose their problems. 3. Identify the services that will alleviate the problems. 4. Find resources to support for the services. 5. Deliver the services.

8. The Delaware-Maryland project relies on “partnerships” to deliver services.

9. The efficiency of the project depends on 1. Locating the target population 2. Engaging the target population 3. Linking the target population with the partners 4. Routing the clients to the partners in the network 5. Delivering appropriate services 6. Producing specific outcomes (impacts) as a result of the services

10. Overall efficiency is gauged by the proportion of the target population Located Diagnosed Referred to services Provided services Helped by services

11. Areas of Evaluation Process EvaluationImpact Evaluation Locating Clients Diagnosing Clients Referring Clients to services Providing Services Measuring the Impact of Services

12. The Model (Figure 1) portrays the flow of a target population through a series of filters. As the target population of 2400 farmers flows through the service delivery system an increasing number drop out at each stage.

13. How to interpret Figure 1 The target group is the estimated 2400 disabled farmers who reside in Delaware and Maryland. This estimate was generated from a random survey. A farmer may seek help the AgrAbility program (columns B and C) or a program can reach out to a farmer (columns D,E, and F). Line 1 shows that 280 of the estimated 2400 disabled farmers were contacted either by AgrAbility or another program. Line 2 shows that 168 farmers out of the 280 farmers who were contacted were subsequently evaluated. Line 3 shows that 120 of the farmers who were evaluated were subsequently referred for service. Line 4 shows that 105 of those who were referred for service subsequently received a service. Line 5 shows that 90 out of the 105 farmers who received a service subsequently experienced an improvement in their quality of life.

14. Figure 1 Systems Model of Delaware-Maryland AgrAbility Program (DMAP), Target Population = 2400 Disabled Farmers Sought Help From Did Not Seek Help Contacted by DMAP 110 Other 110 DMAP 60 Other ? None ? Assessed 110 Assessed ? Referred 80 Referred ? Service Provided 60 Service Provided ? Quality of Life Improved 50 Quality of Life Improved ? Assessed 58 Assessed ? Not Assessed Referred 40 Referre d ? Not Referred Service Provided ? No Service Provided Quality of Life Improved ? Quality of Life Improved 40 Quality of Life Improved ? B C D E F 1 2 3 4 5 Service Provided 45

15. Six Process Measures are employed Each measure indicates the efficiency of a different part of the system

16. 1.Market Penetration The percentage of the target population that contacts social service agencies for help. Include all agencies- AgrAbility and other agencies that provide services to farmers with disabilities

17. Market Penetration Measurement is based on Target Population: the estimated 2400 disabled farmers residing in MD and DE

18. The farmers who were contacted are indicated in line 1 of Figure 1

19. Figure 1 Systems Model of Delaware-Maryland AgrAbility Program (DMAP), Target Population = 2400 Disabled Farmers Sought Help From Did Not Seek Help Contacted by DMAP 110 Other 110 DMAP 60 Other ? None ? Assessed 110 Assessed ? Referred 80 Referred ? Service Provided 60 Service Provided ? Quality of Life Improved 50 Quality of Life Improved ? Assessed 58 Assessed ? Not Assessed Referred 40 Referre d ? Not Referred Service Provided ? No Service Provided Quality of Life Improved ? Quality of Life Improved 40 Quality of Life Improved ? B C D E F 1 2 3 4 5 Service Provided 45

20. 1.Market Penetration (220)/(2400) The percentage of the target population that contacts social service agencies for help

21. 1.Market Penetration (B1+ C1)/(2400)=.09 The percentage of the target population that contacts social service agencies for help

22. Market Penetration is a crude measure of public service advertising reach.

23. 2.Outreach (B1 + C1+DI)/TP % of the target population (TP) identified and contacted by a specific program

24. The target population that was reached is shown in cells B1, C1, and D1 of Figure 1.

25. Figure 1 Systems Model of Delaware-Maryland AgrAbility Program (DMAP), Target Population = 2400 Disabled Farmers Sought Help From Did Not Seek Help Contacted by DMAP 110 Other 110 DMAP 60 Other ? None ? Assessed 110 Assessed ? Referred 80 Referred ? Service Provided 60 Service Provided ? Quality of Life Improved 50 Quality of Life Improved ? Assessed 58 Assessed ? Not Assessed Referred 40 Referre d ? Not Referred Service Provided ? No Service Provided Quality of Life Improved ? Quality of Life Improved 40 Quality of Life Improved ? B C D E F 1 2 3 4 5 Service Provided 45

26. 3.Assessment Efficiency. (B2+D2)/(B1+D1) Assessment Efficiency is the percentage of potential clients eligible for assessment that was assessed The clients who were assessed are show in cells B2 and D2 of Figure 1. Clients eligible for assessment are shown in cells B1 and D1 of Figure 1

27. 3.Assessment Efficiency. (100 +58)/ (110 +60)=.93 The percentage of potential clients eligible for for assessment that was assessed

28. 4.Referral Efficiency. (B3+D3)/(B2 with positive screen + D2 with positive screen) Referral efficiency is the percentage of clients with a positive screen that was referred for service. Clients who were referred for services are shown in cells B3 and D3. Those with a positive screen are in cells B2 and D2.

29. Figure 1 Systems Model of Delaware-Maryland AgrAbility Program (DMAP), Target Population = 2400 Disabled Farmers Sought Help From Did Not Seek Help Contacted by DMAP 110 Other 110 DMAP 60 Other ? None ? Assessed 110 Assessed ? Referred 80 Referred ? Service Provided 60 Service Provided ? Quality of Life Improved 50 Quality of Life Improved ? Assessed 58 Assessed ? Not Assessed Referred 40 Referre d ? Not Referred Service Provided ? No Service Provided Quality of Life Improved ? Quality of Life Improved 40 Quality of Life Improved ? B C D E F 1 2 3 4 5 Service Provided 45

30. 5.Service Delivery Efficiency. (B4 + D4)/B3 +D3)= (60+ 45)/(80 + 40)=.88 Service delivery efficiency is the percentage of clients referred for service that received service. Clients who received a service are show in cells B4 and D4. Clients who were referred for service are shown in cells B3 and D3.

31. The Delaware-Maryland Project employs the SF-36 Quality of Life Instrument as an Impact Measure

32. Additional impact measures that are specific to farmers with disabilities and assistive technology will be incorporated in the evaluation These measures are being developed by a committee drawn from AgrAbility programs and national program staff

33. The SF-36 is the most widely used quality of life instrument in health and social services

34. T he SF-36 is calculated for 8 health scales (0-100) 1. Physical limitations 2. Limitations in social activities 3. Limitations in work and home roles 4. Bodily pain 5. Psychological distress and wellbeing 6. Limitations because of emotional problems 7. Energy and fatigue 8. General health perceptions

35. The SF-36 has been validated through a numerous studies Several thousand scientific articles employ the SF-36 as a quality of life measure for measuring program impact

36. All active Delaware and Maryland clients are currently completing SF-36 interviews

37. The SF-36 serves two purposes The SF-36 is used in the initial assessment to develop a service plan. The SF-36 is also used in a 1-year follow-up evaluation to assess changes in the quality of life subsequent to the initial assessment.

38. 6.Improved quality of Life: (# of Clients with improved quality of life) / (# of Clients that Received AgrAbility Service) Clients with improved quality of life are shown in cells B4 and D4. Those who received a service from AgrAbility are shown in cells B3 and D3. (B4 + D4)/(B3 +D3)= (50 + 40)/(60 + 45)= 90/105=.86

39. Finally, 2 questions for the audience: Which of the measures focus on the operation of partnerships in AgrAbility programs? What other techniques should be used to evaluate partnerships?