AgrAbility NTW McGill QOL Webinar November 28, AM (MT) By Robert J. Fetsch, Extension Specialist, Professor Emeritus & Director, Colorado AgrAbility Project, Human Development & Family Studies Colorado State University AANTWMcGillQOL (Rev a)

Need speakers or headphones to hear the presentation Meeting > Manage My Settings > My Connection Speed – Dial-up not recommended Questions about presentation – If you have a web cam/microphone, click the “Raise Hand” icon to indicate that you have a question We will activate your microphone – If you don’t have a microphone, type into chat window and hit return Basic Webinar Instructions

4 quick survey questions Session recorded and archived with PowerPoint files at  Problems: use chat window or Basic Webinar Instructions

Increased interactivity = increased technical issues – Be patient while we push our boundaries – Meeting > Audio Setup Wizard Disconnection with presenters – Hang on – we’ll reconnect as soon as possible Disconnection with participants – Log in again Known Webinar Issues

Current SRAP QOL Impact Measurement & Plans for the Next Four Years By Robert J. Fetsch (CSU) With Thanks to Brad Rein for 22+ Years of Leadership ( )

Current SRAP QOL Impact Measurement & Plans for the Next Four Years By Robert J. Fetsch (CSU) Welcome to Aida Balsano, National Program Leader— Child and Family Development Division of Family and Consumer Sciences Strengthening Families, Farms, Communities and the Economy

Part 1: Current State of SRAP QOL Impact Measurement Part 2: Plans for the Next Four Years

Part 1: Current State of SRAP QOL Impact Measurement

Our AgrAbility Mission The AgrAbility Mission is to enhance and protect quality of life and preserve livelihoods. It’s about supporting and promoting growth and independence. Ultimately it’s about hope. Source: National AgrAbility Project. (2011). It’s about hope [DVD]. Author: Purdue University.

Why Do We Do Program Evaluation? To make sure we’re on track with our mission To find out what works To be accountable and to be good stewards of the tax $ we receive To use valid and reliable instruments to document our AgrAbility program impacts

History of National AgrAbility Project Evaluation Committee (NAPEC) Early 2006—Kathryn Pereira, Evaluation Specialist NAP U of WI, invited all SRAP’s to join in an AgrAbility evaluation study. NAPEC met approximately bi-monthly (2007- Present) via teleconference/face-to-face (N = 6-25 participants/meeting).

NAPEC Includes: 32 SRAP Team Members 2 NAP Evaluation Team Members—Bob Aherin & Chip Petrea Bob Fetsch as Chair since 2007

NAPEC Produced Results Created and tested a practical procedure that 15 SRAPs are using to assess how successful they are at increasing their clients’ quality of life levels (QOL) and their Independent Living and Operating Levels (ILOS).

NAPEC Produced Results Began to address SRAPs’ highest need for evaluation related assistance and training by measuring change or impact as a result of AgrAbility services with the McGill QOL and ILOS. This is according to the most current National AgrAbility Project needs assessment & program evaluation summary for 2010 (B. Aherin, personal communication, January 23, 2012).

NAPEC Produced Results Increased involvement by the number of SRAPs who used the McGill QOL and ILOS to collect pre-survey and post-survey data from their new clients from 2 in 2008 (VA & WI) to 6 in 2012 (CO, KS, NE, VA, WI, & WV).

NAPEC Produced Results Increased the number of SRAPs who are committed to use the MQOL and ILOS from 6 in 2008 to 15 in 2012 (AR, CO, KS, ME, MN, MO, NC, NE, OH, OK, TX, UT, VA, WI, & WV). NAP, UIUC, and CSUE THANK YOU!

How many SRAPs are collecting QOL and ILOS data from their new clients?

AgrAbility across the U.S. (source: National AgrAbility Project)

NAPEC Produced Results Published two refereed journal articles plus a report on the effectiveness of AgrAbility. – Christen, C. T., & Fetsch, R. J. (2008). Colorado AgrAbility: Enhancing the effectiveness of outreach efforts targeting farmers and ranchers with disabilities. Journal of Applied Communication, 92(1&2), – Fetsch, R. J., Helfrich, C. M., Field, W. E., & Olson, E. O. (2010, March 2). AgrAbility: A program that works. Goodwill Industries. – Meyer, R. H., & Fetsch, R. J. (2006). National AgrAbility Project impact on farmers and ranchers with disabilities. Journal of Agricultural Safety and Health, 12(4), – Another manuscript is in preparation.

History of NAPEC Who is an AgrAbility Client? An AgrAbility client is an individual with a disability engaged in production agriculture as an owner/operator, family member, or employee who has received professional services from AgrAbility project staff during an on-site visit.

History of NAPEC 2 Questions: – Do our AgrAbility clients increase their QOL levels? – Are our AgrAbility clients more able to live on, operate, and manage their farms/ranches if they choose?

History of NAPEC Six SRAP’s conducted a four-year longitudinal study to answer the 2 questions (June 2007-July 2011). CO, KS, NE, VA, WI, & WV

Measures Used in CO, KS, NE, VA, WI, & WV Study McGill Quality of Life (QOL) Survey & AgrAbility Independent Living & Operating Survey (ILOS) NAP Demographic Data

History of NAPEC By July 13, 2011 CO, KS, VA, & WI entered their 98 matched pre-post-survey data into Excel files and ed them to CSUE for entering and analyzing. – KS49 – WI31 – CO15 – VA 3 – Total98

Q: Do Our AgrAbility Clients Increase Their QOL levels? A: ?

McGill Pre- Post-Survey Changes (Single item Scale, Physical Well-Being, & Physical Symptoms)

McGill Pre- Post-Survey Changes (Support, Experiential Well Being & Psychological Well-Being)

McGill Pre- Post-Survey Changes (Total Score)

Q: Do Our AgrAbility Clients Increase Their QOL levels? A: Yes, they improve on the Total QOL Scale plus on all 6 subscales!

Q: Are Our AgrAbility Clients More Able to Live on, Operate, and Manage Their Farms/Ranches if They Choose? A: ?

AgrAbility Independent Living & Operating Survey (ILOS) (Manage Farm, Complete Chores, & Operate Machinery)

AgrAbility ILOS (Live in Home, Access Workspaces & Modify Machinery)

Q: Are Our AgrAbility Clients More Able to Live on, Operate, and Manage Their Farms/Ranches if They Choose? A: Yes!

The results of this four-state study show that all of the McGill QOL group mean scores increased from pre- to post-survey plus manage farm, operate machinery, and modify machinery***. We invite you to join us! See me today, send me an , or give me a call.

These results look promising, BUT…

How do we know these results are not due to something other than our AgrAbility services?

Part 2: Plans for the Next Four Years

NAP Priority 5 is Evaluation: Provide an accurate picture of AgrAbility’s outcome and impacts through SRAP SRAP Quality of Life survey How successful are we at increasing customer quality of life?

NAP Proposal Reviewer Comments Focus on accountability What’s the impact on clients? What difference does AgrAbility make? Need for greater Evidence-Based information that would support continuation of AgrAbility

NAP Proposal Reviewer Comments Collect QOL data and publish results so Brad Rein can show impact. Form a new control group for comparison. Document impacts and publish results.

Brad Rein Reported “Good News” and “Bad News.”

“Good News” AgrAbility is among the 45 federally funded programs that supported employment for people with disabilities in fiscal year AgrAbility is among the 10/45 programs with a review or study to evaluate the program’s effectiveness. Source: U.S. Government Accounting Office. (2012). Employment for people with disabilities; Little is known about the effectiveness of fragmented and overlapping programs (GAO Publication No ). Washington, DC. (p. i).

“Good News” “…The Department of Agriculture’s AgrAbility program conducted a review of its activities between 1991 and 2011 and found that 11,000 clients had been served, and that 88 percent of those clients continued to be engaged in farm or ranch activities.” Source: U.S. Government Accounting Office. (2012). Employment for people with disabilities; Little is known about the effectiveness of fragmented and overlapping programs (GAO Publication No ). Washington, DC. (p. 27).

“Bad News” “However, this study did not determine whether other factors may have contributed to participants’ positive outcomes.” “No impact study.” Source: U.S. Government Accounting Office. (2012). Employment for people with disabilities; Little is known about the effectiveness of fragmented and overlapping programs (GAO Publication No ). Washington, DC. (pp. 27, 80).

Brad Rein asked us to help respond. So far 15 SRAP’s are working to collect data from AgrAbility clients with an on-site visit (AR, CO, KS, ME, MN, MO, NC, NE, OH, OK, TX, UT, VA, WI, & WV). We welcome the rest of you to join us.

AgrAbility across the U.S. (source: National AgrAbility Project)

Experimental Group (N = 200 with matching pre- and post-surveys) Control Group (N = 100 with matching pre- and post-surveys)

Control Group (N = 100) Cannot be receiving any type of AgrAbility program services or onsite visits regardless of whether they are in USDA funded or Affiliate States.

Where do we find 100 ranchers and farmers for the Control Group? 1.Non-funded AgrAbility Affiliate States previously funded, but not currently funded to provide AgrAbility services, e.g. IA, ID, IL, MI, MT, and PA. 2.Farmers and ranchers with disabilities who call NAP’s for information from non-funded states.

Where do we find 100 ranchers and farmers for the Control Group? 3. Ranchers and farmers with disabilities who call in to currently funded SRAP’s but who decide not to become clients at this time. 4. Wait list group? 5. Fingerlake, NY Goodwill? 6. Where else?

NAPCSUEUIUC

Why Join Us? 1.Document your project’s effectiveness at increasing QOL and ILOS. 2.Enhance your chances of receiving funding next time with empirical evidence of your SRAP’s quality and effectiveness. 3.Increase your chances for outside funding by demonstrating your accountability. 4.Contribute to AgrAbility’s Mission.

Won’t You Join Us? Here’s how: 1.Send an to 2.Seek IRB approval from your Land-Grant University. 3.Study and use the same protocol. 4.Adapt CO to __ on pp. 1-2 & mail. 5.Enter your data into an Excel file that we will provide, proof perfectly & to me.

Thank you very much!

Questions About Control Group Do any of you have a wait-list for prospective clients? Where else can we find 100 farmers and ranchers for the Control Group?

Other Questions?

Protocol Procedure—CO, KS, VA, and WI mailed each new client the Pre-Survey, a cover letter, McGill Pre-Survey, and a stamped, self- addressed envelope with an invitation to complete and return it.

Protocol A McGill Post-Survey was mailed out when the client’s case was closed (CO, KS, VA, WI) using a modified Dillman method (KS, VA) with two-week intervals: 1) cover letter, post-survey, and stamped return envelope; and 2) two weeks later a follow-up cover letter, post-survey, and stamped return envelope was mailed. We ran Paired Sample t Tests.

Protocol Participants were given the choice of completing the survey themselves or of having the items read aloud by the AgrAbility team member. No one was coerced to complete and return their surveys.

A: Yes, they improve on the Total QOL Scale plus on all 6 subscales! -MQOL Total Score -McGill SIS -Physical Well-Being -Physical Symptoms -Support -Experiential Well-Being -Psychological Well-Being

Pros of Using the McGill QOL and Independent Living and Operating Surveys Pros – Today 15 states are using the McGill QOL & ILOS (AR, CO, KS, ME, MN, MO, NC, NE, OH, OK, TX, UT, VA, WI, & WV). – The McGill QOL is a well-tested, valid, and reliable tool. – It provides data that can be aggregated and/or compared across state lines.

Pros of Using the McGill QOL and Independent Living and Operating Surveys Pros – It is short (17 items) and is easy to complete. – The combination of the McGill QOL Tool with the AgrAbility ILOS provides empirical evidence that AgrAbility works.

Cons of Using the McGill QOL and Independent Living and Operating Surveys Cons – It costs you a little extra time and effort. – To use the McGill QOL Tool with non-English- speaking or severely visually impaired farm/ranch workers, it is recommended that one read aloud each item and score the client’s response for them.

Provide assistance and resources to farmers with disabilities, and/or their family members with disabilities, that allows them to begin or continue farming AgrAbility Mission