THE USE OF TWO TOOLS TO EVALUATE HEALTHY START COALITIONS Susan M. Wolfe, Ph.D.

Four Healthy Start Programs Context

Partnership Self-Assessment Tool Coalition Member Assessment The Surveys

Partnership Self-Assessment Tool -- PSAT  Center for the Advancement of Collaborative Strategies in Health  Measures:  Synergy  Leadership  Efficiency  Administration and Management  Non-Financial Resources  Financial and Other Capital Resources  Decision Making  Benefits of Participation  Drawbacks of Participation  Satisfaction with Participation

Coalition Member Assessment -- CMA  Tom Wolff – The Power of Collaborative Solutions  Measures:  Vision: Planning, Implementation, Progress  Leadership and Membership  Structure  Communication  Activities  Outcomes  Relationships  Systems Outcomes  Overall Rating (benefits)  Open Ended Questions

Survey Use

Four Programs, Four Coalitions: Coalition F  Small City  Non-Profit, Community Based Organization  Original coalition left the project.  New coalition formed in 2012  Met quarterly  Two completing coalitions/groups  goal setting and rebuilding  Project and coalition ended May 31, 2014

Coalition F  No survey  Goals and objectives, Local Health System Action Plan, Appreciative Inquiry  Merge in December 2013  In May 2014 program ended  Never reached readiness

Four Programs, Four Coalitions: Coalition H  Large City  Non-Profit, Community Based Organization  Operational since about 2000  Met quarterly  Additional coalition formed 2012  Project and coalition ended May 31, 2014

Coalition H  Previous evaluator survey titled the “Partnership Assessment Tool.”  “Satisfaction with Participation”  Other questions not on the PSAT  Just about everyone was satisfied  Feedback from Presenting Results  Different versions for consumer and professional participants  Comment: “program participants do not understand that they are coalition members. They come to meetings, but do not realize that means membership.”

Four Programs, Four Coalitions: Coalition D  Large City  County (Public) Hospital  Formed in 1994  Met quarterly  Only infant mortality coalition  Beginning in January 2015 will move to monthly meetings / collective impact model

Coalition D  Coalition Member Assessment – 2011 and 2012  Range of answers  Asked to delete items in 2012 to shorten the survey  2013 – Customized Coalition Survey  Included PSAT satisfaction scale 7 respondents -- little or no satisfaction with influence and role on coalition

Four Programs, Four Coalitions: Coalition S  Large City  Metro Health Department  New coalition January 2012  Meets monthly  Only infant mortality coalition  Ongoing – moving to collective impact model with neighborhood action networks

Coalition S  Staff piloted PSAT and CMA  Manager preferred PSAT  Staff overwhelmingly preferred CMA  Administered in 2013 and 2014  Ambiguous items revised / split up  Compared years  Able to see changes over time as coalition developed  Coalition development assessment combined with results from SWOT analysis

What did we learn? Takeaway

Type of Coalition and Coalition Readiness Matter  Meet and Talk / Information Sharing versus Action  Type of coalition relates to survey instrument used  Meet and Talk / Information Sharing less likely to take action based on results

Participatory Approach  Staff selected tool  Coalition member feedback helpful to revise tool  Increases utility of findings

Funding Agency / Local Effects  What a coalition does depends on its requirements and the local culture.  Loose guidelines / little accountability  See what other coalitions in the area do

Partnership Self-Assessment Tool  Satisfaction scale results varied  Program management preferred it, but staff did not  Review for wording for program participants and community members  “Diversion of time and resources away from other priorities or obligations”  “Insufficient influence in partnership activities”  Focus -- inner-workings of the coalition

Coalition Member Assessment  Scales all worked well – good internal consistency  Some items ambiguous and needed to be divided into two:  “There are opportunities for members to take leadership roles, and members are willing to take them.”  “The coalition has a viable organization structure that functions competently”  Review and simplify wording for program participants and community members  Includes assessment of inner workings of the coalition and perceptions of outcomes and impact  Includes open ended items that allow for qualitative input

Susan M. Wolfe, Ph.D. CEO Susan Wolfe and Associates, LLC Cedar Hill, Texas (682)