Participatory Evaluation Mary Phillips, BME Former Circles of Care Program Coordinator, Oakland and an Evaluator, Los Angeles, CA

It’s Not Just an Evaluation Methodology “We all have a lot in common. We all care very deeply about the communities that we come from and about American Indian and Alaska Native communities across the country. We all share deep concern about the well-being and future of Indian children and families. We are all very self-sacrificing. We are in this business because of our personal beliefs and our desire to make a difference. Our personal backgrounds, our training, our education, and our professional experience all contribute to our desire to make a difference.” Holly-Echo Hawk, M.S., Echo Hawk and Associates

It’s Not Just an Evaluation Methodology Participation Evaluation is a key component of the systems of care evaluation process that informs the mission, vision and authority of the Tribe/community throughout the Circles of Care program. Participation evaluation allows the community to voice its cultural values and promote cultural appropriateness in systems changes development at all levels.

It’s Not Just an Evaluation Methodology The Circles of Care initiative represented one of the first opportunities for American Indian/Alaska Native AI/AN families and their communities have substantial input into the design of services to address the needs of their children, from their own understanding of these needs. The Standard Infrastructure Program Announcement, INF 05 PA and the Circles of Care III NOFA have included community involvement and participation measures that you MUST address throughout the application and are considered in the review process.

How it Works blends evaluation process components to actively involve members of the community to collect qualitative data that better informs the system. Having members who are knowledgeable about cultural norms of the community bared an unobtrusive context to present systems change.

The Process of Participatory Evaluation Planning the assessment will require a working process between the project staff and evaluation staff and the key informants of each community. Taking into consideration the current status of delivery systems for Native youth and mental health, developing the assessment tool(s) was pivotal in getting necessary information on contextual issues of the system. Example: Community Readiness Model was developed at the Tri-Ethnic Center for Prevention Research at Colorado State University to measure systems change in Native American and other communities.

Community Readiness Model An innovative methodology to evaluate systems change and focus community efforts toward realizing its objectives. It identifies characteristics related to problem awareness and readiness for change within the community, fostering sound strategic planning, ongoing feedback, and realistic assessments of accomplishments.

Community Readiness Model The tool focused on collective information from these Native American resources: 3 community identified leaders, 3 agency directors, 2 cultural advisors, 3 community members and 2 youth group representatives. Input from key stakeholders and community members was collected through interview questions related to: – Community Efforts, Knowledge of Efforts, Leadership, Community Climate, Knowledge About the Issues, Resources for Prevention Efforts.

Community Readiness Model The information collected allowed our program to measure where the urban Native American community stands with respect to its knowledge and readiness to implement a specific program (Strategic Plan for Urban Native Americans and their families Oakland, CA

Ongoing Particpation The evaluation participation process also incorporated resource development within the community. Linkages were made throughout the project as community coordinators and cultural leaders were provided information on the core elements of the system of care. This process required continuous interaction between agencies and cultural groups that concentrated the efforts to achieve child and adolescent behavior goals among agencies and family empowerment through focus groups.

Items from Standard Infrastructure Program Announcement and Circles of Care NOFA Respond to each of these required information items with statements to let the reviewer know what work has been done; the work that the program intends to do; and how you will address each issue. Though not all requirements listed here are given points, providing this information can help understand the ability of the program to connect and involve the community. These should be written in Sections B. Proposed Approach, C. Staff, Management and Relevant Experience, D. Evaluation and Data

Items from Standard Infrastructure Program Announcement and Circles of Care NOFA Program Goals: –Engage their community members in assessing service system needs, gaps, potential resources, and plan infrastructure development strategies that meet those needs. –Increase the participation of families, tribal leaders, and spiritual advisors in planning and developing service systems and treatment options based on the values and principles of the AI/AN community served by the project. – (Circles of Care III NOFA, p.2)

Items from Standard Infrastructure Program Announcement and Circles of Care NOFA Grantees must use funds to carry out the following required activities: –Facilitate culturally respectful strategic planning activities engaging community members, key stakeholders, youth, elders, spiritual advisors, and tribal leaders to identify outcome expectations and measures; (Circles of Care III NOFA, p.3)

In “Section D: Evaluation and Data,” add the following bullet: –Describe how project staff will work with evaluation staff to support the evaluation effort. The program planning activities and the evaluation activities will be expected to inform each other in a constant cycle, locally and collectively. (Circles of Care III NOFA, p.12) The Circles of Care process values participatory, community-based outcomes that are reflected heavily in the planning process. Items from Standard Infrastructure Program Announcement and Circles of Care NOFA

Providing a venue for the Tribal/community to receive information and respond to with appropriate audience members can include meetings at all levels within the community. Example: Two community-visioning meetings were conducted at 2 separate locations within the community, and were documented based on invited quests, attendance, activities planned to collect feedback, and information passed on to community participants. This type of community visioning planning resulted in setting priority areas to build a Youth Development Facility.

Participatory Feasibility Assessment Complete the feasibility assessment and process evaluation, leading to adoption of the model. (Circles of Care III NOFA, p.4) –Strengthening Tribal capacity for evaluation of service systems through ongoing involvement with staff and evaluation partners. –Careful deliberation on setting up effective strategies with an “evaluation team”, to engage all participants that have a stake in the evolution of the infrastructure. –This integration of tasks contributes to and OPEN style of communication that in turn enhanced the collaboration of evaluation and program staff.

Participatory Evaluation related to President’s New Freedom Commission Goals The agenda of the President’s New Freedom Commission’s Report towards a transformation of mental health systems states: –2. Mental Health Care is Consumer and Family Driven Involve consumers and families fully in orienting the mental health system toward recovery. (Circles of Care III NOFA, p.5) The consumer of the community has a voice in the process and is able to tell their story at the point in the process they choose. Parents are empowered through ongoing interaction in the process.

Participatory Evaluation related to President’s New Freedom Commission Goals Circles of Care approach emphasizes flexibility as stakeholders shift positions and encourage converge of a the cultural knowledge base within children’s mental health. Example: A parent that attends parent focused support groups from the AI/AN clinic and has a child receiving services that is also attending the school in the same community will better inform on how services are impacting the service needs and accessibility.

Participatory Evaluation related to President’s New Freedom Commission Goals The outcome of involving parents can result in a cultural definition of Severely Emotionally Disturbed: –“Emotional disturbance is a temporary disharmony often involving the family, school, and community, which may affect the mental, physical, spiritual, and/or emotional well being of its members.” Choctaw Nation of Oklahoma, Circles of Care I grantee.