1. AIA Web Conference on Culturally Competent Practices in Program Evaluation Considering Culture in Evaluation and Applied Research Rodney K. Hopson Duquesne University hopson@duq.edu

2. 2 Acknowledgements Collaborative and funded work from lectures, co- presentations, research grants, published papers, commissioned evaluation work: National Science Foundation California Endowment American Evaluation Association/Center for Disease Control Relevance of Culture in Evaluation Institute Heinz Endowments Lincoln U, Manchester Metropolitan U, Claremont Graduate U Development of pipeline programs to encourage additional practice AEA Graduate Diversity Internship: http://www.eval.org/gedip.htm RWJF Evaluation Fellowship: http://www.rwjf- evaluationfellows.org/

3. Learning objectives: Recognize relevance and value-addedness of culture in evaluation theory and practice Identify strategies for assisting evaluators and agencies in becoming culturally responsive Encourage the development of tips and tools for working with agencies who work with traditionally underserved and/or minoritized groups

4. 4 Warm up/Introductions Who are you? What do you do? What disciplines, connections, experiences do you bring into evaluation? How are your disciplines,…important in the way you think about evaluation? How do you currently think about the role of culture in evaluation and applied research? How does your background help you think about the role of culture in evaluation and applied research?

5. 5 Reflections of an ethnoevaluator/culturally responsive evaluator/sociolinguistevaluator Curry School bridge at UVa between education & arts/sciences Linking (trans)disciplines: (educational) evaluation, sociolinguistics, anthropology (Hopson, 2000) Exploring connections between ethnography and evaluation (Hopson, 2005) Beyond “technocist imperatives of most program evaluations” Extending Fetterman’s pioneering work and Stakes’ seminal notions on countenance of educational evaluation Challenging and resisting mainstream thinking on ways of thinking about cultural diversity and difference (Hopson, 2009) Legitimizing knowledge of communities of color or traditionally underserved communities Building spaces of hope, praxis and social action for same communities

6. New answers to increasing complex questions @ culture

7. 7 Warm up/Introductions/Review Who are you? What do you do? What disciplines, connections, experiences do you bring into evaluation? How do they help you think about evaluation? What previous backgrounds or experiences do you bring that might assist you in thinking culture in evaluation? How do you currently think about the role of culture in your practice? How do you intend to use this discussion of culture in evaluation to influence your practice?

8. 8 Culture the way of life of a group of people, the complex of shared concepts and patterns of learned behavior that are handed down from one generation to the next through the means of language and imitation. (Barnouw, 1985) the ever-changing values, traditions, social and political relationships, and worldview created, shared and transformed by a group of people bound together by a combination of factors that include a common history, geographic location, language, social class, and religion… (Nieto, 1999)

9. 9 Cultural Contexts, Characteristics, Locations, and Perspectives Demographic, sociopolitical, contextual dimensions Characteristics as dynamic, multifaceted, learned, created, influenced From personal to global position Majority vs. minority More than about race but inclusive of other identifications (SenGupta, Hopson & Thompson-Robinson, 2004; Nieto, 1999)

10. Emerging approach/model used to guide evaluation System and culmination of evaluation strategies Theoretical and political positioned (as are all evaluations) Demographic, sociopolitical, and contextual dimensions, locations, perspectives, and characteristics of culture matter Privileging lived experiences, esp. communities and populations of color Avoiding the phenomenon of “evaluating down” Culturally Responsive Evaluation

11. (Hopson, 2009) Theoretical and Practical Intersection of CRE: Advocacy, Race, Power Decolonizing/ indigenous positions, epistemologies, and frameworks Critical theories and epistemologies of race Social agenda and advocacy theories, models and approaches in evaluation

12. 12 Cultural Competence A set of academic and interpersonal skills that allow individuals to increase their understanding and appreciation of cultural differences and similarities within, among, and between groups. This requires a willingness and ability to draw on community-based values, traditions, and customs, and to work with knowledgeable persons of and from the community in developing focused interventions, communications and other supports. (Orlandi, 1992)

13. Talking as culturally responsive

14. 14 Scenario: Dialogue for Diversity and Social Change (DDSC) first impressions What elements of culture, at what levels, seem salient to this scenario at first glance? What elements of culture are you assuming will not be as salient, based upon your initial impressions? What is your own cultural position in relation to these cultural elements? How do we think about the relevance of culture in all stages of evaluation for those in public health, education, and other helping professions?

15. 15 Culturally Responsive Evaluation Framework Step 1: Prepare for the evaluation. Step 2: Engage stakeholders. Step 3: Identify the purpose of the evaluation. Step 4: Frame the right questions. Step 5: Design the evaluation. Step 6: Select and adapt instrumentation. Step 7: Collect the data. Step 8: Analyze the data. Step 9: Disseminate and use the results. (Frierson, Hood & Hughes, 2002)

16. Cultural Responsive Evaluation 7 Collect the data 8 Analyze the data 3 Identify purpose of the evaluation 4 Frame the right questions 1 Prepare for the evaluation 5 Design the evaluation 6 Select and adapt instrumentation 2 Engage stakeholders 9 Disseminate and use the results 16

17. 17 1 Prepare for the Evaluation Examine the sociocultural context of the evaluand, including History Community Intersecting cultural identifications Assemble an evaluation team whose collective lived experience is appropriate to the context of the evaluand.

18. 18 Engage Stakeholders Develop a stakeholder group representative of the population served by program. Seek to include direct and indirect consumers. Pay attention to distributions of power. Include multiple voices.

19. 19 Identify Purpose of Evaluation Alignment of purpose with its intended consumers Program operates in ways that are respectful of cultural context Program resources are equitably distributed Benefits of program evaluation are equitable

20. Applying Stages 1-3 (LaFrance, 2004) Reflections on embedding culturally competent evaluation in Indian Country Build understanding of values that underlie programs and projects and create value-added evaluative contribution Engage stakeholders in participatory manner Build ethic of participation and capacity building that values community, relationships, respect Frame purpose by building conceptual picture/model Careful of “too sequential and narrative driven” logic model 20

21. 21 Scenario: Stages 1-3 What elements of background and context are important here? What more would you want to know? Who was included on the evaluation team and what presumed skills, traits do they bring to the evaluation process? Based on the stated purpose of this evaluation, who do you understand to be the major stakeholders?

22. 22 Frame the Right Questions Include questions of relevance to significant stakeholders. Determine what will be accepted as evidence in seeking answers to the questions. Examine whose voices are heard in the choice of questions and evidence. Is the lived experience of stakeholders reflected in these choices?

23. 23 Design the Evaluation Build design appropriate to both evaluation questions and cultural context. Seek culturally appropriate methods that combine qualitative and quantitative approaches. Try to collect data at multiple points in time, extending the time frame of the evaluation as needed. Construct control or comparison groups in ways that respect cultural context and values.

24. 24 Select & Adapt Instrumentation Establish reliability and validity of instruments for the local population. Norms must be appropriate to the group(s) involved in the program. Language and content of instruments should be culturally sensitive. Adapt instruments as needed and conduct additional validation studies.

25. Applying Stages 4-6 (Jay, Eatmon, & Frierson, 2005) Evaluation of undergraduate STEM research program designed for students of color Deliberate design of evaluation team intimately connected with program of study and background of program, including similar lived experiences of participants Questions were sensitive to lived experiences of participants and focused on substance of participant experiences Beyond attention to traditional issues of success but exploring issues of persistence as students of color Attempt to address nuances and subtleties relative to experiences and impact of program 25

26. 26 Scenario: Stages 4-6 What/whose perspectives are represented in the evaluation questions, and what other questions might have been posed? Whose perspectives are accepted as credible evidence? Credible to whom? How well does the time frame in this study match the needs and rhythms of this context?

27. 27 Collect the Data Procedures used to collect both qualitative and quantitative data must be responsive to cultural context. Nonverbal as well as verbal communications provide qualitative data. Careful training of data collectors in both technical procedures and culture is key. Shared lived experience provides optimal grounding for culturally-responsive data collection.

28. 28 Analyze the Data Cultural context is a necessary component of accurate interpretation. Disaggregate data to examine diversity within groups. Examine outliers, especially successful ones. A cultural interpreter may be needed to capture nuances of meaning. Stakeholder review panels can assist in accurate interpretation.

29. 29 Disseminate & Use the Results Cultural responsiveness increases both the truthfulness and utility of the results. Communication mechanisms must be culturally responsive. Inform a wide range of stakeholders. Make use consistent with the purpose of the evaluation. Consider community benefit

30. Applying Stages 7-9 (Manswell Butty, Reid, & LaPoint, 2004) Discussion of urban school-to-career intervention program using culturally responsive evaluation approach Input derived from school stakeholders on how best to analyze and interpret data in ways that provided meaning in particular contexts Findings disaggregated by gender and age to get breakdown of career attitudes and beliefs for participants Findings provided to numerous stakeholders in particular ways (e.g. student findings presented in student-friendly manner) 30

31. 31 Scenario: Stages 7-9 What additional data collection procedures might have been useful to consider in designing a culturally responsive evaluation? Given the findings briefly summarized, what aspects of cultural context might add meaning to guide recommendations? Were results shared in a culturally congruent way?

32. 32 Take Home/Next Steps/Questions Culture is relevant, if not central, to all aspects of the evaluation process. Thinking through project plans Incorporating CRE approaches and thinking in project planning and evaluation steps What key ideas do you take away from CRE? How can you inform your own evaluation project with cultural lenses? What questions do you continue to have about CRE?

33. 33 References Barnouw, V. (1985). Culture and personality (4 th ed.). Homewood, IL: The Dorsey Press. Frierson, H. T., Hood, S., & Hughes, G. B. (2002). Strategies that address culturally-responsive evaluation. In Frechtling, J., The 2002 user-friendly handbook for project evaluation. National Science Foundation. Hopson, R. (2009, forthcoming). Reclaiming knowledge at the margins: Culturally responsive evaluation in the current evaluation moment. In Ryan, K. & Cousins, B. (Eds.). International Handbook on Evaluation. Thousand Oaks, CA: Sage. Hopson, R.K. (2005). Commentary: Reinventing Evaluation. Anthropology and Education Quarterly, 36(3).

34. References Hopson, R. K. (2003). Overview of multicultural and culturally competent program evaluation: Issues, challenges and opportunities. Woodland Hills, CA: The California Endowment.Hopson, R.K., Ed. (2000). How and Why Language Matters in Evaluation. New Directions for Evaluation, No. 86. San Francisco: Jossey-Bass. Jay, M., Eatmon, D., & Frierson, H. (2005). Cultural reflections stemming from the evaluation of an undergraduate research program. In S. Hood, R. K. Hopson, & H. T. Frierson (Eds.) The role of culture and cultural context: A mandate for inclusion, the discovery of truth, and understanding in evaluative theory and practice (pp. 201-216). Greenwich, CT: Information Age Publishing, Inc. 34

35. References King, J. A., Nielsen, J. E., & Colby, J. (2004). Lessons for culturally competent evaluation from the study of a multicultural initiative. In M. Thompson-Robinson, R. Hopson & S. SenGupta (Eds.), In search of cultural competence in evaluation: Toward Principles and Practices, New Directions for Evaluation, Number 102 (pp.5-19). San Francisco: Jossey-Bass. La France, J. (2004). Culturally competent evaluation in Indian Country. In M. Thompson-Robinson, R. Hopson, & S. SenGupta (Eds.), (pp. 39-50). San Francisco: Jossey-Bass. 35

36. 36 References Nieto, S. (1999). Affirming diversity: The sociopolitical context of multicultural education (3 rd ed.). Boston: Allyn & Bacon. Orlandi, M. A. (Ed.) (1992). Cultural competence for evaluators: A guide for alcohol and other drug abuse prevention practitioners working with ethnic/racial communities. U.S. Department of Health and Human Services, Office for Substance Abuse Prevention. DHHS Publication No. (ADM)92-1884. SenGupta, S., Hopson, R., & Thompson-Robinson, M. (2004). Cultural competence in evaluation: An overview. In M. Thompson-Robinson, R. Hopson & S. SenGupta (Eds.), (pp.5- 19). San Francisco: Jossey-Bass.