IMPLEMENTING COMPLEX EVIDENCE- BASED PROGRAMS FROM SERVICE TO SCIENCE AND BACK AGAIN REAP Conference Santa Fe, NM March 20, 2008

STUDY DETAILS RESEARCH TEAM Susan Carter, PI Gladys Levis-Pilz Heather Yazzie Support provided by a FIRE grant from the Office of Juvenile Justice and Delinquency Prevention, US DOJ

THE PROBLEM We are faced with the paradox of non- evidence-based implementation of evidence- based programs (Drake, Gorman, & Torrey, 2002) Great emphasis on identifying evidence-based programs, but little attention paid to their implementation beyond fidelity discussions, especially for complex programs

RESEARCH QUESTIONS (After Fixen et al, NIRN, 2005) BASED ON STAGES OF PROGRAM IMPLEMENTATION Exploration and Adoption Program Installation Initial Implementation Full Operation Innovation and Sustainability

DESIGN AND METHODS BASED ON NIYLPS PROJECT VENTURE Created by McClellan Hall, Founder of National Indian Youth Leadership Project, Gallup, NM Grew out of camp program into year round model in 1990 Named by NREPP as a Model Program in 2004 Re-reviewed by NREPP, designated evidence-based program in 2007 Universal prevention model based on experiential, youth development principles for Native American mid-school youth

Project Venture 3 levels of non-negotiables Population served Program components Positive approach Outdoor adventure and experiential challenge –Experiential education –Wilderness outdoor adventure –Physical education –Nature challenge Service learning –Active learning –Academics in action –School or community service –Parent involvement Native culture based leadership –Culture and values –Language and history –Cross-cultural exchange and awareness –Elders Program delivery 200 hours of contact annually Risk management principles followed, including sequential progression of elements including –Physical safety –Mental health safety Youth to follow four levels of progression through program –In school (highly recommended) –After school –Weekend and overnight –Intensive camp experience Outcomes measured Focus on Native youth Middle school aged youth* High school aged Service Staff Mix of risk levels (program should not have stigma of prevention program)* Bold: Basic non-negotiable (element must be in place, even from beginning) Regular: Phased-in non-negotiable (element must be in place, but may be phased in over time) Italic: Recommendation (element is recommended, but not required) *:Modification allowed (element may differ if program is appropriately modified)

DESIGN AND METHODS Phase 1: In-depth Case Studies of a few PV programs –Identify all known PV adoption sites in US –Select 8 diverse sites considering rural/urban, new/experienced agency, agency type, etc. –Send invitation letter –Conduct on-site program observations –Conduct semi-structured interviews –Review documents related to implementation –Complete PV Replication Instrument –Complete saw/read/heard about it table

8 Project Venture: currently ~36 replication sites = PV replication site There are approximately 36 replication sites across 17 states

Interview Guide Questions Exploration and Adoption How did you hear about PV in the first place? How/why was the decision made to select? What are the components of your program? Is the programming running fully or partially? Community Support? School Support? Who makes the decisions for PV? Is it important that your facilitators be American Indian? Program Installation How did you prepare to implement PV? Who made those decisions? Staffing? Financing? Restructuring? What about now, how is PV implemented? Staffing? Financing? Restructuring? Initial Implementation Participants, who? How are they selected? Risk levels? PV components: which?, how configured? Full Operation How did you implement PV components? Did you begin with all components or phase them in? Who benefits? How do you feel about PV? What kinds of out comes have you had? Are you measuring outcomes? Tell me about this. Would you share data with NIYLP? Do you use it to improve programs? Does your program hold true to the vision of PV? (go through Program Implementation instrument with them) Innovation and Sustainability How do you manage change in population, financing, community political/tribal, drift/ fidelity? Any plans to sustain? How?

DESIGN AND METHODS Phase 2: Broad study of all PV programs –Develop survey based on Phase 1 findings –Administer via web, mail, phone to all known PV sites

PRELIMINARY FINDINGS FROM CASE STUDIES From earlier Project Venture studies, Blooms Taxonomy seemed a useful framework upon which to hang observational data Life lessons available to individual youth increased through exposure to other facilitators and participants and these lessons also promoted higher levels of cognitive and affective learning (as detailed below).

CASE STUDY FINDINGS (CONT.) The researchers attention kept returning to an idea of evolution or levels resulting from provider-participant and participant- participant interactions. These thoughts in turn led to a review of the educational psychologist Benjamin Blooms taxonomy of levels of learning (Bloom, B. (ed.) Taxonomy of Educational Objectives: The Classification of Educational Goals,1956, New York: Longmans Green).

CASE STUDY FINDINGS (CONT.) This classic theory provides a classification of levels of thinking and feeling behaviors thought to be important in the process of learning. The taxonomies are both descriptive and prescriptive as it orders categories of cognitive, affective, and psychomotor skills. From basic/low levels to ever more abstract/higher levels of learning.

While experimental studies are crucial for advancing our understanding of what works in adventure programming, the qualitative perspective has the power to show us how programming works. Intentionally designed and executed qualitative research enriches our understanding of what goes on in the black box between pre and post tests and provides us with the stories to tell and prescriptions for action.

Blooms Taxonomy: A Promising Analytic Framework Cognitive Skills (from less to more complex) Knowledge Comprehension Application Analysis Synthesis Evaluation

Blooms (cont.) Affective Skills (from less to more complex) Receiving phenomena Responding to phenomena Valuing Organization Internalizing values