Peering into the black box between pre and post: A theory of change related to interactive/ experiential delivery methods Susan Carter Evaluation Coordinator National Indian Youth Leadership Project with J. Fred Springer, EMT Associates

A presentation for the 6 th Annual Research and Evaluation in Adventure Programming Symposium Gaithersburg, MD March 17-19, 2010

The Multi-Site High Risk Youth Study funded by SAMHSA Fred Springer, EMT, Co-PI 48 site quasi-experimental design 10,500 youth 4 point longitudinal study Common instrument Comprehensive process data Hierarchical statistical design American Evaluation Association Study of the Year award, 2000 Largest direct service ATOD prevention evaluation ever implemented

4 Common Factors Reducing 30-day Drug Use Effect Size Life-skills Focused Connection Building CoherentIntrospective Orientation Intense Contact

HRY Common Factor Findings Promoting introspection / connectedness are critical common factors Out of school more effective Manualized programs less effective Intensity counts Promoting behavioral skills more effective than focus on knowledge and attitude change

Application: Project Venture Project Venture identified as one of eight best programs in HRY study ◦PV included all of the common factors Incorporates connectedness building and introspection Uses interactive experiential program delivery strategy

Essential Elements of Project Venture Population ◦Middle school aged American Indian youth ◦High school aged AI service staff ◦Mix of risk levels (at risk + high risk) Program Components ◦Experiential education ◦Connecting to the natural world ◦Physical and social-emotional challenge ◦Service learning ◦Focus on developing positive relationships ◦Positive Youth Development approach

Project Venture: Overview Experiential, adventure-based positive youth development program for American Indian youth Recognized by NREPP as evidence-based The only American Indian ATOD prevention program currently in NREPP Widely disseminated throughout the US and Canada Twenty year implementation and evaluation history

Project Venture: Strategies Outdoor, adventure activities: ropes course, rappelling, canoeing, backpacking, camping, mountain biking, etc. Service Learning Meaningful projects that are youth centered and designed, address cultural, environmental and other community needs. In a research-supported, year long sequential program.

Project Venture: Evaluation Findings Positive substance abuse prevention findings (prevent, reduce, delay onset) especially related to alcohol abuse Increased internal assets Increased external assets, especially community and peer domains

Purpose of proposed study We propose to advance knowledge concerning how adventure based strategies and activities can be strengthened, promoted, and replicated.

Hypothesis Interactive /experiential activities that clearly incorporate reflective learning skills and promote connectedness to positive people and environments will promote intended behavioral change more effectively than interactive/experiential activities that have less focused emphasis on these interim outcomes.

Method notes Quasi-experiment Baseline, exit, six and twelve month follow up Assess similar participants in Project Venture programs: ◦6 programs using an enhanced version of PV ◦6 “regular” PV programs ◦(perhaps 6 groups with no PV)

Measures Behavioral outcomes (ATOD, etc.) Connectedness (e.g., Michael Karcher’s Hemmingway scales) Reflective learning (perhaps adapted from education literature)

Analysis notes Rigorous development and testing program to develop the measures Rigorous statistical methodology including propensity scores for improving comparison equivalence Hierarchical modeling to account for nesting effects, treatment interactions, treatment settings Strong fidelity/coherence measurements

High Risk Youth Study, SAMHSA mapped onto Project Venture Characteristics of Most Successful Prevention Programs How Project Venture Incorporates these Characteristics through Interactive Experiential Delivery Strategies EngagementCultural content, leadership, values / experiential learning / connecting to the natural world / physical and emotional challenge Intensity150 + hours year (3 + hours per week) 68 sessions / In & after school, weekends, summer Reflective OrientationExperiential learning cycle, debriefing; Outdoor adventure: problem solving; Service learning: student led, problem solving; Camp, treks: journaling, values, norms, natural world, spiritual awareness; Native cultural values: spiritual traditions; In school: Problem solving, skill building, socialization Connectedness BuildingOutdoor adventure: team building, trust building; Service learning: youth led, meaningful, relationship emphasis; service ethic; Camp, treks: build caring, cooperative community; Native cultural values; family events, cultural traditions Life-skills FocusedOutdoor adventure: cooperation, leadership, efficacy ; Service learning: problem solving, leadership, cooperation; Camp, treks: leadership, efficacy; In school: problem solving, skill building Positive EnvironmentCultural content; adult participation, interaction, trust, support; CoherenceClear purpose and process; training; collaboration; implementation project Community SupportGuided by Native Elders; Native cultural values; positive youth development approach; collaboration.

Other study ideas Secondary data analysis of 5 years of pre/post survey data from Project Venture 7 th grade participants (600) and matched comparison group (400) collected between AND Longitudinal follow-up study of PV program participants who were in 7 th grade in 2006 – including original program survey (adapted for older youth cohort) AND life history interviews