1. American Educational Research Association Annual Meeting AERA San Diego, CA - April 14, 2006 Denise Huang Theory and Methods in OST Research

2. 2 / 28 ExperimentalQualitative Evaluation PurposeNew knowledge, TruthTo uncover and understand what lies behind any phenomenon Mission achievement, product delivery OutcomeGeneralizable conclusionsEmergence of theorySpecific decisions ValueExplanatory and predictive power Rich descriptive data Insiders’ point of view Determining worth and social utility ImpetusCuriosity and ignoranceUnderstandingNeeds and goals Conceptual BasisCause and effect relationshipsExperiences as a whole, in context, interactive Means ends processes Key eventHypothesis testingObservations of multiple, shifting axioms Assessing attainment of an objective Classic paradigmsExperimental method Correlational method Grounded theory approachSystem approach Objective approach DisciplineControl & manipulation of variance IterationsProgram planning and management CriteriaInternal and external validityFit, understanding, generality, and control Isomorphism and credibility Functional typePure and applied True experimental Quasi-experimental Grounded theory Ethnography, Phenomenon approach Case studies, etc. Formative – summative Process – product

3. 3 / 28 What do all these mean?  The gold standard for research design and analysis involve experimental randomization.  Satisfactory research findings required generalization and replication.  The afterschool population has distinct characteristics Ethical issues Self-selective group Difficulty in obtaining comparison group (consent forms) Accuracy of data and availability of longitudinal records High transience  Qualitative strategies are vital tools in finding out the intricate rich data about a program and help to explain, elaborate, and triangulate quantitative findings…  Matching study questions with design and appropriate methodology

4. 4 / 28 Most afterschool studies are evaluation research The systematic application of social research procedures for assessing the conceptualization, design, implementation, and utility of social intervention programs (Strauss & Corbin, 1990). Focus on finding and explaining educational /social effects or outcomes from the intervention, and Devising instructional strategies that will improve the outcomes.  Formative and summative evaluations  Process and impact evaluations

5. 5 / 28 Different types of evaluation research Process Evaluations: Formative studies Evaluations that assesses the conduct of the program during the initial design and testing stages with the intent to improve the program Program monitoring Systematic examination of program coverage and delivery- (target population, fidelity, efficiency) Identifying successful implementation strategies for program diffusion

6. 6 / 28 Impact Evaluations Assesses the changes in the well-being of individuals that can be attributed to a particular intervention, such as a project, program or policy (Strauss & Corbin, 1990). Summative evaluation Summative evaluation provides information on the product's efficacy ( it's ability to do what it was designed to do) By looking at the intervention group, the evaluator can examine the learning materials and learning process together with the outcomes- - hence the name Summative Evaluation. Impact Evaluation Impact evaluation involves constructing a counterfactual Random selection and isolation from interventions are seldom practicable and sometimes ethically difficult to defend. Quasi-experimental method is often used.

7. 7 / 28 Which theoretical perspective informed your work, and why did you select this theory? Goal setting theory Social cognitive theory The contextual understanding of the social ecological model

8. 8 / 28 The role that theory play in research Theory is an invention aimed at organizing and explaining specific aspects of the environment (Pedhazur & Schmelkin, 1991) Theory provides the researcher with a “selective point of view-an orientation Helps to determine what variable is relevant and which are not relevant  Being a way of seeing, a theory is also a way of not seeing.

9. 9 / 28 How did this theory shape your research methods? Mixed methods Process/outcome SEM HLM

10. 10 / 28 Examples: Formative, Process, & Summative The ASL/KidzLit Formative Study ASL is a reading and writing project that was based on the theoretical foundation of the literature-based instructions (social cognitive theory) How do the LA’s BEST teachers evaluate the ASL project in terms of training, materials, and support? To what extent have staff capabilities been enhanced after the ASL training? If the staff members teach during the regular school day and in the after-school program, to what extent have they carried the ASL strategies into their regular school day teaching? What changes in student attitudes might be associated with ASL? Are there any preliminary indicators that participation in ASL is affecting students?

11. 11 / 28 Teacher survey I-formative evaluation For the effectiveness of the ASL training, the evaluation based on ASL’s four core components: Care Talk Read Connect To evaluate the ASL project as a whole, teachers’ perspectives were examined: ASL materials ASL training Support for the ASL project For an affective/motivational outcome the following indicators were examined: Students’ enjoyment Teacher attitude

12. 12 / 28 The student survey Based on the social cognitive theory that learning is a social phenomenon and social collaboration is essential in the classroom climate, we investigate on students’: Enjoyment of their reading class; Perception of the social support in the classroom; Perception of their relation with their teachers. Based on the motivational theory, we look at the students’: Attitudes towards reading and writing; and Reading and writing self-efficacy. And students’ perception of the classroom climate: The autonomy orientation of their classroom climate; and The emphasis on effort allocation.

13. 13 / 28 Qualitative study Exploring the intellectual, social, and organizational capitals at LA’s BEST Relationships Expectations & Social Norms Trust Staff & Student Relationships Student Engagement Organizational Capital Social Capital Intellectual Capital Internal NetworksExternal Networks Innovation and Staff Development Competence Attitudes Problem-solving & Agility

14. 14 / 28 Quantitative outcome evaluations Keeping kids in school study 4 cohorts of students (all LA’s BEST participants and a stratified random sample of non-participants) 3 participation level ( 1,2,3+ years) 6-9 th grade in 1998-1999 followed through to 2002-2003 Chi-square analysis Cox survival analysis

15. 15 / 28 Comparison of dropout rates for LA’s BEST vs. LAUSD non-participants (3+ Years).

16. 16 / 28 The Afterschool Hours-SEM and HLM Examining the Relationship between Afterschool Staff-Based Social Capital and Student Engagement in LA’s BEST What are LA’s BEST staff perceptions of collective staff efficacy, teamwork, and communication, and the quality of their relationships with students? What are student perceptions of their relationships with LA’s BEST staff? To what extent do they value education and have high aspirations for their futures? What are their reported levels of student engagement in LA’s BEST and the day school? How are staff-student relationships, teamwork and communication, and collective staff efficacy at the site level, related to student perceptions of their relationships with staff? What is the association between student perceptions of their relationship with staff, their value of education, future aspirations, and engagement in the afterschool program and day school?

17. 17 / 28 Hierarchical Linear Modeling Examine the relationship between the social capital predictors measured by the staff surveys (i.e., staff-student relationships, collective staff efficacy, and communication and teamwork) and student perceptions of social capital (i.e. staff-student relationship) as measured by the student surveys. The demographic variables of gender, grade level, and languages spoken were also included in each model to control for individual student differences. A total of 2,270 students and 395 staff from 50 school sites were included in the HLM analysis.

18. 18 / 28 Structural Equation Modeling

19. 19 / 28 DOJ study-using propensity matching & cost-benefit analysis  Quasi-experimental design  Longitudinal sampling of academic and juvenile crime data  Advanced multilevel propensity scores methods to establish study samples  Hierarchical growth modeling and survival analysis (multilevel discrete-time hazard)  Students were followed from 1994-1995-2002-2003 school years  Cost-benefit analysis

20. 20 / 28 How did your method(s) contribute to our knowledge of OST, and what were the limitations of the method? Better comparison sample—propensity matching The importance of “dosage” Importance of goal setting Importance of employing a continuous improvement model

21. 21 / 28 Why results are inconsistent? Linking results to program content __Design___________________________________________ Linking results to implementation procedures(program fidelity) __Treatment________________________________________ Linking results to program attendance __Dosage___________________________________________

22. 22 / 28 What does that theory mean for what you would do with children and youth during the out-of-school time and inform program design? Importance of having A theory of change A logic model A continuous improvement strategy

23. 23 / 28 Why create a logic model? They are powerful tools for designing, planning, implementing, and evaluating OST programs They set up a plan of how a program is expected to work Provide a map of how to achieve goals (cause and effect-program process and outcomes) Stimulate clear thinking, preparation, and organization

24. 24 / 28 Example: Theoretical-based logic model designed for the Afterschool Partnership study Program Support Management Staff /Resources Periodical Evaluation for Continuous Improvement Periodic Assessment of Student Performance Linkage to School Day Research Based Practice & Strategies Research Based Practice & Strategies Content Practice Quality After School Center Quality Content Aligned to Standards Content Aligned to Standards Set Goals Align activities to goals Motivation Engagement Opportunities to Practice Process Structure Content Process

25. 25 / 28 Example: Outcome based logic model Linkage with School Content Academic Design Content Academic Design Process Afterschool Program Implementation Process Afterschool Program Implementation Attitudes & Skills Engagement Linkage with Standard Research-based Internal Program Structure External Communication & Support Student/Adult Interaction Parent Satisfaction Structure Program Structure Structure Program Structure Indicators Design & Process Immediate Outcome Expected Outcome Long-term Outcome Professional Development Achievement

26. 26 / 28 Using data for continuous improvement Plan of Action Periodic Assessment Analysis of Results Specification of Monitoring Continuous Adjustment Data Based decision Making Process Theory and Research- based Programming Student Engagement Student Retention Student Improvement

27. 27 / 28 Looking Forward Gap in knowledge as we are working to develop the 21 st century skills: Technology divides The contextual environmental Social capital and human networking system

28. dhuang@cse.ucla.edu