1. RTI Professional Learning Webinar Series: Using Research to Select and Design Effective Interventions – Session 3 October 1, 2009 Our session will start momentarily. While you are waiting, please do the following: Enter/edit your profile information by going to: Tools - Preferences - My Profile… Fill out the info on the “identity” tab and click “OK” To view the profile of another use, hover your mouse over his or her name in the participants window Configure your microphone and speakers by going to: Tools – audio – audio setup wizard Confirm your connection speed by going to: Tools – preferences – connection speed

2. RTI Professional Learning Webinar Series: Using Research to Select and Design Effective Interventions. 2 Session #3 10/1/09 3-4:30 pm Education Research & Determining “What Works” Session #3 10/1/09 3-4:30 pm Education Research & Determining “What Works”

3. Counting ALL Participants After you enter your profile information to join the webinar as usual: 1.Go to the Chat Window 2.Type your district name and school name or organization name or “parent” 3.Type the names of every person in attendance with you. 4.Send it to “This Room” 3

4. Who are we?  Teachers  Administrators  Counselors or Social Workers  County Office or DOE  Parents  Pk-5, 6-8, 9-12, post-secondary  Statewide Organizations  Other? 4

5. Welcome and Introductions  John Wight, GaDOE  Dr. Kimberly Anderson, REL-SE at SERVE Center at UNCG 5

6. 6 REGIONAL EDUCATION LABORATORY- SOUTHEAST (REL-SE) Executive Director Dr. Ludwig van Broekhuizen Toll Free: 800-755-3277 www.serve.orgwww.serve.org Operated by SERVE Center at UNCG

7. 7 Serving Alabama, Florida, Georgia, Mississippi, North Carolina, and South Carolina Georgia liaison: Dr. Kim Anderson kanderson@serve.org 404-657-6174

8. 8 The Regional Educational Laboratory (REL) System

9. REL-SE MISSION To serve the educational needs of the Southeast, using applied research, development, dissemination, and training and technical assistance, to bring the latest and best research and proven practices into school improvement efforts. 9

10. REL-SE Services 1.Outreach and dissemination of research, evaluation, and policy information 2.Technical assistance to SEAs and LEAs 3.“Issues & Answers” publications 4.Randomized Controlled Trial experimental studies on interventions of relevance to our region AMSTI/Alabama and K-PAVE/Mississippi 10

11. 11 WEBINAR SERIES OVERVIEW SESSION 1 SESSION 2 SESSION 3 SESSION 4 Series Introduction RTI Overview INTRODUCTION Understand Scientifically Based Research and the Evidence- Based Decision- Making Cycle SBR & EBDM Understand methodologies used in ed. research, and their strengths and weaknesses for determining “what works” TYPES OF RESEARCH METHODS Understand how to assess the rigor and findings of studies Practice using the Critical Reading Protocol CRITICAL READING PROTOCOL FOR STUDIES ABOUT INTERVENTIONS

12. IES Practice Guide RTI Math IES Practice Guide RTI Reading STATE OF RESEARCH ON RTI Determine the desired outcomes, program features, implementatio n issues, and extent of evidence on an intervention. INTEVENTION REVIEW PROTOCOL Learn what Fidelity of Implemen tation is and how to monitor it effectively. FIDELITY OF RTI IMPLEMENTATION Learn from colleagues in GA how they put the concepts and tools from this series into action. PROMISING PROGRESS: LOOKING FORWARD Webinar Series Overview sessions 5 & 6 session 7 session 8 session 9 12

13. TODAY’S OBJECTIVES 1. Reflect on sessions 1 and 2 2. Understand the different types of research methods used in education studies 3. Understand their strengths and weaknesses for determining “what works” 4. Discuss why it is important to know this as we consider purchasing/creating interventions 5. Preview session #4 13 Types of Research Methods hand-out posted on GaDOE website NCTM/Gersten math study posted on GaDOE website

14. 14 Here we go… Objective #1 Reflect on sessions 1 and 2

15. Reflect on Sessions 1 and 2 Session 1 Series Overview and RTI introduction Session 2 SBR and EBDM 1. What info. from the sessions stood out for you? 2. What is the relevance of these for your RTI practice? 3. What opportunities and challenges have you encountered/might you encounter in putting this information to use on a regular basis? 5 min. discussion Facilitators lead their groups Individuals utilize chat box 15

16. 16 Objectives #2 and #3 Know the types of research methods used in education studies, and their strengths and weaknesses for determining “what works”

17. The EBDM Cycle 17

18. 18 The Main Types of Research: Overview 1.Descriptive 2.Correlational/Regression 3.Quasi-experimental 4.Experimental 5.Meta-analysis Types of Research Methods hand-out posted on GaDOE website

19. Preparatory Pondering: 1 min.  What types of studies have I read, heard about, and/or utilized recently?  What kinds of information did these studies provide—and what were they lacking?  Education studies or research studies from other fields. 19

20. Descriptive Studies  Very useful for illuminating what is “going on”  Case Studies (e.g. turnaround schools research)  Surveys (e.g. community surveys, polling)  Analyses of institutional records (e.g. desk audits)  Site visits/observations (e.g. GAPSS Analysis)  Trends in achievement data (e.g. AYP data)  A limitation is, cannot determine causal relationships. 20

21. Our Familiarity with Descriptive Studies 1.What experiences have we had with descriptive studies recently? 2.How have these informed our work? 3.What new descriptive studies could help us with RTI decision making, and why? 5 min. Chat Facilitators lead their groups Individuals utilize chat box 21

22. Correlation & Regression Studies  A correlation study can tell the degree to which two or more variables are related.  E.g., “Student SES is related to achievement.”  Regression is just an extension of correlation: You can use knowledge of one variable to predict the value of another.  E.g. Predict a student’s propensity to drop out by the presence of certain “risk factors” like retention/overage for grade, attendance, parenthood…  Neither can determine causal relationships. 22

23. Our Familiarity with Correlation and Regression Studies 1.What experiences have we had with correlation/regression studies? 2.How have these informed our work? 3.What new correlation/regression studies could help us with RTI decision making, and why? 5 min. Chat Facilitators lead their groups Individuals utilize chat box 23

24. Quasi-Experimental Studies  Can study groups already using/not using an intervention (“Experimental vs. Comparison”)  No random assignment; matching of groups  Gets closer to identifying causality Pre Treatment Post Experimental O X O Comparison O O  A challenge may be selection bi as 24 Appalachian Regional Comprehensive Center (ARCC) (2006). SBR vs. WAG. ARCC Staff Retreat Presentation. http://www.arcc.edvantia.org

25. Our Familiarity with Quasi- Experimental Studies 1.What experiences have we had with quasi- experimental studies recently? 2.How have these informed our work? 3.What new quasi-experimental studies could help us with RTI decision making, and why? 5 min. Chat Facilitators lead their groups Individuals utilize chat box 25

26. Experimental Studies  Also called randomized control trials (“RCTs”)  The “gold standard”  Gets at causality  Does receiving one treatment (“experimental”) lead to better outcomes than another (“control”)?  Uses random assignment to ensure that groups are equivalent at pretest Pre Treatment Post Experimental R O X O Control R O O R= Random assignment 26

27. Experimental Studies Examples 1.Tennessee STAR study 2.REL-SE AMSTI study 3.REL-SE K-PAVE vocab. study A limitation is, RCTs can be very expensive and time-consuming; an RCT design alone does not illuminate how the results were achieved. 27 Appalachian Regional Comprehensive Center (ARCC) (2006). SBR vs. WAG. ARCC Staff Retreat Presentation. http://www.arcc.edvantia.org

28. Our Familiarity with Experimental Studies 1.What experiences have we had with RCT studies recently? 2.How have these informed our work? 3.What new RCT studies could help us with RTI decision making, and why? 5 min. Chat Facilitators lead their groups Individuals utilize chat box 28

29. Meta-analysis  Impacts/effects are often inconsistent across studies.  As studies accumulate on an intervention/strategy, a need arises to summarize results across them.  E.g.: NCTM/Gersten math meta-analysis  Be aware of how authors determined which studies were in/out of their meta-analysis. 29 NCTM/Gersten math study posted on GaDOE website

30. Our Familiarity with Meta-Analysis Studies 1.What experiences have we had with meta- analysis studies recently? 2.How have these informed our work? 3.What new meta-analyses could help us with RTI decision making, and why? 5 min. Chat Facilitators lead their groups Individuals utilize chat box

31. Questions ?! Comments ?. 31

32. Let’s practice  I’ll show basic information on 5 education research studies.  Using your Types of Research Methods hand- out and the slides we just went through, identify which research method the studies employed.  Justify your answer. 32

33. ID This Research Design #1  Citation: Nussbaum, E.M. (2005). The effect of goal instructions and need for cognition on interactive argumentation. Contemporary Educational Psychology, 30, 286-313.  Question: What type of goal instruction (persuasive focused or reason focused) is most effective in helping students produce elaborate and multisided arguments?  Methods: Students participating in the study were randomly assigned to 3 conditions: persuasive focused, reason focused, and control.  Findings: The persuasive and reason-generating goal instruction conditions resulted in students generating more elaborate arguments than students generated in the control. 33

34. What type of research design do you think the researchers used? Why?  Descriptive  Correlational  Quasi-Experimental  Experimental  Meta-Analysis 34

35. ID This Research Design #2 Citation: Shin, N., Jonassen, D.H., & McGee, S. (2003). Predictors of well-structured and ill-structured problem solving in an astronomy lab. Journal of Research in Science Education, 40, 6-33. Question: What student factors are related to high scores on a problem-solving task? Findings: Two student variables were found to have significant positive correlations with their scores on a problem-solving task: their ability to develop arguments and their level of foundational science knowledge. 35

36. What type of research design do you think the researchers used? Why?  Descriptive  Correlational  Quasi-Experimental  Experimental  Meta-Analysis 36

37. ID This Research Design #3 Citation: Lee, O. & Anderson, C.W. (1993). Task engagement and conceptual change in middle school science classrooms. American Educational Research Journal, 30, 585-610. Research Question: What patterns of task engagement emerge as students work on science tasks in class? Findings: Four major patterns of task engagement emerged among the 12 students in the study: intrinsic motivation to learn science, motivated to learn science, task avoidance, and active task resistance. The authors concluded that even with high-quality, constructivist curriculum materials, it is important to understand why some students might be resistant. 37

38. What type of research design do you think the researchers used? Why?  Descriptive  Correlational  Quasi-Experimental  Experimental  Meta-Analysis 38

39. ID This Research Design #4  Citation: Bennett, J., Lubben, F., & Hogarth, S. (2006). Bringing science to life: A synthesis of the research evidence on the effect of context-based and STS approaches to science teaching. Science Education, 10, 347-370.  Research Question: What is the average effect across studies examining the impact of context-based and STS approaches to science teaching?  Findings: The 17 studies suggested that context-based and STS approaches to science teaching improved attitudes towards science as compared to control groups. 39

40. What type of research design do you think the researchers used? Why?  Descriptive  Correlational  Quasi-Experimental  Experimental  Meta-Analysis 40

41. ID This Research Design #5  Citation: Rainey, L., Harman, P., & McColskey, W. (1996). A report of the integrated science student impact study at the University of Alabama. Greensboro, NC: SERVE Center  Research Question: Do students in the Integrated Science Program (“ISP”) perform significantly better on performance assessments as compared to students not enrolled in the ISP?  Methods: Students in the ISP and not in ISP were matched based on race, gender, and academic ability.  Findings: In a test of science process skills, students enrolled in the ISP performed significantly better than students not in the ISP. 41

42. What type of research design do you think the researchers used? Why?  Descriptive  Correlational  Quasi-Experimental  Experimental  Meta-Analysis 42

43. 43 Objective #4 Discuss why it is important to know this as we consider purchasing/creating interventions

44. Reflection 1. How might this knowledge about the types of research methods (and their strengths and weaknesses) inform our decisions about intervention purchase/design? 2.What are opportunities and challenges for using this knowledge on a regular basis? 3. Other reflections? 44 3 min. Chat Facilitators lead their groups Individuals utilize chat box

45. Objective # 5 Preview session #4 Putting Research to Daily Use: Critiquing Research Studies October 15 3 - 4:30pm 45

46. 46 NEXT SESSION OVERVIEW SESSION 1 SESSION 2 SESSION 3 SESSION 4 Series Introduction RTI Overview INTRODUCTION Understand Scientifically Based Research and the Evidence- Based Decision- Making Cycle SBR & EBDM Understand methodologies used in ed. research, and their strengths and weaknesses for determining “what works” TYPES OF RESEARCH METHODS Understand how to assess the rigor and findings of studies Practice using the Critical Reading Protocol CRITICAL READING PROTOCOL FOR STUDIES ABOUT INTERVENTIONS

47. Logistics for accessing session #4 materials and webinar  John Wight 47

48. Thank you for being here today! Dr. Kim Anderson – kanderson@serve.org – 404-657-6174 The SERVE Center/REL-SE – 800-755-3277 – www.serve.org www.serve.org John Wight, GaDOE  jwight@doe.k12.ga.us jwight@doe.k12.ga.us  404-656-0478 48