1. RTI Professional Learning Webinar Series: Using Research to Select and Design Effective Interventions – Session 4 October 15, 2009 Our session will start momentarily. While you are waiting, please do the following: Enter/edit your profile information by going to: Tools - Preferences - My Profile… Fill out the info on the “identity” tab and click “OK” To view the profile of another use, hover your mouse over his or her name in the participants window Configure your microphone and speakers by going to: Tools – audio – audio setup wizard Confirm your connection speed by going to: Tools – preferences – connection speed

2. RTI Professional Learning Webinar Series: Using Research to Select and Design Effective Interventions. 2 Session #4 10/15/09 3-4:30 pm Critical Review of Research Studies on Interventions Session #4 10/15/09 3-4:30 pm Critical Review of Research Studies on Interventions

3. Counting ALL Participants After you enter your profile information to join the webinar as usual: 1.Go to the Chat Window 2.Type your district name and school name or organization name or “parent” 3.Type the names of every person in attendance with you 4.Send it to “This Room” 3

4. Who are we?  Teachers  Administrators  Counselors or Social Workers  County Office or DOE  Parents  PK-5, 6-8, 9-12, post-secondary  Statewide Organizations  Other? 4

5. Welcome and Introductions  John Wight, GaDOE  Dr. Kimberly Anderson, REL-SE at SERVE Center at UNCG 5

6. 6 REGIONAL EDUCATION LABORATORY- SOUTHEAST (REL-SE) Executive Director Dr. Ludwig van Broekhuizen Toll Free: 800-755-3277 www.serve.orgwww.serve.org Operated by the SERVE Center at UNCG

7. 7 Serving Alabama, Florida, Georgia, Mississippi, North Carolina, and South Carolina Georgia liaison: Dr. Kim Anderson kanderson@serve.org 404-657-6174

8. 8 The Regional Educational Laboratory (REL) System

9. REL-SE MISSION To serve the educational needs of the Southeast, using applied research, development, dissemination, and training and technical assistance, to bring the latest and best research and proven practices into school improvement efforts. 9

10. REL-SE Services 1.Outreach and dissemination of research, evaluation, and policy information 2.Technical assistance to SEAs and LEAs 3.“Issues & Answers” publications 4.Randomized Controlled Trial experimental studies on interventions of relevance to our region AMSTI/Alabama and K-PAVE/Mississippi 10

11. Series Framework: EBDM Cycle 11

12. 12 WEBINAR SERIES OVERVIEW SESSION 1 SESSION 2 SESSION 3 SESSION 4 Series Introduction RTI Overview INTRODUCTION Understand Scientifically Based Research and the Evidence- Based Decision- Making Cycle SBR & EBDM Understand methodologies used in ed. research, and their strengths and weaknesses for determining “what works” TYPES OF RESEARCH METHODS Understand how to assess the rigor and findings of studies Practice using the Critical Reading Protocol CRITICAL READING PROTOCOL FOR STUDIES ABOUT INTERVENTIONS

13. IES Practice Guide RTI Math IES Practice Guide RTI Reading STATE OF RESEARCH ON RTI Determine the desired outcomes, program features, implementation issues, and extent of evidence on an intervention. INTEVENTION REVIEW PROTOCOL Learn what Fidelity of Implemen- tation is and how to monitor it effectively. FIDELITY OF RTI IMPLEMENTATION Learn from colleagues in GA how they put the concepts and tools from this series into action. PROMISING PROGRESS: LOOKING FORWARD Webinar Series Overview sessions 5 & 6 session 7 session 8 session 9 13

14. TODAY’S OBJECTIVES 1. Reach back to Session 3 2. Understand how to assess the rigor and findings of studies on interventions 3. Practice using the Critical Reading Protocol for Studies About Interventions 4. Debrief the Protocol process and discuss the relevance of this type of process for RTI decision making 5. Preview session #4 14 Types of Research Methods handout posted on GaDOE website Carnegie Learning 2001 & Witzel Multisensory Algebra studies Critical Reading Protocol For Studies About Interventions handout

15. Here we go! Objective #1 Reach back to Session #3: Types of Research Methods 15

16. We discussed 5 types of research methods. What do you remember? What thoughts/questions remain?  Descriptive  Correlational  Quasi-Experimental  Experimental  Meta-Analysis 2 min. Chat Facilitators lead their groups Individuals utilize chat box 16

17. Quick Poll 1: Recall Studies using which method can make claims that the intervention studied caused changes in student achievement? A.Descriptive B.Correlational C.Quasi-Experimental D.Experimental E.Meta-Analysis 17

18. Quick Poll 2: Recall Studies using which method can discover relationships between variables (e.g., a particular instructional strategy and academic achievement)? A.Descriptive B.Correlational C.Quasi-Experimental D.Experimental E.Meta-Analysis 18

19. Quick Poll 3: ID This Research Design  Citation: Bennett, J., Lubben, F., & Hogarth, S. (2006). Bringing science to life: A synthesis of the research evidence on the effect of context-based and STS approaches to science teaching. Science Education, 10, 347-370.  Research Question: What is the average effect across studies examining the impact of context-based and STS approaches to science teaching?  Findings: The 17 studies suggested that context-based and STS approaches to science teaching improved attitudes towards science as compared to control groups. 19

20. What type of research design do you think the researchers used? Why? A.Descriptive B.Correlational C.Quasi-Experimental D.Experimental E.Meta-Analysis 20

21. Quick Poll 4: ID This Research Design  Citation: Rainey, L., Harman, P., & McColskey, W. (1996). A report of the integrated science student impact study at the University of Alabama. Greensboro, NC: SERVE Center  Research Question: Do students in the Integrated Science Program (“ISP”) perform significantly better on performance assessments as compared to students not enrolled in the ISP?  Methods: Students in the ISP and not in the ISP were matched based on race, gender, and academic ability.  Findings: In a test of science process skills, students enrolled in the ISP performed significantly better than students not in the ISP. 21

22. What type of research design do you think the researchers used? Why? A.Descriptive B.Correlational C.Quasi-Experimental D.Experimental E.Meta-Analysis 22

23. Objective #2 Put our knowledge of research methods together with other key information to assess the rigor and findings of studies on interventions 23

24. Critical Reading Protocol for Studies About Interventions  Looks at key elements of a research study’s design and results in order to determine its quality and rigor.  This protocol can be used in the process of compiling information about interventions and strategies, and determining “what works.”  This protocol is in the GaDOE RTI manual. 24

25. Critical Reading Protocol for Studies About Interventions See Protocol handout 1.Authorship: Who is the author? Is it the vendor or a third party? Who funded the study? Is there an obvious bias? 2.Sample: What was the sample? How closely do the participants in the study mirror your population? 25

26. Critical Reading Protocol for Studies About Interventions 3.Research design: What kind of design did the researchers use? What kind of questions can this design answer (and not answer)? How were the groups formed? *use your Types of Research Methods handout 26

27. Critical Reading Protocol for Studies About Interventions 4.Results: What kinds of outcomes were reported on? Are the results statistically significant? Practically significant? 5.If you were to implement this intervention what outcomes could you reasonably expect to achieve? 27

28. Objective #3 Practice using the Critical Reading Protocol for Studies About Interventions  Participants read sample program evaluation publications and utilize Protocol to determine the quality and rigor of the study  Then, whole group discussion 28

29. Sample Publication #1: Carnegie Learning 2001 Cognitive Tutor  1 min. introduction to this publication  10 min. read and answer Protocol questions (in small facilitated groups and/or in chat box)  10 min. whole group discussion: What did we learn about this research study and its application for our RTI decision making? 29

30. POLL: Protocol Question #1 Authorship: Who is the author? Is it the vendor or a third party? Who funded the study? Is there an obvious bias? A.The author is/has a relationship to the vendor, and as such may be biased B.The author is a third party, and as such should be neutral C.Cannot determine D.We have a question about this…? 30

31. POLL: Protocol Question #2 Sample: What was the sample? How closely do the participants in the study mirror your population? A.The participants closely mirror our population B.The participants are not similar to our population C.Cannot determine D.We have a question about this…? 31

32. POLL: Protocol Question #3 Research design: What kind of design did the researchers use? What kind of questions can this design answer (and not answer)? How were the groups formed? A.The design was quasi-experimental B.The design was correlational C.The design was experimental D.Cannot determine or we have a question about this…? 32

33. POLL: Protocol Question #4 Results part 1: What kinds of outcomes were reported on? A.Results were Texas End of Course Exam B.Results were local assessments C.Cannot determine D.We have a question about this…? 33

34. POLL: Protocol Question #4 Results part 2: Are the results statistically significant? Practically significant? A.Results were statistically significant B.Results were statistically AND practically significant (in our view, the difference between scores of C.T. students and comparison group students would justify C.T.’s cost, for us) C.Cannot determine D.We have a question about this…? 34

35. POLL: Protocol Question #5 If you were to implement this intervention what outcomes could you reasonably expect to achieve? Why? A.We would expect to achieve similar results B.We would not expect to achieve similar results C.Cannot determine D.We have a question about this…? 35

36. Sample Publication #2: Witzel 2005 Multisensory Algebra  1 min. introduction to this publication  10 min. read and answer Protocol questions (in small facilitated groups and/or in chat box)  10 min. whole group discussion: What did we learn about this research study and its application for our RTI decision making? 36

37. POLL: Protocol Question #1 Authorship: Who is the author? Is it the vendor or a third party? Who funded the study? Is there an obvious bias? A.The author is/has a relationship to the vendor, and as such may be biased B.The author is a third party, and as such should be neutral C.Cannot determine D.We have a question about this…? 37

38. POLL: Protocol Question #2 Sample: What was the sample? How closely do the participants in the study mirror your population? A.The participants closely mirror our population B.The participants are not similar to our population C.Cannot determine D.We have a question about this…? 38

39. POLL: Protocol Question #3 Research design: What kind of design did the researchers use? What kind of questions can this design answer (and not answer)? How were the groups formed? A.The design was quasi-experimental B.The design was experimental C.The design was descriptive D.Cannot determine or we have a question about this…? 39

40. POLL: Protocol Question #4 Results part 1: What kinds of outcomes were reported on? A.Results were SC’s “PACT” test scores B.Results were scores on a local assessment designed specifically for use in the study C.Cannot determine D.We have a question about this…? 40

41. POLL: Protocol Question #4 Results part 2: Are the results statistically significant? Practically significant? A.Results were statistically significant B.Results were statistically AND practically significant (in our view, the amount of difference between scores of M.A. students and comparison would warrant M.A.’s cost, for us) C.Cannot determine D.We have a question about this…? 41

42. POLL: Protocol Question #5 If you were to implement this intervention what outcomes could you reasonably expect to achieve? A.We would expect to achieve similar results B.We would not expect to achieve similar results C.Cannot determine D.We have a question about this…? 42

43. OBJECTIVE #4 Debrief the Critical Reading Protocol process, and Discuss the relevance of this type of process as we consider purchasing/creating interventions 43

44. Debrief & Discuss 1.What do we think about the Protocol? 2.How does this Protocol process compare to how we normally think about the research behind interventions we are considering? 3.What is the value (or not) of this type of process as we consider purchasing/creating interventions? 3 min. Chat Facilitators lead their groups Individuals utilize chat box 44

45. Objective #5 Preview session #4 October 28, 2009 3-4:30pm State of Research on RTI Math: What do we know? What do we not know? Dr. Sybilla Beckmann, UGA 45

46. IES Practice Guide RTI Math IES Practice Guide RTI Reading STATE OF RESEARCH ON RTI Determine the desired outcomes, program features, implementa- tion issues, and extent of evidence on an intervention. INTEVENTION REVIEW PROTOCOL Learn what Fidelity of Implementa -tion is and how to monitor it effectively. FIDELITY OF RTI IMPLEMENTATION Learn from colleagues in GA how they put the concepts and tools from this series into action. PROMISING PROGRESS: LOOKING FORWARD Webinar Series Overview sessions 5 & 6 session 7 session 8 session 9 46

47. http://ies.ed.gov/ncee/wwc/publications/practiceguides/ 47

48. Accessing session #5 materials and webinar  John Wight 48

49. Thank you for being here today! Dr. Kim Anderson – kanderson@serve.org – 404-657-6174 The SERVE Center/REL-SE – 800-755-3277 – www.serve.org www.serve.org John Wight, GaDOE  jwight@doe.k12.ga.us jwight@doe.k12.ga.us  404-656-0478 49