1. Comparing the Effects of Two Versions of Professional Development on Science Curriculum Implementation and Scaling-Up Session 43.030 April 13, 2005 American Educational Research Association Annual Meeting Montreal, Canada

2. Introduction and Overview Paul R. Brandon Curriculum Research & Development Group University of Hawai‘i at Mānoa

3. 3 Project Purpose A randomized study of the effects of variations in professional development (PD) on –program implementation –student achievement –scale-up First phase (ending 2/28/06) to prepare for five-year second phase (if funded)

4. 4 FAST The project compares two versions of teachers’ PD for Foundational Approaches in Science Teaching (FAST). –An award-winning middle-school inquiry- based science program –Shown to positively affect achievement and other student outcomes

5. 5 Phase-I Project Tasks Prepare the 5-day PD institute with electronic resource and on-line course Prepare instruments for the second phase: –Teacher questionnaire –Teacher log –Observation procedures –Student assessments

6. 6 Focus on Span of PD Traditionally, FAST PD is delivered in a 10-day institute. We will compare it with an alternative 5- day institute followed by an online course, with an electronic multimedia resource.

7. 7 Phase-II Hypothesis The 5-day inquiry-based science training institute, followed by an on-line university-credit course with computer-based multimedia, will have more favorable outcomes than a 10-day institute without the follow-up university course.

8. 8 Examining the effects of PD on: levels of classroom implementation intensiveness and extensiveness of long- term use of inquiry-based science (i.e., scale-up) levels of student achievement

9. 9 Experimental Design Randomized cluster sample, with schools (N = 80) as clusters One teacher per school Even if attrition is 25%, statistical power will be.80.

10. 10 Merit of the Study Addresses at least five deficits in the literature: –Few studies on effects of PD time span –Few studies about using technology in PD –Few randomized studies –Few studies of the effects of PD on student learning –Few, if any studies, of inquiry science PD

11. 11 Order of Presentations Gray, Nguyen, & Speitel: Description of PD Brandon: Log and questionnaire development and questionnaire validation Taum: Observation guide development Ayala: Student assessment development Lawton: Comparison of early effects of the two versions of PD

12. Introduction and Overview Paul R. Brandon Curriculum Research & Development Group University of Hawai‘i at Mānoa

13. Developing and Implementing an Alternative Version of FAST Professional Development Mary E. Gray ThanhTruc T. Nguyen Thomas W. Speitel University of Hawai ‛ i at Mānoa

14. 14 FAST Foundational Approaches in Science Teaching Inquiry-based, middle school science Physical, biological, and earth sciences Exemplary program (USDOE, 2001)

15. 15 FAST Professional Development All FAST teachers must participate in PD –Certification required to purchase science curricula Content Inquiry Standards Assessment Classroom organization and management Safety

16. 16 The Challenge To develop and implement an alternative version of FAST I professional development Year Round Schedules

17. 17 Considerations Current needs of science teachers Future possibilities (emerging technologies) Unique challenges (the dynamic nature of inquiry)

18. 18 Review of Literature

19. 19 Review of Literature Professional Development Strong subject area content (Kennedy, 1999) Focus on higher-order teaching strategies (Porter, Garet, Desimone, Yoon, & Birman, 2000) –Collaborative, Same subject-grade-school, Active learning, Consistent with teacher goals

20. 20 Review of Literature Professional Development Quantity linked to improvements (Radford, 1998; Supovitz, Mayer, & Kahle, 2000; Fishman, Marx, Best, & Revital, 2003) –science content knowledge, process skills, and attitudes Quantity linked to standards based teaching (Supovitz & Turner,2000) Problems with researching inquiry-based teaching and student learning (Fishman et al., 2003)

21. 21 Review of Literature On-line Learning 3 million adult learners on-line (Waitts & Lewis, 2003) Convenience and increased flexibility (Hülsmann,1999) Evidence of implementation of various pedagogical approaches (Schlager & Schank, 1997)

22. 22 Review of Literature On-line Learning Quantifiable learning outcomes were not significantly linked to technology adoption (Jones and Paolucci,1998). Focus on motivation, skills and knowledge, self- directed learning, interactive competence, and technology skills (Kabilan, 2004) Increased efficacy and self-perception in teachers (Huai, Braden, White, & Elliot, 2003)

23. 23 FASTPro

24. 24 How are the traditional PD and FASTPro different? Two-weeks in duration, face-to-face One-week in duration, face-to-face Conduct 43 investigations Extensive modeling and practicing time Conduct 19 investigations Electronic resource WebCT-based learning community

25. Inquiry Teaching and learning science through inquiry is a new experience for many teachers and requires a significant change in attitude and behavior.

26. 26 FASTPro – Addressing Inquiry Strategies (Loucks-Horsley, Hewson, Love, & Stiles,1998) –Examples Immersion in inquiry into science Coaching and mentoring Technology for professional learning

27. 27 FASTPro – Addressing Inquiry Change (Hord, Rutherford, Huling-Austin, & Hall,1987; Guskey, 2000) –takes time and persistence awkwardness and frustration expected –as teacher’s progress through a change process, their needs for support and assistance change Optimal Mix (Guskey, 2000) –A combination of teacher, program and change agent that will help create a positive relationship for PD to be effective

28. 28 FASTPro

29. 29 FASTStart One week face-to-face institute Teachers conduct nearly half of actual student investigations Use inquiry methodology including modeling, discussion, assessment, and practice

30. 30

31. 31 FASTeR Observe interactions of students and teachers View and reflect upon investigations not experienced See detailed step-by- step procedure suggestions Movies and animations Web interface –Quicktime and Flash plugins DVD-ROM medium

32. 32 Examples from FASTeR

33. 33 Examples from FASTeR Slideshow 9, Density and the Cartesian DiverSlideshow 9 Slideshow 22, Identifying Unknown SubstancesSlideshow 22

34. 34

35. 35 FASTForward

36. 36 Formative Findings Suggest that teachers were able to implement successfully and share teaching practice Added utility, focused and meaningful to teachers’ own environment

37. 37 Future considerations Infuse technological aspects into the FASTStart face-to-face experience Address credit equivalencies to enable more robust activity requirements Expand to include FAQ’s, indexing, and rich descriptions Expand to include a Website as well as DVD- ROM Automate some computer assisted instruction and feedback mechanisms

38. 38 Reflections…

39. 39 Reflections… Teamwork and flexibility Coordination of equipment and resources

40. 40 Future Considerations Building capacity Application to other CRDG quality educational programs

41. 41 Mahalo.

42. Developing and Implementing an Alternative Version of FAST Professional Development Mary E. Gray ThanhTruc T. Nguyen Thomas W. Speitel University of Hawai ‛ i at Mānoa

43. Instrument Development for a Study Comparing Two Versions of Inquiry Science Professional Development Paul R. Brandon Alice K. H. Taum University of Hawai‘i at Mānoa

44. 44 Identifying Constructs on teaching science with inquiry methods –Reviewed FAST and other inquiry science documents and worked closely with inquiry science experts. on the context within which inquiry-based science is taught. –Reviewed 55 books and key articles on curriculum indicators and school effectiveness.

45. 45 Developing the Teacher Log Purpose: To identify the extent to which teachers implement the key features of inquiry-based science. Reviewed the recent literature on logs. Kept the instrument short to avoid overburdening teachers (21 items). Is completed immediately after finishing each student science investigation.

46. 46 Focus of the Teacher Log Instrument addresses topics such as: –students’ questioning behaviors. –the teacher’s use of questioning strategies. –the teacher’s circulation about the classroom. –teacher-led discussions about variations in the data.

47. 47 Developing the Teacher Questionnaire Purposes: –To obtain information about implementation that is most appropriately collected once a year. –To collect information on implementation of investigations not covered on logs.

48. 48 Focus of the Questionnaire 150 items address topics such as: –implementation of key features. –the investigations taught during the entire year. –the adequacy of materials, equipment, etc. –teacher demographics. –teacher attitudes toward science. –teacher participation in science activities outside the classroom

49. 49 Types of Validation Studies to Date Two types of questionnaire-data validation studies conducted to date: –analyses addressing the relationship between the responses on the log and the questionnaire (“concurrent” validity) –an analysis examining the theoretical model underlying our study (construct validity)

50. 50 Relationship Between Log and Questionnaire Responses Compared results on total scores for five items that the two instruments had in common Correlation =.54; effect size =.50

51. 51 Examining the Theoretical Model Second set of analyses examined whether the expected relationships were found among some questionnaire variables. We regressed a variable measuring program implementation on five independent variables measuring: –teachers’ ongoing learning about science. –the resources available for teaching science. –the number of science courses taken. –the number of years have taught K–12 science –teachers’ attitudes toward teaching science

52. 52 Validity Argument Teachers’ ongoing learning about science (including learning that occurred in recent PD) should predict implementation more than –science education background. –years of experience teaching science. –attitudes toward teaching science. –classroom resources available.

53. 53 Results of Second Analysis Teacher learning about science outside the classroom: beta =.19(significant at the.05 level) Teacher attitudes toward teaching science: beta =.17 (significant) Classroom and school resources for teaching science: beta =.04(not significant) Number of university science courses taken: beta = -.02 (not significant) Number of years the teacher has taught K–12 science: beta = -.02 (not significant)

54. Instrument Development for a Study Comparing Two Versions of Inquiry Science Professional Development Paul R. Brandon Alice K. H. Taum University of Hawai‘i at Mānoa

55. Coding Teachers in Science Classrooms using the Inquiry Science Observation Guide Alice K. H. Taum Curriculum Research & Development Group University of Hawai‘i at Mānoa

56. 56 Inquiry Science Observation Guide Introduction and Overview –FAST, SCUP Project, Purpose and Development of ISOG General coding guidelines Descriptions for each of the six Activities Definitions of Activity details Code Sheet Recording Sheet Reconciling Recording Sheet

57. 57 Development of the Code Sheet 1.5 year process Collaborative efforts between researchers at the University of Hawai‘i at Mānoa, Stanford University and Sonoma State University; FAST teachers; curriculum developers, and coders 36+ revisions

58. 58 Goal of the Code Sheet Identify strings of activities/teacher behaviors using the details in columns A, B, C and D. –questioning strategies used –level of engagement between teacher and students –creating a profile of teachers implementation of FAST in the classroom through science inquiry

59. 59 Design of the Code Sheet 6 activities. 66 possible activity details to plug into the 6 activities. Multiple variations between an activity and the activity details which describe an activity string of teachers pedagogical practices.

60. 60 1. Teacher directs student(s) _____A_______ to _____B_____ _____C_____.

61. 61 1. Teacher directs student(s) individually (1A1) to discuss (1B2) procedures (1C3). 1. Teacher directs student(s) _____A_____ to _____B______ _____C_____.

62. 62 Inquiry Science Observation Recording Sheet Notes the start time of when the “broader” and “other activities” a teacher is engaged in begins Allows for comments to facilitate the reconciliation process Designed to record multiple activities occurring at the same time

63. 63 Broader and Other Activities –The broader activity captures the larger activity occurring: Through interactive questioning (2A2b), the teacher introduces or provides an overview of (2B1) science investigation (2C4). –The other activities capture the details occurring within the broader activity: Teacher questions students through clarifying (3A1a) questioning and responds to student comment (3B1) by repeating (3C2) the comment and probing further (3C6).

64. 64

65. 65 Reconciling Codes Two coders are paired to compare their independent codings, identifying any differences between them. Differences are then discussed and each coder provides a rationale for his or her selected code. When necessary, a review of the DVD is conducted by the paired coders. Coding differences are discussed until consensus is reached between coders.

66. 66 Inquiry Science Observation Reconciling Code Sheet

67. 67 Challenges greatest challenges: –establishing universal definitions for terms and phrases What constitutes a “procedure”? What is the difference between a science “term” and a science “concept”? –standardizing the recording of activity strings using the Recording Sheet broader verses other activities –focusing on the teacher, rather than on the students

68. 68 What Next? To date we have: – videotaped 15 teachers; –140 videotape cassettes have been digitized to DVDs; –116 have been quality-checked for audio clarity and teacher visibility; and –8 coders are trained and ready to begin coding!

69. Coding Teachers in Science Classrooms using the Inquiry Science Observation Guide Alice K. H. Taum Curriculum Research & Development Group University of Hawai‘i at Mānoa

70. Developing Student Outcome Measures Frameworks Carlos C. Ayala Sonoma State University

71. Scaling Up: Student Measures The Buck Stops Here Carlos Ayala

72. Let’s Talk Charge Develop student assessment instruments in order to tease out differences between professional development models. 2 Session Pre Test 3 Session Post Test Suite

73. 73 Where we expect differences Student’s content knowledge Student’s science inquiry skills Student’s views of the nature of science Student’s efficacy toward science Student’s motivation to learn science

74. 74 Final Assessment Suite Pre-test –30 item multiple-choice/ short answer test (α=.86) –Attitudinal Survey Post-test –30 item multiple-choice/ short answer test –Attitudinal survey –Mānoa River performance assessment

75. 75

76. 76 Knowledge Type Framework Type of KnowledgeDefinition Examples Prompt DeclarativeKnowing thatConcepts & facts “What is percolation?” ProceduralKnowing howActions, steps, “How do measure how much & procedures water soil will hold?” SchematicKnowing whyPrinciples & “How does the water cycle mental models work?”

77. 77

78. 78 Content Validity Reviewed curriculum and created content matrices Curriculum developers parsed content down Linked matrices to materials Created test linking items to matrices Piloted assessments Talk alouds

79. 79 As students become more engaged in the FAST curriculum and the teacher more fully implements the curriculum, students will be more proficient at Science Inquiry. –Design and conduct a scientific investigation –Use appropriate tools to gather, analyze and interpret data –Develop descriptions, explanations predictions and models using evidence Targets based on –Duschl’s Transformations in Three Domains –Pottenger’s Deductive Explanatory Inquiry Student Science Inquiry NRC. (1996). National Science Education Standards. Washington D.C.: National Academy of the Sciences.

80. 80 Transformation 1 Data to Evidence: Deciding if the data are evidence, irrelevant and/or problematic. Transformation 2: Evidence to patterns or models decisions about selecting tools for identifying patterns or models Transformation 3: Patterns and models to explanations. Deciding how the patterns or models lead to explanations. Reformulation: From explanations to new questions: Deciding what next questions to ask and what new data are needed. Student Science Inquiry

81. 81

82. 82 As students become more engaged in the FAST curriculum and the teacher more fully implements the curriculum, students will understand that –anyone can be a scientist. –science knowledge is useful. –science knowledge builds over time. –science is creative. Nature of Science Lederman, Abd-El-Khalick, Bell and Schwartz (2002) Views of Nature of Science Questionnaire; Toward Valid and Meaningful Assessments of Learner’ Conceptions of the Nature of Science.

83. 83 As students become more engaged in the FAST curriculum, the greater control they will feel towards science, science investigations and science knowledge. –I can make accurate measurements during a science investigation. –I can make appropriate predictions about what will happen during a science investigation. Self Efficacy Britner, S. and Pajares F., (2001) Self-Efficacy Beliefs, Motivation, Race and Gender in Middle School Science.

84. 84 As students become more engaged in the FAST curriculum, patterns of motivation may change. Dweck Groups (goals and epistemic beliefs) –Mastery Orientation –Ego Orientation –Helpless Orientation Motivation Haydel, A., & Roser, R. (2001). On the links between students' motivation patterns and their perceptions of, beliefs about and performance on different types of science achievement, Multidimensional Approach to Achievement

85. 85 Next Steps 760 Tests and Surveys completed Collect post test data Run analyses Provide results to group

86. Developing Student Outcome Measures Frameworks Carlos C. Ayala Sonoma State University

87. The Differential Effects of Two Versions of Professional Development on Teachers’ Self-Efficacy to Implement Inquiry-Based Science Brian Lawton University of Hawai‘i at Mānoa

88. 88 Purposes Effects of both versions of PD institutes on teacher self-efficacy to implement program Differences between the two versions Factors associated with self-efficacy change

89. 89 Data Collection Qualitative and quantitative methods used Self-efficacy scale –Developed to facilitate focus group discussion Focus groups –Conducted immediately following the institutes –Provide in-depth information about changes in self-efficacy Five stages of inquiry science 1.Introducing new science investigations 2.Facilitating valid experimental design 3.Facilitating the investigation process 4.Constructing meaning 5.Linking knowledge to new situations Seven teachers from the 10-day and seven teachers from the 5-day participated in the study

90. 90 Data Collection (cont.) Self-Efficacy Scale 1.STAGE: Introducing new science investigations - my ability to introduce students to new science investigations by reviewing and tying in previous work. Now Low High ability ability Before Low High ability 12345678910 123456789

91. 91 Data Analysis Only presenting focus group results Analyzed for changes in self-efficacy –Two categories: Statements implying an increase in self-efficacy Statements implying no change in self-efficacy Categories analyzed to identify the influencing factors The statements and factors were quantified

92. 92 Results, Influencing Factors Overall seven factors were identified as influencing self-efficacy. 1 Factors identified as most influential in increasing self-efficacy 2 Factor identified as most associated with no change in self-efficacy Factors influencing increased self-efficacy Content/pedagogical knowledge 1 Program Structure 1 Professional Collaboration Experience Factors influencing no change in self-efficacy Practice/feedback 2 Student characteristics Experience Time issues

93. 93 Results: Changes in Self-Efficacy Changes in self-efficacy 10-day version 5-day version Statements identified as increases in self-efficacy 118 Statements identified as no change in self-efficacy 69

94. 94 Discussion Group increases in self-efficacy Changes in self-efficacy between the groups Groups perceived gains in content and pedagogical knowledge Importance of 5-day follow-on support

95. 95 Further Study A study comparing the groups after the 5- day group has received all their training.

96. 96 Thanks!

97. The Differential Effects of Two Versions of Professional Development on Teachers’ Self-Efficacy to Implement Inquiry-Based Science Brian Lawton University of Hawai‘i at Mānoa

98. Comparing the Effects of Two Versions of Professional Development on Science Curriculum Implementation and Scaling-Up Session 43.030 April 13, 2005 American Educational Research Association Annual Meeting Montreal, Canada

99. 99 Questions and Comments Paul Brandon - Overview Mary Gray and Truc Nguyen - Description of PD Paul Brandon - Log and questionnaire development and questionnaire validation. Alice Taum –Observation guide development Carlos Ayala - Student assessment development Brian Lawton - Comparison of early effects of the two versions of PD Paper, Presentation and Contact Info available at: http://hisii.hawaii.edu/SCUP/research/