Measuring Teaching Practices Reformed Teaching Observation Protocol1 Describes teaching process on five subscales Lesson Design by instructor Propositional knowledge of instructor Procedural Knowledge (what students do) Student-Student Interactions Student-Teacher Interactions Five items per subscale Each item scored 0 (never occurred) to 4 (very descriptive of the class Maximum Score = 100 (25 x 4) Typical scores range from 20-80 Observer perspective Requires trained observers Quantitative (ordinal) data: 0-4 scale for each of the 25 items 0 (Never occurred) to 4 (Very descriptive of the class) Qualitative data: Field notes 1Sawada et al., 2002

Measuring Teaching Practices Reformed Teaching Observation Protocol1 18) There was a high proportion of student talk and a significant amount of it occurred between and among students (quantity of interactions) No student-student talk Students talk to each other at least once (about lesson content) 1 Student-student talk occurs at least 10% of the time during the course of the class 2 Student-student talk occurs more than 25% of the time during the course of the class 3 In any given moment during the lesson, students are more likely to be talking to each other than the teacher (>50% student to student) 4 Supplemental rubric created by Budd et al. Greater inter-rater reliability per item. Higher scoring classrooms feature more active learning practices. Requires trained observers Quantitative (ordinal) data: 0-4 scale for each of the 25 items 0 (Never occurred) to 4 (Very descriptive of the class) Qualitative data: Field notes 1Sawada et al., 2002; Budd et al., 2013

Variations in Teaching Practices Teaching Geoscience at US Higher Education Institutions On the Cutting Edge has been using the Reformed Teaching Observation Protocol (RTOP) to measure teaching practices of more than 200 instructors in a variety of undergraduate geoscience classes at a range of US colleges and universities. Here are some of the things we have discovered.

Variations in Teaching Practices Teaching Geoscience at US Higher Education Institutions

RTOP Score vs. Sub-Categories ~all Faculty have high Propositional (Content) Knowledge Best correlations with high RTOP score = STI, SSI, Procedural Knowl

0.05 Small Medium Large Minimum 14 13 Maximum 78 89 69 Median 38.5 38 34 First Quartile 26 31 22 Second Quartile Third Quartile 53 47.5 45 Fourth Quartile   Small Medium Large - 0.83 0.05 0.10

Intro Majors 0.39 Intro Majors Label Total Score Minimum 13 16 Maximum 89 78 Median 37 First Quartile 28.25 26 Second Quartile Third Quartile 47 53 Fourth Quartile   Intro Majors 0.39

Associate Baccalaureate Masters Research/Doctoral Minimum 20 17 13 14 Maximum 71 89 78 84 Median 43.5 36 39.5 35 First Quartile 32 31.25 32.25 25 Second Quartile Third Quartile 52 44 49.5 47 Fourth Quartile   2yc bacc master doctoral - 0.85 0.77 0.15 0.98 0.49 0.10

OtCE Participation vs. RTOP scores How did participation in On the Cutting Edge events and use of web resources impact teaching practice? Workshop only 42

Observed Teaching Practices   Most Traditional Lecture n=10 Mean Traditional Lecture Mean Transitional Lecture n=22 Mean Active Learning n=12 Most Active Learning n=11 No/few questions asked by instructor 50% 27% 0% No/few questions from students 60% 36% 9% Students are passive/not asked to do anything 70% 4% No student-student interaction/ conversation 80% 32% Student-student interactions or group work 59% 100% 91% Students read graphs, maps, use data 20% 67% 45% Students answer open-ended questions 17% Instructor assesses students (new or prior knowledge) 10% 18% 33% Lesson adjustments based on student work or prior knowledge 54%

Example Class Breakdowns RTOP = 32, Transitional Minutes 8 55 65 Activity 1 lecture Activity 2 Activity 1: Present LOs, review with 3 conceptest questions Activity 2: Group Venn diagram activity with shout out review RTOP = 51, Student-Centered Minutes 7 20 30 37 45 54 70 75 Activity 1 Lecture 1 Lecture 2 Activity 2 Lecture 3 Activity 3 Lecture 4 Act. 4 Activity 1: Present LOs and Brainstorming Think-Pair-Share Activity 2: Group interpretive drawing activity with class discussion on implications Activity 3: Activity connecting prior and common knowledge to content Activity 4: Reflection, students summarize 3 most important concepts

InTeGrate Materials Development Process Teams of authors spent 1-2 years creating new resources for their courses. They received instruction on pedagogy through workshops and webinars and feedback about their lessons from a team of coaches and assessment specialists. Did it have an impact on their teaching practice? InTeGrate materials are created by teams composed of three authors. We have an approximately equal mix of participants for 2-year colleges, private institutions, and public 4-year schools that include instructors both masters and PhD granting departments. One of the goals of the project is to create materials that can be used in all of these settings. These materials have to meet the five overarching goals that are common to all InTeGrate projects.

Teaching Practice Data (Reformed Teaching Observation Protocol; RTOP) Student-centered (83%) 56.2 41.5 Integrate pilot lessons (taught by the people who created the modules) typically scored in the student-centered range of RTOP. Data represents multiple lessons from five modules taught by 11 instructors. Average RTOP scores from a range of intro courses were recorded by Budd et al at 41.5. Average for InTeGrate lessons was much higher at 56.2. Minerals Climate EJ Soils Living on the Edge Budd et al., 2013 Average RTOP score 41.5 InTeGrate pilot lessons Average RTOP score 56.2

Conclusions: RTOP data identifies 3 classroom types: Traditional Lecture, Active Lecture (Transitional), Active Learning 2. InTeGrate lessons taught by authors correlate with increased student engagement as measured by RTOP 3. Moving from less to more student engagement in class: Increased use of questioning strategies for student input Increased use of students working together during class Increased discussion between faculty and students Increased assessment (and adjustment) to student learning An Active Learning class is very likely to have some traditional lecturing, along with having students participate in a brief group discussion, think-pair-share or class exercise during every class period.

0.01 Adjunct Asst Assoc Full Inst/Lect Minimum 20 17 13 14 Maximum 62 84 89 78 71 Median 31 42 42.5 37 34.5 First Quartile 23.5 32.5 26 24.5 Second Quartile Third Quartile 44 55.5 53.75 50 41.75 Fourth Quartile Adjunct Asst Assoc Full Inst - 0.05 0.04 0.21 0.98 0.87 0.02 0.18 0.01 0.06