Applying the Scientific Method to Teaching Science Remarkable effort; remarkable success Dr. Douglas Duncan University of Colorado, Boulder We’ve been teaching for a long time….. Today 2000 years ago How effective are we? How effective could we be?

FTEP – Using Clickers in the Classroom Discipline-based Education CU a $15M, 10+ yr. effort! Faculty Collaborators: Michael DubsonPhysics Steve Pollock “ Noah Finkelstein “ Kathy Perkins “ X Carl Wieman “ Valerie Otero Education Mike Klymkowsky Biology xMichelle Smith “ xLeilani Arthurs Geology Stefanie Molburn Sociology Doug Duncan Astronomy Seth Hornstein Astronomy X= Now are faculty at other universities 2 Ph.D. Students: Angel Hoekstra Chandra Turpen Charles Baily Lauren Kost Ben Spike Kara Gray Heidi Iverson May Lee Mike Ross Robert Talbot Colin Wallace Bethany Wilcox Postdocs : Angel Hoekstra Steve Goldhaber Laurel Mayhew Noah Podolefsky Rachel Pepper Wayne Schlingman

Dispel two common misconceptions: 1. Why do science departments have to do education research? Shouldn’t it be done by the School of Education? The Ed School will not teach kinematics, E & M, thermodynamic equilibrium. But we care how well our students understand these topics. So we take the education and success of our students into our own hands.

2. Don’t fall into the logical trap, “This way of teaching worked for me, it should work for my students.” That confuses necessary and sufficient. It washes out students who aren’t like you. Don’t expect them to all be “mini-me’s.”

Just over 50% of university students who start with a physics major switch to something else. Why? A.They were not as good at math B.The were less smart C.The were less motivated D.They were less satisfied 7 universities and colleges 600 hours interviews with roughly 400 science students Surveys of thousands Data on SAT and GPA (grade ave.)

Just over 50% of university students who start with a physics major switch to something else. Why? A.They were not as good at math B.The were less smart C.The were less motivated D.They were less satisfied 7 universities and colleges 600 hours interviews with roughly 400 science students Surveys of thousands Data on SAT and GPA (grade ave.) Mentoring helps!

Watch out! Give similar advice to men and women If this is the only kind of photo of scientists…

What was the #1 complaint of students who switched out of physics? A.Too much work B.Don’t like my classmates C.Poor teaching D.Something else 7 universities and colleges 600 hours interviews with roughly 400 students Surveys of thousands Data on SAT and GPA (grade ave.)

What was the #1 complaint of students who switched out of physics? A.Too much work B.Don’t like my classmates C.Poor teaching D.Something else 7 universities and colleges 600 hours interviews with roughly 400 students Surveys of thousands Data on SAT and GPA (grade ave.)

What was the #1 complaint of students who remained physics majors? A.Too much work B.Don’t like my classmates C.Poor teaching D.Something else 7 universities and colleges 600 hours interviews with roughly 400 students Surveys of thousands Data on SAT and GPA (grade ave.)

What was the #1 complaint of students who remained physics majors? A.Too much work B.Don’t like my classmates C.Poor teaching D.Something else 7 universities and colleges 600 hours interviews with roughly 400 students Surveys of thousands Data on SAT and GPA (grade ave.) Now being redone!

If you are going to believe the results I present, you have to believe we can accurately measure student’s understanding of science topics. What is the best way to measure how deep and thorough students’ understanding is?

Over the last 20 years, the following methodology has been developed to study the success of science teaching and learning… Hundreds of interviews with students Determine right and common wrong answers (misconceptions) to important questions Construct a multiple-choice test where the wrong answers are commonly believed misconceptions Give the test to thousands of students Over the last 20 years, the following methodology has been developed to study the success of science teaching and learning… Hundreds of interviews with students Determine right and common wrong answers (misconceptions) to important questions Construct a multiple-choice test where the wrong answers are commonly believed misconceptions Give the test to thousands of students Force Concept Inventory (FCI) Almost entirely conceptual

Mazur (1997; 2004) (b) …

conventional conceptual 49% 69%

R. Hake, ”…A six-thousand-student survey…” AJP 66, (‘98). = post-pre 100-pre traditional lecture Force Concept Inventory In a traditional lecture class, students master about 25% of the concepts (that they don’t already know).

Traditional Model of Education Instruction via transmission Individual Content

Students brains are not empty. A cautionary tale…. about a violin…. From Carl Weiman’s* “Physics of Everyday Life” class. * Nobel prize winner AND good teacher

(b) Only 10% of students gave the correct answer. Fifteen minutes later in the same lecture!

Teaching by telling is surprisingly ineffective - if you want students to master concepts. Students minds must be active to learn. “Peer discussion” of conceptual questions forces students to talk and reason during class. Clickers hold them accountable. Anything that gets students to think and explain is good!

Consider a tiny acorn, and a giant oak tree. An old, dead, dry log from that tree weighs 10,000x as much as the acorn. Where does MOST of the mass come from? A. Sunlight B. Water C. Dirt D. Minerals in the soil E. The air Old, dead, dry logs

6 CO H 2 O → C 6 H 12 O O H 2 O Carbon dioxide water glucose (sugar) oxygen water Photosynthesis The correct answer is “The Air.” Carbon dioxide from the air!

Notice the excitement! You discover something new to you, …and benefit from peer discussion. A lot of the teaching at CU is now like this.

= post-pre 100-pre red = traditional, blue = interactive engagement Physics learning at the University of Colorado leads the US! The graph shows the fraction of everything taught students learn thoroughly during the semester. Red and blue histogram bars are for 52 classes throughout the US. F01 F99 F04 F05 S05 S04F07 © Copyright Steven Pollock 2007 Traditional lecture (popular professor) Clickers with student-student discussion Clickers + tutorials What difference does interactive engagement make?

This presupposes effective use of peer discussion and clickers. Questions need to be challenging. Student need to actively participate. (not just listen) - otherwise you will not realize the learning gains I see plenty of ineffective clicker use.

It is very important that you learn about traxoline. Traxoline is a new form of zionter. It is monotilled in Ceristanna. The Ceristannians gristerlate large amounts of fevon and then bracter it to quasel traxoline. Traxoline may well be one of our most lukized snezlaus in the future because of our zionter lescelidge. The Monotillation of Traxoline Now I could ask you clicker (or written) questions: 1.What is traxoline? 2.Where is traxoline monotilled? 3.How is traxoline quaselled? 4.Why is it important to know about traxoline?

Bloom’s Taxonomy Don’t get stuck at the lowest level of questioning! (factual recall)

Variation in Types of Questions Trad & PER instructors More experienced instructors vary the complexity of questions.

More experienced users spend more time discussing incorrect responses Most of the time individual answers are solicited before peer discussion.

More experienced clicker users engage more with students 2 New users (yellow) vs. 2 Experienced Instructors

It is enormously valuable to wander around and listen to your students arguments while they debate a provocative question.

Peer instruction reveals your students’ assumptions and arguments, which often are not what you expect.

Fish is Fish…

Peer Instruction and Clicker use means more work for students than just taking notes. It means they have to talk to the person next to them. Who they may think is an idiot. Without explanation students protest. Real student quote: “I expected you to teach me, I didn’t expect to have to learn!” You must explain why they are doing this, or they will protest! -- They should discuss what it means to learn Metacognition

How to get good clicker questions? Effective multiple-choice questions have believable “distracters.” 39 1)Talking with other instructors that have taught the course in the past. 2)Talking with your students one-on-one before class, after class, during office hours. 3)Using student responses to open- ended questions that you include in HW and exams. 4)Asking your students to come up with answers that will be used as the choices. 5)Use researched and documented student misconceptions.

Peer instruction and clickers are relatively easy to implement. Even upper division classes are using them. But remember – it’s not the technology that counts, it’s what you do with it. If you hear someone say, “Do clickers work?” “Do tutorials?” Tell them…

A general principle: The more they think about the technology, the less they think about the science…..(cognitive load)

What the Most Successful technology implementers do: 1.Decide what we want our students to do 2.Find the best technology to encourage that behavior e.g. “encourage discussion in class.” Or: more personal engagement Determine your goal first!

Tutorials in Introductory Physics Reconceptualize Recitation Sections Materials Classroom format / interaction Instructional Role

Proven, Published Curricula D.E. Trowbridge and L. C. McDermott, "Investigation of student understanding of the concept of acceleration in one dimension," Am. J. Phys. 49 (3), 242 (1981). D.E. Trowbridge and L. C. McDermott, "Investigation of student understanding of the concept of velocity in one dimension," Am. J. Phys. 48 (12), 1020 (1980) R.A. Lawson and L.C. McDermott, "Student understanding of the work-energy and impulse- momentum theorems," Am. J. Phys. 55 (9), 811 (1987) L.C. McDermott and P.S. Shaffer, "Research as a guide for curriculum development: An example from introductory electricity, Part I: Investigation of student understanding." Am. J. Phys. 60 (11), 994 (1992); Erratum to Part I, Am. J. Phys. 61 (1), 81 (1993). P.S. Shaffer and L.C. McDermott, "Research as a guide for curriculum development: An example from introductory electricity, Part II: Design of instructional strategies." Am. J. Phys. 60 (11), 1003 (1992) L.C.McDermott, P.S. Shaffer and M. Somers, "Research as a guide for curriculum development: An illustration in the context of the Atwood's machine," Am. J. Phys.62 (1) (1994). More: see D.E. Trowbridge and L. C. McDermott, "Investigation of student understanding of the concept of acceleration in one dimension," Am. J. Phys. 49 (3), 242 (1981). D.E. Trowbridge and L. C. McDermott, "Investigation of student understanding of the concept of velocity in one dimension," Am. J. Phys. 48 (12), 1020 (1980) R.A. Lawson and L.C. McDermott, "Student understanding of the work-energy and impulse- momentum theorems," Am. J. Phys. 55 (9), 811 (1987) L.C. McDermott and P.S. Shaffer, "Research as a guide for curriculum development: An example from introductory electricity, Part I: Investigation of student understanding." Am. J. Phys. 60 (11), 994 (1992); Erratum to Part I, Am. J. Phys. 61 (1), 81 (1993). P.S. Shaffer and L.C. McDermott, "Research as a guide for curriculum development: An example from introductory electricity, Part II: Design of instructional strategies." Am. J. Phys. 60 (11), 1003 (1992) L.C.McDermott, P.S. Shaffer and M. Somers, "Research as a guide for curriculum development: An illustration in the context of the Atwood's machine," Am. J. Phys.62 (1) (1994). More: see

Tutorial vs. Trad'l Recitation You already know what they will find difficult!

Tutorial Materials Hands-on, Inquiry-based, Guided, Research-based

Phys lecture students 3 lectures/wk (No lab) U. Washington Tutorials 50 min/wk, 30 students, 1 grad TA + undergrad Learning Assistant (Weekly prep + LA seminar) Interactive Lectures Peer Instruction, pers. resp. system Text trad or PER based Online HW System CAPA or MP

CU-Boulder LA Model Emulation Auburn University Black Hills State University Boise State Boston University California Polytechnic State California State University - LB Cornell University Florida International University Indiana University (UTA Program) James Madison (Planned 11/12) Louisiana State University Marshall University (Planned 11/12) New Jersey Institute of Technology North Dakota State University Rutgers University Seattle Pacific University South Dakota State Towson University UNC Chapel Hill University of Arizona University of Arkansas University of Maine University of Maryland University of Minnesota - St Paul University of Oklahoma University of Texas - Austin University of Texas - El Paso Utah State University Virginia Tech Western Kentucky University

red = traditional, blue = interactive engagement Physics learning at the University of Colorado leads the US! The graph shows the fraction of everything taught students learn thoroughly during the semester. Red and blue histogram bars are for 52 classes throughout the US. F01 F99 F04 F05 S05 S04F07 © Copyright Steven Pollock 2007 Traditional lecture (popular professor) Clickers with student-student discussion Clickers + tutorials What difference does interactive engagement make?

Colin Wallace. From relativity to discipline-based education research.

The professor is still key!

What student voices reveal about using clickers It is important to use clickers often enough and to include a variety of question types “In this class, no, they do not help us learn class material. It feels like she uses them just for attendance purposes and then doesn’t really fully go over them. It’s mostly just a waste of time… My physics professor used them very well… let us discuss them with our classmates, and then went over the right answer, thoroughly explained [the clicker question], and then told us why the other options were wrong, that really helped.”(journalism student)

Univ. of Colorado Upper-Level Course Transformation Longitudinal After upper div. E&M. (Only students who took intro without Tutorials) Upper division majors’ BEMA scores S. Pollock, 2007 PERC, and Phys. Rev STPER 5 (2009)

Univ. of Colorado Upper-Level Course Transformation Longitudinal BLUE: students who took freshman E&M with Tutorials Upper division majors’ BEMA scores S. Pollock, 2007 PERC, and Phys. Rev STPER 5 (2009)

Univ. of Colorado Upper-Level Course Transformation Longitudinal Yellow: students who had been E&M LAs (3.1 ±.1) (3.0 ±.1) (3.3 ±.1) (3.2) Grade in course S. Pollock, 2007 PERC, and Phys. Rev STPER 5 (2009) Upper division majors’ BEMA scores

Student interview comments… “Clicker points [help] “cushion” the overall grade a little, instead of all points coming from exams.” “A lot of people hate [clickers] because they say it is the only reason that they go [to class,] but that is bullshit. They just want to skip [class] without losing ‘points.’ I like the interaction [clickers incite], it helps me know what I need to study more. Most kids that don’t like them, I feel, don’t have too many ‘solid’ reasons [for] why.” “I think clickers are good as a learning device but at the same time being graded on them sucks. It also takes away from the student experience. Instead of thinking we’re only statistics to the university, we actually are statistics represented in our [histogram] graphs every day.”

I SSUES OF TECHNOLOGY AND “ MULTI - TASKING ” ARE CRITICAL, AND WILL BECOME MORE SO …. PHeTPHeT (phet.colorado.edu)

I think I’m a great multi-tasker. Am I? I bet the professor thinks we’re taking notes…. Technology is in your class more than you think!

I NTERVIEW QUOTES “I would say that I probably send 16 text messages in a class, definitely.” “When you go to class, it’s a time to sit down and text.” Sometimes people are texting all the time and it really gets annoying. “I don’t really have any set rules as to how I conduct myself with my phone [in class]… I find text messaging is not very intrusive. You are competing for the attention of your students. To best benefit learning, instructors should set clear policies – for on-task clicker use and off-task cell phone use, and use regular meta-narrative explaining “ appropriate device behavior ” to students.

Avg. Score no phone – Avg. Score phone users = 4.7 ± 1.4 % Tamir and Mitchell (Harvard) FMRI study: talking about yourself on Facebook activates the brain’s reward system, like gambling does for many people. Your students find this hard to give up. They are “high” on it! 30%70%

CU Professor Diane Sieber’s experiment: Track laptop use; note performance correlations With their permission, share this information with students so they can make their own choices AssessmentTotal students Students with laptops open (consistently) prior to test Test average, entire class Test average, laptop subset Average test score improvement, subset that stopped using open laptops in class Test # (=-12)-- Test # (=-13)+16 (av of 4 students) Test# (=-13)+11 (av of 9 students) Note that this was a pure lecture, non-clicker class.

FTEP – Using Clickers in the Classroom Canadian Study of Laptop use Students were asked to use laptops to take notes on a lecture. ½ the students – randomly assigned – were also asked to look up stuff “when you feel you can spare some time….” What do you think was the effect on grades? 64

Those sitting in view of a laptop did even worse!

…It’s weird sitting in class taking notes and somebody’s on Facebook looking at their pictures from a party they had two nights ago. Sometimes the guys are sitting there watching a snowboard video. I’ve seen people Skyping, I don’t know how they get away with that in class, I don’t know why you’d have just the video feed. It’s just distracting in general if somebody’s playing a game.

N OBEL LAUREATE D ANIEL K AHNEMAN : You can do several things at once, but only if they are easy and undemanding. You have a limited budget of attention – cognitive load – and if you go beyond that you fail. It is a mark of effortful activities that … you cannot do more than one at a time. My speculative warning: if students can multi- task in your class and succeed, in the future your teaching will be replaced by a machine.

Where is the wisdom we have lost in knowledge? Where is the knowledge we have lost in information? T. S. Elliot

Experts don’t just know a lot of stuff…. They know which knowledge to access for a given type of problem How to apply it They self-check (metacognition) They know which knowledge to access for a given type of problem How to apply it They self-check (metacognition)

Metacognition is critically important, yet often overlooked.  Thinking explicitly about reasoning, problem solving, and testing one’s ideas.  Metacognition is essential for effective learning in complex situations  A new wrinkle – deciding if you can multi-task.

Your students don’t believe this. You need to coach them. Metacognition.

V ARIATION A CROSS C OURSES How much does enforcement of a clear technology-use policy account for this variation?

cationIssues/research/papers_talks.h tm