Assessing multiple research-based transformations in second semester physics. Steven J. Pollock University of Colorado at Boulder ( per.colorado.edu) We examine the impact of several proven reforms 1,2 in a large course, including Peer Instruction 3 and Tutorials 4 Goals:  Study impacts of multiple layered coordinated activities  Investigate necessary features of individual reforms and the dynamics among them.  Improve/examine student learning & expectations/beliefs.  Increased engagement with physics and teaching of physics Guiding Research Questions:  Which elements most closely associate with learning gains?  What traditionally not-studied factors shape students’ conceptual development, attitudes, beliefs, and success?  What are the impacts of/on undergrad Learning Assistants? Phys students 3 lectures/wk (No lab) LON- CAPA 5 online personalized homework Text 6 HRW Tutorials 50 min/wk, 30 students, 1 grad TA + 1 undergrad Learning Assistant (LA) Weekly prep sessions + LA seminar Interactive Lectures Peer Instruction with H-ITT personal response system Introduction TOPIC:PREPOST (CLICK) POST (EXAM) Flux through closed surface (0 charge)15%75%90% Capacitance: dependence of voltage difference on plate separation 35NA80 Circuits: compare bulb brightness (current model sufficient) * Circuits: compare bulb brightness (current and voltage models needed) 5 BEMA: BEMA: 65 RC circuits: short time behaviour40, BEMA=459085, BEMA=85 Lenz’ law (qualitative)post lect: 55pre tut: 8085  Data are for selected concepts from tutorials. (Results as %, rounded to nearest 5%)  "Post Click" are multiple choice questions answered in class via clickers  Clicker q's encourage peer discussion, but exams are individual.  “LA” = Undergrad Learning Assistant. “TA” = grad teaching assistant  Data from other institutions shown for comparison at bottom 8  44% average gain. Note: BEMA is hard - normalizing to physics grad TA pre-score would yield a normalized class gain of  60%.) LEARNING GAINS: Pre/post (BEMA) 8 LEARNING GAINS: Tutorials End Notes Conclusions 1.McDermott, L., Redish, E., American Journal of Physics, 67(9), , 1999, Redish, E., Teaching Physics with the Physics Suite, Wiley Hake, R., Am. J. Phys 66, 64-74, (‘98) Hake R., Cons Ecology 5(2):28, (‘02) 3.Mazur, E., Peer Instruction, Prentice Hall McDermott, Shaffer, Tutorials in Introductory Physics, Prentice Hall CAPA homework system: see 6.Halliday, Resnick, Walker, Fundamentals of Physics Wiley Adams et al., "Design and Validation of the Colorado Learning About Science Survey", Perkins et al., "Correlating student attitudes with student learning using the CLASS", both in review for Proceedings of the PERC 2004, Sacramento, CA. 8.BEMA: Private communication, R. Chabay and B. Sherwood, see also 9.McDermott, L.C. and P.S. Shaffer (1992). Am. J. Phys. 60, 994 (1992), and part II, p Elby, A. Am. J. Phys 69(7), (2001) 11.Redish, Saul, Steinberg, Am. J. Phys (1998) 12.Pollock, S. “No Single Cause…”, PERC Finkelstein, N., Pollock, S., to appear in PRST (2005)  Learning gains (via multiple measures) are high => this is a successful transformed course. (Average posttest BEMA exceeds current CU phys majors at end of their Jr. level E&M.)  Attitudes/beliefs/expectations 10,11 show modest declines (towards “novice”), but less so for higher learning-gain students. (Student evaluations of course are high.)  No single reform element alone appears to account for the measured successes, but the combination of course elements has a large impact 12,13.  LA program is successful (pre/post content gains, LA satisfaction, and interest in teaching) In 3 semesters, 24 STEM majors have been LA’s. Of those, 8 were LA’s twice or more (continuing with more School of Ed courses) and (at least) 3 are headed to K-12 teaching. Course Structure LEARNING GAINS: GENDER  Female pretest, and gain, on BEMA marginally lower.  Final course grades are statistically identical: females outperform on hw and particip, compensating for slightly lower exam scores. Bins are course grades (which correlate strongly with BEMA gain). CLASS 7 is a Likert-scale survey of attitudes and beliefs about learning.  Note correlation of final grade with CLASS beliefs at the start  More negative shift in beliefs for lower performing students. ATTITUDES AND BELIEFS EXAMINING LOW OUTCOMES (on BEMA post). Can we characterize students who do NOT end up with high BEMA scores? Bottom quartile on BEMA post The rest of the class Ave online participation grade *87% 87% * Pretest BEMA **22%28% Grade in mechanics **76%84% Pretest CLASS (all categories) **63%72% Normalized learning gain on BEMA24%52% Self-reported outside study time/week6.8 hrs5.6 hrs Females31%22% Undeclared Majors11%5% Course Grade ***Solid CSolid B * Participation is not significantly different: these are not "slackers" **Weak at-risk markers. Can not easily/uniquely pre identify this population. ***low gainers not largely D/F students (for whom we rarely get BEMA score)  Statistical uncertainty  1-2% for entries in "tail" column, typically half that for rest.  Bottom quartile => roughly BEMA (post)<50%  CLASS shows “percent favorable” in all categories combined. Special thanks to PhysTEC (APS/AIP/NSF), NSF STEM-TP, NSF CCLI (DUE # ), the CU Physics Dep’t, the entire Physics Education Research at Colorado group, and the Physics Education Group at the Univ. of Washington. Acknowledgements Data Collected Shown here is all new data from Fall ‘04 “Phys 1120” (Physics II) We have collected data for 2 semesters of reformed Phys I (reported elsewhere) Pre and post conceptual exams: BEMA 8 (Brief E&M Assessment) (Fa ‘04): N=331 matched of 462 students Pre/post CLASS (Colorado Learning about Science Survey 7 ) Weekly online Tutorial pretest 4 Posttests given in class (collaborative, with clickers) Posttests on course exams: 75% mult. choice, 25% long answer. Attendance (lecture and Tutorial) Collected but not shown here: Weekly online attitude, and course surveys/long response Homework: online and long answer INDIVIDUAL REFORM ELEMENTS Correlating normalized BEMA gain with:Correlation coefficient Pretest BEMA 0.01 Average course exam score0.62 (p<<.05) Overall course grade0.56 (p<<.05) Grade in Phys (p<<.05) Tutorial homework0.32 (p<<.05) CLASS attitudes/beliefs (all categories, pre)0.29 (p<<.05) Long answer homeworks0.18 (p<<.05) In class clicker score0.15 (p<.05) Conventional homework0.11 (p<.05) Lecture attendance * 0.09 Tutorial Attendance * 0.04 Tutorial attendance is close to 90%, lecture attendance is close to 80% INFLUENCING FACTORS. BEMA gain BEMA PRE Course grade Tut. HW Lec. Att. CAPA HW. Mech. grade BEMA PRE.01 Course grade Tutorial HW Lecture Att CAPA homework Mechanics grade CLASS Matrix of correlation coefficients  Correlation coefficients in red are not statistically significant (p>.05)  CLASS => % favorable on all scored categories on pretest  Normalized BEMA gain does not correlate with pretest (c.f. Hake 2 )  Course grade based heavily (60%) on exams (25% long answer, highly conceptual) covering far more material than BEMA.  No single course reform correlates strongly with BEMA gain References Examining Impacts of Reforms Population Effects Evidence for Conceptual Learning Favorably compares to primary implementation of Tutorials * U. Wash. Tutorials on Circuits: 9 (bulb brightness, current model) => 50% post instruction without Tutorials, 75% with Tutorials. ROLE OF TUTORIALS: Relation to Course Goals  A significant fraction of exams cover material beyond Tutorial coverage.  High correlation of grade with Tutorial hw (graded by TA's) => tight coupling with other hw/exam questions/course goals IMPACT ON LEARNING ASSISTANTS. Normalized BEMA learning gain for LA’s  45%. Average final score (75%) is statistically indistinguishable from Physics Grad students pre-score. Some quotes from first time “initially tentative” Learning Assistants Fa ‘04: LA #1:...in the course of this semester I have advanced my conception of teaching far beyond where it was... My model of how classes should be run has also been radically changed... I hardly realized that there were so many options and methods for running a class... Now that I have a base knowledge of techniques and philosophies I believe that through self-evaluation and awareness my teaching can grow immensely. #2: I gain more confidence and ideas in my teaching styles... The Tutorials have helped me determine how to build the correct rapport with my students... I have enjoyed the experience in physics... More importantly, it has showed me teacher preparation is the foundation for student success in learning. #3: I was expecting to learn more about physics as an LA, but had no idea it would be so much... Personally, being an LA has been one of the most rewarding experiences of my college career.... I have found it to be more challenging and involving than I expected, which makes me like it much more... Where before I had considered teaching as a possibility for my future, I have now determined I definitely want it to be part of my life in some fashion or another