Calibrated Peer Review (CPR) TM at Texas A&M University by Dr. Wendy Keeney-Kennicutt Associate Director, First Year Chemistry Program CPR Master Administrator, TAMU CPR Master Administrator,

HISTORY AT TAMU cpr.tamu.edu  CPR TM was introduced to First Year Chemistry in an NSF-sponsored Multi- Initiative Dissemination spring workshop and a CPR summer workshop. We started in the fall.  To avoid FERPA* issues & because of a joint NSF CPR grant with UCLA and our Center for Teaching Excellence, CPR was housed on a secure TAMU server.  I volunteered to be the CPR TM master administrator because of experience. *Family Educational Rights and Privacy Act (1974) – protects the privacy of student education records

USE OF CPR TM AT TAMU USE OF CPR TM AT TAMU  In the last 3 years, the use of CPR TM at TAMU been used by approximately 10,000 undergraduate and graduate students doing 10,000 undergraduate and graduate students doing 320 new assignments in 320 new assignments in 200 courses spread over 200 courses spread over 30 majors in 30 majors in 7 colleges. 7 colleges.

MAJORS USING CPR TM MAJORS USING CPR TM  Accounting  Ag. Economics  Animal Science  Archeology  Biochemistry  Biology  Botany  Bus. Admin.  Chemistry  English  Ed. Psychology  Film  French  Kinesiology  Learn. Comm.  Math  Microbiology  Nutrition  Physics  Poultry Science  Psychology  Secondary Ed.  Vet Integr. Bio. Sci.  Wildlife & Fish. Sci.  Zoology  MISSING:  Engineering Starting F06Starting F06  Geosciences

How Did CPR TM Become So Popular? (1) Creation of “W” Courses  Our Faculty Senate decided that for graduation, there would be a writing-intensive (W) discipline-specific course graduation requirement.  The Core Curriculum Review Committee report entitled "Educational Leadership at the Beginning of the 21st Century" (March 28, 2000, as amended and approved by the Faculty Senate, May 8, 2000), established this requirement. The first writing-intensive course graduation requirement went into effect in the Fall 2004 catalog. The requirement for a second course is scheduled to be in place in the Fall 2006 catalog. "Educational Leadership at the Beginning of the 21st Century" "Educational Leadership at the Beginning of the 21st Century"

(2) CPR was recognized as good pedagogy campus-wide  I co-ran a CPR workshop for TAMU and local community college faculty, supported by TAMU College of Science mini-grant in Summer 2003 and invited everyone.  The Director of the TAMU Writing Center participated, recognized its effective pedagogy and gave CPR her blessing for use in “W” courses.  By Fall 2003, we had CPR TM on our server, TAMU had quit using SSN as SIDs and we were set to go. How Did CPR TM Become So Popular?

(3) Continued Support for our TAMU users: A good website: cpr.tamu.edu with extended FAQs A good website: cpr.tamu.edu with extended FAQs Regular workshops supported by WALS*, Center for Teaching Excellence (CTE), Information Technology Services, The Writing Center, and the Office of the Vice- President for Information Technology Regular workshops supported by WALS*, Center for Teaching Excellence (CTE), Information Technology Services, The Writing Center, and the Office of the Vice- President for Information Technology A local users group which has started to meet regularly A local users group which has started to meet regularly 2 months summer salary for the TAMU CPR master administrator; two other administrators are paid through their position at CTE 2 months summer salary for the TAMU CPR master administrator; two other administrators are paid through their position at CTE Access to the actual program to solve problems Access to the actual program to solve problems *WALS = Writing for Assessment and Learning in the Natural and Mathematical Sciences (the UCLA/TAMU NSF project) How Did CPR TM Become So Popular?

What are the issues?  TAMU has its own CPR webpage, so students occasionally log on to the wrong one.  TAMU is disconnected from the UCLA server, so we don’t have access to the library assignments. This will be changing under the upgraded system.  Note: We have found that there is more student buy- in when instructors write their own assignments: The assignments reflect the instructor’s wording and tone The assignments reflect the instructor’s wording and tone CPR fits into the course flow better CPR fits into the course flow better The instructor is much more familiar with the program after they write assignments. The instructor is much more familiar with the program after they write assignments.

So what happened to CPR in Chemistry?  In Fall 2002 and Spring 2003, we incorporated CPR in all the Chemistry 101/102 classes  ~ 2600 students per semester  Individual instructors (besides myself) had minimal training in CPR and were not involved  Assignments were taken directly from the library 4 assignments, dropping 1 to allow for absences 4 assignments, dropping 1 to allow for absences I and a staff member managed the problems list, all the grades, and student questions I and a staff member managed the problems list, all the grades, and student questions  So what do you think was the outcome? I kept track using SALG.

Student Assessment of Learning Gains

First Year’s Results from My Class This was not good…… Everyone but me stopped doing CPR. I knew that scientific writing/critiquing were invaluable skills, so I kept trying new approaches to improve student attitude.

Overview of Study (manuscript in preparation)  I collected SALG data for 7 semesters from my students (1515 total; 94-98% of classes participated), as I continued to make CPR more palatable Likert scale (1=strongly disagree…….5=strongly agree) Likert scale (1=strongly disagree…….5=strongly agree) I enjoyed doing the CPR assignmentsI enjoyed doing the CPR assignments The CPR assignments helped me learn some chemistryThe CPR assignments helped me learn some chemistry The CPR assignments helped me improve my writing skillsThe CPR assignments helped me improve my writing skills The CPR assignments helped me learn to critique my own writing and that of othersThe CPR assignments helped me learn to critique my own writing and that of others Yes/No + explanation Yes/No + explanation Do you think that future classes should do CPR?Do you think that future classes should do CPR? I had both quantitative and qualitative data for analysis I had both quantitative and qualitative data for analysis

Improvements Made Over Time  Prepared more thorough instructions and a supportive website  Wrote most of my own assignments  Became more proactive at listening to students & adjusting grades when appropriate  Told my students upfront in the syllabus that the class was a writing-intensive class and included my teaching philosophy  Gave a “CPR lab holiday” – since I did 7 labs rather than 10  Invited students to let me review their essays before submission  Increased importance of the text entry from 20% to 30%  Increased CPR’s worth from 3-5% to 12% of class grade  Took classroom time to demonstrate CPR  Used Bloom’s Taxonomy to show importance of critiquing

Why is writing important? Synthesis Analysis Application Use of information to solve problems; transfer of abstract or theoretical ideas to practical situations. Interpretation Identification of connections and relationships Translation Restatement in your own words; paraphrase; summary Recall Verbatim information; memorization with no evidence of understanding Evaluation Combination of information to form a unique product; requires creativity and originality Judgment: the ability to make decisions and support views; requires understanding of values Identification of component parts; determination of arrangement, logic, semantics Bloom’s Taxonomy – categorizing level of abstraction of questions level of abstraction of questions

Improvements Made Over Time  Prepared more thorough instructions and a supportive website  Wrote most of my own assignments  Became more proactive & adjusted grades when appropriate  Told my students upfront in the syllabus that the class was a writing-intensive class and included my teaching philosophy  Gave a “CPR lab holiday” – since my class did 7 labs, not 10  Invited students to let me review their essays before submission  Increased importance of the text entry from 20% to 30%  Increased CPR’s worth from 5% to 12% of class grade  Related that CPR is a grade in its own right & helps poor test-takers  Took classroom time to demonstrate CPR  Used Bloom’s Taxonomy to show importance of critiquing  Emphasized that most students are novice reviewers and that I would gladly look at anyone’s grade

What I saw was a significant increase in student acceptance and understanding of CPR over time. and understanding of CPR over time. QuantitativeResults Quantitative Results

Group 1: Students with negative CPR experience, wanted future classes to do CPR Group 2: Students with negative CPR experience, did not want others to do CPR Group 3: Students with positive CPR experience, wanted others to do CPR Group 4: Students with positive CPR experience, did not want others to do CPR

QualitativeResults Qualitative Results “Do you think that future classes should do CPR? Explain.”  Over 7 semesters, there were 550 totally positive responses 550 totally positive responses 515 totally negative responses 515 totally negative responses 174 mixed responses 174 mixed responses 25 neutral responses 25 neutral responses Total: 1264 responses Total: 1264 responses  The qualitative part of this study gave invaluable insight into student attitude about CPR and how it changed as I made changes in presentation, student support and grade intervention.

QualitativeResults – Negative Comments Qualitative Results – Negative Comments  On writing in a chemistry class: “I have never viewed chemistry as being a subject where you write things;” “We could take English to learn how to write correctly;” “I didn’t understand why writing a paper and grading other students papers had anything to do with chemistry.” “I have never viewed chemistry as being a subject where you write things;” “We could take English to learn how to write correctly;” “I didn’t understand why writing a paper and grading other students papers had anything to do with chemistry.”  On the peer review process: “They ask you to grade the essays, but then your opinion of how that person did would be wrong. I just don’t see how your opinion could be wrong.” “They ask you to grade the essays, but then your opinion of how that person did would be wrong. I just don’t see how your opinion could be wrong.”  Other: Too time consuming, waste of time, not related to the subject; it harmed their grade; was worse than lab; their peers lacked motivation; added to stress Too time consuming, waste of time, not related to the subject; it harmed their grade; was worse than lab; their peers lacked motivation; added to stress

QualitativeResults – Positive Comments Qualitative Results – Positive Comments  On writing in a chemistry class: “Calibrated Peer Reviews forces the student to look into the topic way more than what he or she would do out of a textbook. I know the CPR has tremendously helped me understand each topic better although I didn’t exactly enjoy it.” “Calibrated Peer Reviews forces the student to look into the topic way more than what he or she would do out of a textbook. I know the CPR has tremendously helped me understand each topic better although I didn’t exactly enjoy it.” “The CPR really helped me understand the topics. It reinforced the material by forcing me to teach myself and explain it to others through writing. It was very helpful.” “The CPR really helped me understand the topics. It reinforced the material by forcing me to teach myself and explain it to others through writing. It was very helpful.” “I think the first one is bad because you don’t really know what you’re doing and how to approach the whole thing, but after doing it you realize that you are learning the subject because you had to write a paragraph on it. It was a big help whether people will admit it or not.” “I think the first one is bad because you don’t really know what you’re doing and how to approach the whole thing, but after doing it you realize that you are learning the subject because you had to write a paragraph on it. It was a big help whether people will admit it or not.”

QualitativeResults – More Positive Comments Qualitative Results – More Positive Comments  Overall: “Although CPR was one of my least favorite things to do in this class, I think the good in it outweighs the bad. I think that especially in the science fields, students don’t have to do a lot of writing and so they don’t develop communication skills that they will need later on in life. I think communication is very important and it is something that you just have to work on. I think students will look back and wish they would have done more stuff like CPR.” “Although CPR was one of my least favorite things to do in this class, I think the good in it outweighs the bad. I think that especially in the science fields, students don’t have to do a lot of writing and so they don’t develop communication skills that they will need later on in life. I think communication is very important and it is something that you just have to work on. I think students will look back and wish they would have done more stuff like CPR.” “It seems like a pain at the time, but I can already see how much I learned from it. Please continue to do it, it helps more than people realize.” “It seems like a pain at the time, but I can already see how much I learned from it. Please continue to do it, it helps more than people realize.”  Other positives: Helped link chemistry to real life and their professional future, developed time management and research skills Helped link chemistry to real life and their professional future, developed time management and research skills

QualitativeResults – Overview Qualitative Results – Overview

Final Comments  CPR is an example of the move from teacher- centered learning to student-centered learning  If you add writing, peer review and technology to the mix, should I have been surprised at the level of student angst?  Students have a more positive experience when the instructor actively promotes & demonstrates CPR in the classroom, actively promotes & demonstrates CPR in the classroom, makes the assignments a significant part of grade, makes the assignments a significant part of grade, is involved in writing assignments, is involved in writing assignments, gives personalized support gives personalized support regrades when warranted regrades when warranted

Thanks to my co-authors: Dr. Nancy Simpson, Director, Center for Teaching Excellence Ms. A. Baris Gunersel, Ph.D. Student, Educational Psychology Graduate Research Assistant Center for Teaching Excellence