The Many Uses of Self-Assessment Edward F. Gehringer Department of Computer Science North Carolina State University Our work in peer and self-assessment is funded by Google and NSF http://tinyurl.com/CHEP-Self

Outline What is self-assessment useful for? http://tinyurl.com/CHEP-Self Outline What is self-assessment useful for? How is self-assessment used? How reliable is self-assessment? How can self-assessment be improved? How can self-assessment scores be used? Examples of ethical analyses Gehringer, Many uses of self-assessment efg@ncsu.edu

Advantages of self-assessment? http://tinyurl.com/CHEP-Self Advantages of self-assessment? Gehringer, Many uses of self-assessment efg@ncsu.edu

Some advantages of self-assessment http://tinyurl.com/CHEP-Self Some advantages of self-assessment Every student gets immediate feedback Judging correctness of answers deepens students’ understanding of material Students become aware of own strengths, progress, gaps Gehringer, Many uses of self-assessment efg@ncsu.edu

More advantages of self-assessment http://tinyurl.com/CHEP-Self More advantages of self-assessment Student becomes an autonomous learner A useful life skill Gehringer, Many uses of self-assessment efg@ncsu.edu

Outline What is self-assessment useful for? http://tinyurl.com/CHEP-Self Outline What is self-assessment useful for? How is self-assessment used? How reliable is self-assessment? How can self-assessment be improved? How can self-assessment scores be used? Examples of ethical analyses Gehringer, Many uses of self-assessment efg@ncsu.edu

How is self-assessment used? http://tinyurl.com/CHEP-Self How is self-assessment used? To teach students how to study Capstone courses Medical schools, for clinical skills E-portfolio movement Summatively …. Capstone courses serve to encourage self-assessment by requiring students to apply previously-learned skills to new contexts. Self-assessment is often used in medical schools as a way of improving students’ clinical skills. Nursing education, like teacher education, emphasizes becoming a reflective practitioner. Thinking about how current material relates to material learned in other courses can benefit anyone in a degree program. The e-portfolio movement is another practice that encourages self-assessment. Students put together work that they have produced in various contexts. The very act of assembling it promotes self-assessment, even if it is not identified as such. The most direct summative use is for the staff to grade students on the quality of their self-assessment, perhaps as one component of the grade for an assignment. There are also several approaches to combining self-assessment grades with peer grades or staff grades, as we shall see. Gehringer, Many uses of self-assessment efg@ncsu.edu

Outline What is self-assessment useful for? http://tinyurl.com/CHEP-Self Outline What is self-assessment useful for? How is self-assessment used? How reliable is self-assessment? How can self-assessment be improved? How can self-assessment scores be used? Gehringer, Many uses of self-assessment efg@ncsu.edu

Do students tend to overrate themselves? http://tinyurl.com/CHEP-Self Do students tend to overrate themselves? Boud & Falchikov: 27 studies 16/11 Gonzalez & Murthy: self-assessment of problem sets “Slightly higher” Kahoot 1. How many studies on self-review find that students overrate themselves? All Most Few None Boud & Falchikov: 16 studies that report overrating by students, 11 that report underrating. All before 1995 Gonzales and Murthy: Self-assessment of problem sets in a Signals and Systems course. Self-assessment scores were “slightly” higher than instructor assessments. Gehringer, Many uses of self-assessment efg@ncsu.edu

Do students tend to overrate themselves? http://tinyurl.com/CHEP-Self Do students tend to overrate themselves? Medical schools Liang: 3rd-yr. course Fudan U. med school 117 students Self-assessment mean: 95.5% Peer-assessment mean: 94.7% Tutor assessment mean: 90.5% Papinczak: U. Queensland med school 1st yr. Bachelor of Medicine students, n = 125 On each assignment, self-assessment mean lower than staff (p < 0.001). Gehringer, Many uses of self-assessment efg@ncsu.edu

Do students tend to overrate themselves? http://tinyurl.com/CHEP-Self Do students tend to overrate themselves? Kulkarni: Two iterations of Coursera MOOCs 1st iteration: Self-grades 6% above median peer grade 2nd iteration: Self-grades 7.5% higher That’s self-grades vs. peer grades. What about staff grades? 1st time: 48% of staff grades > peer grades 40% below peer grades 2nd time: 36% of staff grades > peer grades 46% below peer grades Gehringer, Many uses of self-assessment efg@ncsu.edu

Do students tend to overrate themselves? http://tinyurl.com/CHEP-Self Do students tend to overrate themselves? What factors may play a role? Level of detail in the rubric Amount of training in self-assessment How many assessments they’ve seen Whether they think high self-ratings will help or hurt their grade Gehringer, Many uses of self-assessment efg@ncsu.edu

“Good” vs. “bad” students http://tinyurl.com/CHEP-Self “Good” vs. “bad” students Boud & Falchikov: 11 studies Low performers overrated selves in all 11. Kahoot 2: Good students Tend to overrate themselves Rate themselves realistically Tend to underrate themselves Usually underrate themselves Boud and Falchikov report results from eleven studies that looked at this issue between 1932 and 1988 (all but one of which were after 1962). In the other 2 papers, high performers were called “more accurate raters.” High performers under- rated selves in 9 of 11. Gehringer, Many uses of self-assessment efg@ncsu.edu

What about summative self-review? http://tinyurl.com/CHEP-Self What about summative self-review? Boud & Falchikov: 7 studies address this point … Would students cheat??? … Wilkowski et al., 2014 5/2 70.3% of students in 1st MOOC awarded selves full credit 9.9% of these submissions were blank or nonsense 8.5% were evid. plagiarized Five of the seven said that students overrated themselves in summative situations. In the remaining two studies, it depended on level (undergraduates overrated themselves while graduate students underrated themselves) or discipline (civil engineers underrated themselves). Kahoot 3: In Wilkowski’s study, what % of students do you think awarded themselves full credit on their self-review for the first MOOC? 90% 70% 50% 30% 8.5% of submissions were duplicates of other submissions (e.g., downloaded from the web, or examples given in class). One artifact was submitted by 40 different students. In the second MOOC, however, only 51.5% awarded selves full credit; 2.8% were blank, and 0.9% were duplicates of other submissions. In a MOOC, students may have a relaxed sense of ethics, since they are not paying the course or using it for credentialing (most of the time, anyway). But, on the other hand, they are under less pressure to cheat. Gehringer, Many uses of self-assessment efg@ncsu.edu

Gender differences? Boud & Falchikov: 6 studies 3 found no differences http://tinyurl.com/CHEP-Self Gender differences? Boud & Falchikov: 6 studies 3 found no differences 1: Women’s self-ratings closer to instructor ratings 1: Men underrated selves less than women 1: Male elementary teacher trainees overrated selves; women secondary trainees overrated selves Kahoot 4: What do more studies find? Men self-assess selves higher than women do There is no difference between men & women Women self-assess selves higher than men do Falchikov and Boud enumerate six studies that addressed the issue between 1969 and 1988. Gehringer, Many uses of self-assessment efg@ncsu.edu

Outline What is self-assessment useful for? http://tinyurl.com/CHEP-Self Outline What is self-assessment useful for? How is self-assessment used? How reliable is self-assessment? How can self-assessment be improved? How can self-assessment scores be used? Examples of ethical analyses Gehringer, Many uses of self-assessment efg@ncsu.edu

Rubrics Why use a rubric? Students can help create the rubric http://tinyurl.com/CHEP-Self Rubrics Why use a rubric? Tell students what to look for “Fairness” in assessment Students can help create the rubric How detailed? Longer rubric: Draw attention to more criteria But also more fatigue Probably less textual feedback Short rubric: More textual feedback Students can miss things Gehringer, Many uses of self-assessment efg@ncsu.edu

http://tinyurl.com/CHEP-Self Rubric advice

Outline What is self-assessment useful for? http://tinyurl.com/CHEP-Self Outline What is self-assessment useful for? How is self-assessment used? How reliable is self-assessment? How can self-assessment be improved? How can self-assessment scores be used? Examples of ethical analyses Gehringer, Many uses of self-assessment efg@ncsu.edu

Combining self-review & peer review http://tinyurl.com/CHEP-Self Combining self-review & peer review was the first review system to incorporate both peer and self-review. Step 1: Instructor gives students a writing assignment. Step 2: Students submit their document to CPR. Kahoot 5. In Calibrated Peer Review, what does “calibrated” refer to? Instructor changes peer scores Students grade sample work Peers review students of equal strength Gehringer, Many uses of self-assessment efg@ncsu.edu

http://tinyurl.com/CHEP-Self Calibration Step 3: CPR gives the student 3 calibration documents for review. Basic idea: Training course for reviewers How they do  how much credence they get Before students review peers, they get 3 works to review 1 exemplary Their agreement with instructor  Reviewer Competency Index Others have known defects Gehringer, Many uses of self-assessment efg@ncsu.edu

http://tinyurl.com/CHEP-Self How Does CPR Work? Step 4: After calibration, CPR gives student 3 peer documents for review. Step 5: CPR gives the student his or her document for self-review. Step 6: CPR provides a detailed report of the peer review and the self-review. Gehringer, Many uses of self-assessment efg@ncsu.edu

http://tinyurl.com/CHEP-Self The Review Process The review process is the same for calibration, peer review, and self-review. The student answers content questions for each document. The student answers style questions for each document. The student assigns a score to each document on a scale of 1 to 10. Gehringer, Many uses of self-assessment efg@ncsu.edu

How self-review fits in http://tinyurl.com/CHEP-Self How self-review fits in Students get full credit for self-reviewing if their grades agree within the range set by the instructor between their self-assessment and the weighted value of their peer reviews. Instructor is notified to check reviews/self-assessment when reviewers of a student did not “train well.” Gehringer, Many uses of self-assessment efg@ncsu.edu

Fair Contribution Scoring http://tinyurl.com/CHEP-Self Fair Contribution Scoring Students given 20 points per team member to divide … Each week they fill out a rubric. Everyone’s points are totaled. Everyone’s total is divided by n to derive a weighting factor. Student’s grade = weighting factor ×project grade 20 pts. per team member to divide among all members of the team, including themselves. Gehringer, Many uses of self-assessment efg@ncsu.edu

SPARK: SPA + SAPA Uses both SPA is computed like in FCS. SAPA > 1  http://tinyurl.com/CHEP-Self SPARK: SPA + SAPA Uses both Self- and Peer Assessment (SPA) Self-Assessment to Peer Assessment Factor (SAPA) SPA is computed like in FCS. SAPA > 1  student rated own contribution higher than team members did; SAPA < 1  team members rated student’s contribution higher than student. Honest assessment through embarrassment Kahoot 6? Students who rate themselves higher than their peers rate them deserve More credit than their peers The same amount of credit as their peers Less credit than their peers Any of the above “The potential embarrassment of receiving a SAPA factor much higher than 1.0 appeared to be a significant motivator in promoting honest assessment.” Gehringer, Many uses of self-assessment efg@ncsu.edu

http://tinyurl.com/CHEP-Self Summary Self-assessment promotes metacognition, produces learning gains. Students have a tendency to overrate selves, but it can be mitigated. When used for grades, self-assessments are usually combined with peer or instructor assessments Gehringer, Many uses of self-assessment efg@ncsu.edu

Self-assessment: pitfalls http://tinyurl.com/CHEP-Self Self-assessment: pitfalls Confounding effects Early studies asked students to predict how well they would do in course. Effort is often included as a criterion. What is a good self-assessment? Is it the same kind of assessment an instructor would give? Could become self-fulfilling prophecy Gehringer, Many uses of self-assessment efg@ncsu.edu

What is self-assessment useful for? Gehringer, Many uses of self-assessment efg@ncsu.edu