Effective Feedback, Rubrics, and Grading Pamela Rist Instructor in Epidemiology Harvard T. H. Chan School of Public Health

My Teaching Experiences Tend to teach in larger introductory level epidemiology or integrated epidemiology and biostatistics courses Often have several TAs who are all working together to help grade the homework Need to be consistent across graders and also make sure the TAs grade the same way I would (if I had the time to grade all the assignments myself) Important to give the students high quality feedback since topics learned in courses I teach are the foundations for future classes

Giving effective feedback Teaching and assessment are interconnected concepts Important to remember that giving feedback is not just to “rank” students – can also be an important way to judge how effectively the information is being relayed and if the students understand the material What would you like to see if you were the student?

General Tips Answer keys Writing comments on papers Be consistent Turn-around times Reviewing common mistakes Giving feedback during seminars/labs

Grading Keys Useful for problems sets or short response questions More detailed than an answer key Used by the TAs to help ensure consistent grading Highlights the “must haves” in an answer Provides advice for how to grade common mistakes Examples

Example Grading Key for problem on calculating and interpreting IRRs b) Calculate the incidence rate among the exposed. (1 point) Take off ½ point if they forget units Give full credit if set-up is right but final number is wrong Partial credit if some part is wrong (but base off the 2x2 table above – if that was incorrect, it’s OK to use the incorrect numbers here) Incidence rate = 30 cases /63,359 person-years = 0.000473 cases/person-year =4.73 cases/10,000 person-years e) Interpret the incidence rate ratio in words. (1 point) Take off ½ if no reference group is stated Make sure they say “rate” not risk or odds (write in correct answer) Heavy smokers have 1.76 times the rate of developing CHD compared to non-smokers.

Example of a Grading Key for a Short Answer Questions from a Study Critique What was the outcome of interest? How was the outcome assessed? How complete was follow-up information? (3 points) 1 point for outcome 1 point for how assessed (self-report for non-fatal events; vital statistics, NDI or family/post office for fatal; confirmed by med record review --- take off ¼ if any of these three major pieces is missing) 1 point for completeness – take of ½ if they only talk about fatal or nonfatal Do you believe that the findings observed in this study are valid? How did the authors address chance, bias, and confounding as potential alternative explanations for their results? (4 points) Need to address chance, bias, and confounding – 1 point each Should also have a conclusion about whether they think it’s valid – if they don’t take off ½ point May get a range of answers on this one – make sure it’s reasonable and take off ½ points as needed if the logic does not work or if it’s under the wrong heading (putting a confounding scenario under chance for example)

Try on your own The investigators indicated that they “adjusted” for a number of factors in their multivariate analyses. What factors did they adjust for? What does it mean to “adjust” for a factor? And what were they trying to accomplish by doing so? (2 points) Things to think about: What are the “must haves”? How much should each “must have” be worth? What would be some common errors?

How would you grade these answers? They adjusted for age, BMI, smoking, parental history of MI, alcohol use, history of hypertension, and physical activity. They “adjusted” for these factors to control for confounding by these factors. The authors adjusted for lifestyle factors and medical history. They were trying to improve the generalizability of their results. They adjusted for age, BMI, smoking, parental history of MI, alcohol use, history of hypertension, and physical activity. They were trying to control for selection bias and make the two groups more alike with respect to any baseline differences in these potential confounders.

Rubrics Another method to help ensure consistent grading Find these helpful for grading group work, discussion board posts, papers, posters, etc. Different formats Often have a lists of dimensions or tasks that need to be covered and how much each contributes to the overall grade Can also include guidance on how to assess each area (for example, 3 point scale or a 5 point scale)

Example of Components in a Poster Project Rubric Background (1 point) What is the current state of knowledge? What is the specific gap in knowledge to be filled by this study? Study Objective (1 point) State your main objective Methods (4 points) State your study design Describe your study population Describe the definition of your exposure and outcome and how they were measured Describe how the exposure and outcome were treated in the analysis. Were they categorized? Did you treat them as continuous variables? Describe your main analysis plan. What statistical tests will you perform? If you use regression, state the type of regression used and the covariates included in the model. Address how chance, bias, and confounding are incorporated into your analysis. For example, did you adjust for confounding? What confounders were included? Were the confounders linear or categorical variables?

Example of Components in a Rubric for a Final Paper Criterion Exceeds Expectations Meets Expectations Below Expectations Demonstrated Knowledge   (50%) Shows complete understanding of the exercise Demonstrates full knowledge of the subject with explanations and elaboration Points: 12.5 Shows substantial understanding of the exercise At ease with subject content Able to elaborate and explain to some degree Points: 9.375 Response shows some understanding of the exercise Uncomfortable with content Only basic concepts are demonstrated and interpreted Points: 6.25 Requirements  (30%) Goes beyond the requirements of the exercise Points: 7.5 Meets the requirements of the exercise Points: 5.625 Does not meet the requirements of the exercise Points: 3.75 Report Format (10%) Format is consistent throughout, including heading styles and captions Figures and tables are presented logically and reinforce the text Points: 2.5 Format is generally consistent, including heading styles and captions Figures and tables are neatly done and provide intended information Points: 1.875 Mostly consistent format Figures and tables are legible, but not convincing Points: 1.25  Adapted from: https://sharepoint.louisville.edu/sites/sphis/tlr/Shared%20Documents/Course+Syllabus+Rubric+Examples.doc.