Blackboard Rubrics: The good, the bad, and the ugly Adel Gordon & Pepijn Kalis University of Northampton | Blackboard 10th April 2013

Introductions Adel Gordon @adelgordon University of Northampton Learning Technologist Been at Northampton for 13 years, an LT for 6 Pepijn Kalis pepijn.kalis@blackboard.com Blackboard EMEA Sr. Specialist Working for Blackboard for 5 years

Rubrics What are rubrics? Scoring guide Authentic assessment tool Working guide for students

Let’s do some tasting assessing

Chocolate Chip Cookies 4 Delicious 3 Good 2 O.K. 1 Poor Number of Chips Chocolate chip in every bite Chips in about 75% of bites Chocolate in 50% of bites Too few or too many chips Texture Chewy Chewy in middle, crisp on edges Texture either crispy/crunchy or 50% uncooked Texture resembles a dog biscuit Color Golden brown Either light from overcooking or light from being 25% raw Either dark brown from overcooking or light from undercooking Burned Taste Home-baked taste Quality store-bought taste Tasteless Store-bought flavor, preservative aftertaste – stale, hard, chalky Richness Rich, creamy, high-fat flavor Medium fat contents Low-fat contents Nonfat contents

Let’s see the cookie rubric in Blackboard Demo

The good,… Rubrics for teaching Rubrics for learning In practice

Rubrics for teaching Clarify learning goals from the offset Design materials and activities that address those goals Communicate those goals to students Guide feedback on students’ progress Assess products to degree to which the goals are met Andrade, 2005 Enable timely grade allocation whilst justifying them at the same time Feedback > Feed Forward … Where as a lot of student feel that rubrics help with their learning by setting out goals, tutors perspectives of using rubrics tend to lean towards the grading advantages... they serve a good purpose of marking, enabling them to mark consistently and with more objective focus. Andrade reported that in areas where resistance/hesitation to rubrics existed resulted from a lack of understanding of the learning and teaching benefits of a rubric. It was felt that tutors would be more receptive if they understood how rubrics can enhance learning and teaching.

… teaching Rubric design based on learning outcomes PLAN TEACH ASSESS REFLECT Make adjustments to teaching based on reflections Emphasise the use of rubrics Identify common areas of strengths and weaknesses Teaching model adapted slightly from Introduction to Rubrics by Stevens and Levi. Northampton CAIEROS - module development process - teaching teams start from scratch - learning and teaching specialist - learning technologist - librarian - student - support and align assessment with learning outcomes, activities - use a rubric to layout criteria, aligned with LOs * appropriate/understandable language A few studies that I've read promote the design of rubrics for assessment with the students. Positing that students are more likely to use one if they played a part in designing it than when it see it only as a way for the tutor generate grades. > takes us back to "emphasising the use of rubrics to students"... Jessica Poettecker student union - Develop Partnerships students. Involved them in the development of their learning... Including them in the developments of rubrics can allow for this I would like to look into the use of a rubric for the module development process for programme/module design and evaluating programmes At Northampton we use a module development process where teaching teams start from scratch to develop a module. Throughout this process the teaching team, together with a learning and teaching specialist, learning technologist, student and librarian set out learning outcomes, activities etc to support and align assessment with outcomes. Using a rubric to lay out the criteria for the assessments - aligned with learning outcomes - makes everything really simple as long appropriate/understandable language is used in the marking criteria. Look for patterns Score student work using rubric Adapted from Stevens & Levi, 2013

Rubrics for learning Learning Learner Face to face Online Experiential Surface Strategic Deep thinker Learning: Theme throughout all types of learning is similar – the teaching model is used where refinements are made throughout the overall end to end process learning is improved. Tailored Transparent goals Reducing uncertainty by identifying critical issues in an assignment Evaluating their own performance in order to get immediate feedback Focus efforts for subsequent assignments "Instructional illuminators" Popham, 1997 Learner: Surface learner – bare minimum Strategic – do what is expected to get best grade they can Deep thinker – everything plus more

In practice Rubrics set out expectations to aid the student to understand what they’re being graded against Rubrics let students know how their grade was calculated and where they could improve their work Develops/drives a level of professionalism and enables me to provide more timely feedback Moderation can take place immediately and feedback is ready to be released speedily Rubrics give the marker confidence that you can been more objective then subjective Makes calculating the overall grade easier when using a multifaceted approach to assessment criteria The definitions defined in the marking criteria provide detailed explanations of what a student must do to demonstrate a skill, proficiency, or criterion in order to attain a particular level of achievement. Excellent, very good, good satisfactory etc... Scoring, or marking, work based on these definitions makes the students job of interpreting the tutors judgment more easy to understand the targets for their learning When marking in teams consistency needs to be monitored. In the Tii case study on marker was marking much more harshly than the others which, because of problems with moderation (later), wasn't picked up until after grades were released.

Overview of workflow for assessment process Student submission First grading Internal moderation Provisional grade & feedback to students Record grade in SIS External moderation Overview of workflow for assessment process External moderation Record grade in SIS Provisional grade & feedback to students Internal moderation First marking Student Submission

… the bad, Reporting An actual report However, this report works when working to identify patterns of strengths and weaknesses in achievement of assessment criteria across a set of students. Practicalities of using the report within the workflow… Outcomes system… ?

.. and the ugly Intuitiveness Usability Importing/exporting Saving Integration It works, but mostly it feels more like a workaround or something that is an after thought that hasn’t quite been integrated properly Next slide... Demo...

In practice Let’s have a look: http://nile.northampton.ac.uk Go into Adel-apr-13... 2 browsers? Demo Kaltura, Turnitin, journal, presentation.

Blackboard vs Turnitin Feature Blackboard rubric Turnitin rubric Group submissions Tutors can add personalised feedback to all/any of the marking criteria Multiple markers Export marked work with rubric summary Marking by more than one tutor – Bb - use smart views Tii – Turnitin groups Exporting Bb – No, reporting issue Tii – GradeMark paper download includes rubric Use either Blackboard or Turnitin rubrics for: Individual assignments Setting an indefinite number of criteria and score ranges Setting marks by points or percentage or no marks (feedback only) Specifying feedback for each possible score/criterion in advance Automatically adding up marks Manually overriding marks if required Re-use with future assessments Exporting and importing rubrics (to allow sharing with others) *Limitations Blackboard rubrics cannot be printed by academics once completed – for example – for moderation purposes.   (A change request has been submitted to Blackboard for this but this will not be available until at least 2013.) If adding comments to a Blackboard rubric, remember to save your work often – it does not auto-save so will otherwise be lost in the event of a time-out or computer crash. University of Manchester

References Andrade (2005), Teaching with Rubrics: The Good, the Bad, and the Ugly. College Teaching, [online] Available at: http://www.uri.edu/assessment/uri/guidance/documents/Andrade_2005_Teachingwithrubrics.pdf Stevens, D. D. and Levi, A. J., 2013. Introduction to Rubrics. Virginia: Stylus Publishing University of Manchester, Rubrics – What are they? Why and how should I use them? [online] Available from: http://www.elearning.eps.manchester.ac.uk/rubrics-what-are-they-and-why-and-how-should-i-use-them/ Utah Education Network – Original Chocolate Chip Cookies. [online] Available at: http://www.uen.org/Rubric/rubric.cgi?rubric_id=2730

Contact Adel Gordon Pepijn Kalis University of Northampton Adel.Gordon@northampton.ac.uk http://blogs.northampton.ac.uk/learntech Pepijn Kalis Blackboard Pepijn.Kalis@blackboard.com