Designing Valid Reliable Grading Tools Using VALUE Rubrics Joyce O’Reilly Department of Nursing Education Franklin Pierce University Portsmouth, NH

Objectives Define student learning outcome (SLO) Understand why it is important to assess SLO Identify components of an analytic rubric Understand VALUE Rubrics Create an assessment tool using VALUE Rubrics

What is a student learning outcome – SLO? Glad you asked! The Council of Higher Education Accreditation defines SLO as “the knowledge, skills, and abilities that a student has attained at the end of his or her engagement in a particular set of higher education experiences” (2003, p. 5) Why is it important to assess SLO in higher education?

Why assess SLO? Accrediting bodies and other stakeholders require institutions of higher learning to Provide evidence students are learning what we say they are learning Use the evidence to inform program changes Of the 1400 Association of American Colleges & Universities (AAC&U) member institutions 91% use rubrics to assess student work Of the 91%, 42% use VALUE Rubrics

What is a VALUE Rubric and why should I care? VALUE = Valid Assessment of Learning in Undergraduate Education Collaboration of >100 Association of American Colleges & Universities (AAC&U) members Resulted in 15 rubrics with high reliability and validity Three major categories written communication quantitative reasoning critical thinking

VALUE Rubrics Civic Engagement Foundations and Skills for Lifelong Learning Oral Communication Problem Solving Quantitative Literacy Reading Teamwork Integrative Learning 9. Creative Thinking 10. Critical Thinking 11. Ethical Reasoning 12. Information Literacy 13. Inquiry and Analysis 14. Intercultural Knowledge and Competence 15. Written Communication

Department of Nursing Education Evolution of Assessment Plan Accreditation Commission for Education in Nursing (ACEN) Identified lack of evidence of SLO assessment in undergraduate program Concluded our assessment plan was lackluster Faculty researched nursing program assessment plans Faculty researched assessment tools Decision made to develop assessment tools for benchmark assignments using VALUE Rubrics

Process used by nursing faculty Examined all assignments in all undergraduate courses Identified assignments from all courses that “matched” one or more of the program SLO Dissected the assignment Broke assignment into steps Matched a “trait” from a VALUE Rubric to the assignment step For example “Identify the socioeconomic and health issues of a family” Define Problem (Problem Solving Rubric) “Critique five evidence based articles and identify the gaps in knowledge” Analysis (Inquiry & Analysis Rubric)

What did the process look like? I’m so glad you asked!

SLO # 7 Practice as accountable leaders of conscience demonstrating ethical and professional principles in the care of all individuals, families, and populations at risk in a variety of settings. Ethics Paper Assignment (NR 320) The purpose of this assignment is to assess understanding and analysis of ethical and professional principles in the care of all individuals, families, and populations at risk in a variety of settings.  

SLO #7 Practice as accountable leaders of conscience demonstrating ethical and professional principles in the care of all individuals, families, and populations at risk in a variety of settings. Ethics Paper Assignment The purpose of this assignment is to assess understanding and analysis of ethical and professional principles in the care of all individuals, families, and populations at risk in a variety of settings.  

Ethics Paper Description Students will demonstrate ethical and professional principles by: Discussing the issues of respect, beneficence, and justice relevant to a specific vulnerable population who could be in a research study. Exploring special considerations that may be required for informed consent for the chosen vulnerable population. Describing how they (the student) values the need for ethical conduct in practice and research when dealing with a vulnerable patient and informed consent.

Ethics Paper Description, cont. 4. Describing the interprofessional and intraprofessional resources used to solve the ethical dilemmas of your vulnerable population. 5. Applying the ethical perspectives of respect, beneficience and justice when acting to prevent unethical practice. 6. Analyzing implications for nurses involved in the research involving vulnerable population and discuss barriers or limitations that might be encountered in practice.

Ethics Paper, cont. Students will demonstrate ethical and professional principles by: Discussing the issues of respect, beneficence, and justice relevant to a specific vulnerable population who could be in a research study. Exploring special considerations that may be required for informed consent for the chosen vulnerable population. Describing how they (the student) values the need for ethical conduct in practice and research when dealing with a vulnerable patient and informed consent.

VALUE Rubrics Civic Engagement Foundations and Skills for Lifelong Learning Oral Communication Problem Solving Quantitative Literacy Reading Teamwork Integrative Learning 9. Creative Thinking 10. Critical Thinking 11. Ethical Reasoning 12. Information Literacy 13. Inquiry and Analysis 14. Intercultural Knowledge and Competence 15. Written Communication

  Trait 4 Mastering 3 Proficient 2 Developing 1 Emerging Ethical Reasoning: 1. Ethical Issue Recognition Discusses the issues of respect, beneficence, and justice relevant to a specific vulnerable population who could be in research study. Recognizes ethical issues when presented in a complex, multilayered (gray) context and can recognize cross-relationships among the issues. Recognizes ethical issues when issues are presented in a complex, multilayered (gray) content or can grasp cross-relationships among the issues. Recognizes basic and obvious ethical issues and grasps (incompletely) the complexities or interrelationships among the issues Recognizes basic and obvious ethical issues but fails to grasp complexity or interrelationships.

  Trait 4 Mastering 3 Proficient 2 Developing 1 Emerging Ethical Reasoning: 1. Ethical Issue Recognition Discusses the issues of respect, beneficence, and justice relevant to a specific vulnerable population who could be in research study. (20%) Recognizes ethical issues when presented in a complex, multilayered (gray) context and can recognize cross-relationships among the issues. (20 pt) Recognizes ethical issues when issues are presented in a complex, multilayered (gray) content or can grasp cross-relationships among the issues. (16-19 pt) Recognizes basic and obvious ethical issues and grasps (incompletely) the complexities or interrelationships among the issues. (14-15 pt) Recognizes basic and obvious ethical issues but fails to grasp complexity or interrelationships. (0-13 pt)

Ethics Paper, cont. Students will demonstrate ethical and professional principles by: Discussing the issues of respect, beneficence, and justice relevant to a specific vulnerable population who could be in a research study. Exploring special considerations that may be required for informed consent for the chosen vulnerable population. Describing how they (the student) values the need for ethical conduct in practice and research when dealing with a vulnerable patient and informed consent.

VALUE Rubrics Civic Engagement Foundations and Skills for Lifelong Learning Oral Communication Problem Solving Quantitative Literacy Reading Teamwork Integrative Learning 9. Creative Thinking 10. Critical Thinking 11. Ethical Reasoning 12. Information Literacy 13. Inquiry and Analysis 14. Intercultural Knowledge and Competence 15. Written Communication

  Trait 4 Mastering 3 Proficient 2 Developing 1 Emerging Ethical Reasoning: 1. Ethical Issue Recognition Discusses the issues of respect, beneficence, and justice relevant to a specific vulnerable population who could be in research study. (20%) Recognizes ethical issues when presented in a complex, multilayered (gray) context and can recognize cross-relationships among the issues. (20 pt) Recognizes ethical issues when issues are presented in a complex, multilayered (gray) content or can grasp cross-relationships among the issues. (16-19 pt) Recognizes basic and obvious ethical issues and grasps (incompletely) the complexities or interrelationships among the issues. (14-15 pt) Recognizes basic and obvious ethical issues but fails to grasp complexity or interrelationships. (0-13 pt) 2. Ethical Issue Recognition Explores special considerations might be required for informed consent for the vulnerable population you choose? (15%) Student can recognize ethical issues when presented in a complex, multilayered (gray) context AND can recognize crossrelationships among the issues. (15%) Student can recognize ethical issues when issues are presented in a complex, multilayered (gray) context OR can grasp crossrelationships among the issues. (12-14 pt) Student can recognize basic and obvious ethical issues and grasp (incompletely) the complexities or interrelationships among the issues. (10-11 pt) ethical issues but fails to grasp complexity or Interrelationships. (0-9 pt)

Ethics Paper, cont. Students will demonstrate ethical and professional principles by: Discussing the issues of respect, beneficence, and justice relevant to a specific vulnerable population who could be in a research study. Exploring special considerations that may be required for informed consent for the chosen vulnerable population. Describing how they (the student) values the need for ethical conduct in practice and research when dealing with a vulnerable patient and informed consent.

VALUE Rubrics Civic Engagement Foundations and Skills for Lifelong Learning Oral Communication Problem Solving Quantitative Literacy Reading Teamwork Integrative Learning 9. Creative Thinking 10. Critical Thinking 11. Ethical Reasoning 12. Information Literacy 13. Inquiry and Analysis 14. Intercultural Knowledge and Competence 15. Written Communication

  Trait 4 Mastering 3 Proficient 2 Developing 1 Emerging Ethical Reasoning: 1. Ethical Issue Recognition Discusses the issues of respect, beneficence, and justice relevant to a specific vulnerable population who could be in research study. (20%) Recognizes ethical issues when presented in a complex, multilayered (gray) context and can recognize cross-relationships among the issues. (20 pt) Recognizes ethical issues when issues are presented in a complex, multilayered (gray) content or can grasp cross-relationships among the issues. (16-19 pt) Recognizes basic and obvious ethical issues and grasps (incompletely) the complexities or interrelationships among the issues. (14-15 pt) Recognizes basic and obvious ethical issues but fails to grasp complexity or interrelationships. (0-13 pt) 2. Ethical Issue Recognition Explores special considerations might be required for informed consent for the vulnerable population you choose? (15%) Student can recognize ethical issues when presented in a complex, multilayered (gray) context AND can recognize crossrelationships among the issues. (15 pt) Student can recognize ethical issues when issues are presented in a complex, multilayered (gray) context OR can grasp crossrelationships among the issues. (12-14 pt) Student can recognize basic and obvious ethical issues and grasp (incompletely) the complexities or interrelationships among the issues. (10-11 pt) ethical issues but fails to grasp complexity or Interrelationships. (0-9 pt) 3. Ethical Self-Awareness Describes how they value the need for ethical conduct in practice and research when dealing with a vulnerable patient and informed consent. (10%) Student discusses in detail/analyzes both core beliefs and the origins of the core beliefs and discussion has greater depth and clarity. (10 pt)  Student discusses in detail/analyzes both core beliefs and the origins of the core beliefs. (8-9 pt) Student states both core beliefs and the origins of the core beliefs. (7 pt) Student states either their core beliefs or articulates the origins of the core beliefs but not both. (0-6 pt)

  Trait 4 Mastering 3 Proficient 2 Developing 1 Emerging Ethical Reasoning: 1. Ethical Issue Recognition Discusses the issues of respect, beneficence, and justice relevant to a specific vulnerable population who could be in research study. (20%) Recognizes ethical issues when presented in a complex, multilayered (gray) context and can recognize cross-relationships among the issues. (18-20 pt) Recognizes ethical issues when issues are presented in a complex, multilayered (gray) content or can grasp cross-relationships among the issues. (16-17 pt) Recognizes basic and obvious ethical issues and grasps (incompletely) the complexities or interrelationships among the issues. (14-15 pt) Recognizes basic and obvious ethical issues but fails to grasp complexity or interrelationships. (0-13 pt) 2. Ethical Issue Recognition Explores special considerations might be required for informed consent for the vulnerable population you choose? (15%) Student can recognize ethical issues when presented in a complex, multilayered (gray) context AND can recognize crossrelationships among the issues. (15 pt) Student can recognize ethical issues when issues are presented in a complex, multilayered (gray) context OR can grasp crossrelationships among the issues. (12-14 pt) Student can recognize basic and obvious ethical issues and grasp (incompletely) the complexities or interrelationships among the issues. (10-11 pt) ethical issues but fails to grasp complexity or Interrelationships. (0-9 pt) 3. Ethical Self-Awareness Describes how they value the need for ethical conduct in practice and research when dealing with a vulnerable patient and informed consent. (10%) Student discusses in detail/analyzes both core beliefs and the origins of the core beliefs and discussion has greater depth and clarity. (9-10 pt)  Student discusses in detail/analyzes both core beliefs and the origins of the core beliefs. (8 pt) Student states both core beliefs and the origins of the core beliefs. (7 pt) Student states either their core beliefs or articulates the origins of the core beliefs but not both. (0-6 pt)

Lessons Learned Matching assignments (benchmark) to SLO required tweaking of both assignments and SLO Process forced faculty to understand what the SLO measures Developing rubrics helped identify important pieces of assignments Process provided valuable insight into gaps in the curriculum Allowed specific feedback to students Provided feedback that has been used for course/program improvement Promoted consistency with evaluation (particularly with adjuncts)

References Accreditation Commission for Education in Nursing. (2013). Accreditation manual. Retrieved from http://www.acenursing.net/manuals/SC2013_BACCALAUREATE.pdf Association of American Colleges & Universities. (2007). VALUE. Retrieved from https://www.aacu.org/value Council for Higher Education Accreditation. (September 2003). Statement of mutual responsibility for student learning outcomes: Accreditation, institutions, and programs. Retrieved from http://www.chea.org/pdf/StmntStudentLearningOutcomes9-03.pdf Finley, A. (2011). How reliable are the VALUE rubrics? Peer Review, 13/14(4/1). Retrieved from https://www.aacu.org/peerreview/2011-2012/fall-winter/finley Gleason, B. L., Gaebelein, C. J., Grice, G. R., Crannage, A. J., Weck, M. A., Hurd, P.,…Duncan, W. (2013). Assessment of students’ critical-thinking and problem-solving abilities across a 6-year doctor of pharmacy program. American Journal of Pharmaceutical Education, 77(8), 1-12. Renjith, V., George, A. G. R., & D’Souza, P. (2915). Rubrics in nursing education. International Journal of Advanced Research, 3(1), 423-438.