Leading with Results: How Questioning Validity Fosters Proactive and Engaged Program Assessment 2014 Assessment Institute in Indianapolis October 21 st, 2014 Stephen R. Wallace Anne-Marie Kuchinski Mary Elaine Koren Joeseph Kutter Tawanda Gipson S. R. Wallace © 2014

Session Outcomes 1.Describe how to use a results-driven activity to lead a program through a deeper analysis of student data, with the goal of fostering more engaged and proactive assessment. 2.Describe a process for determining the predictive validity of an assessment. 3.Interpret a correlation matrix by identify assessments that are measuring something in common (or not in common). S. R. Wallace © 2014 Leading with Results

Driving Question Is the costly, high stakes, pre-graduation HESI exam a predictor of success on the post-graduation NCLEX-RN exam (in which one must pass AFTER GRADUATION for licensure)? S. R. Wallace © 2014 Leading with Results

The Context Northern Illinois University Nursing, B.S. degree program –Must pass NCLEX for accreditation and program review –Student Learning Outcomes –Mix of program assessment methods S. R. Wallace © 2014 Leading with Results

The Context Student Learning OutcomesAssessment Methods 1.Evaluate safe, quality, patient-centered, evidence-based nursing care to individuals, families, and communities. NCLEX-RN (1-7) 2.Evaluate critical thinking/clinical reasoning when providing nursing care. HESI Exam (1-7) 3.Implement quality improvement related to patient care.Classroom Assessments (1-7) 4.Establish collaborative relationships with members of the interdisciplinary team. Portfolio (1,2) 5.Incorporate information management principles, techniques, and systems when providing nursing care. Student Survey (1-7) Faculty Survey (1-7) 6.Provide leadership role in a variety of healthcare settings for the purpose of providing and improving patient care. Baccalaureate supplemental Alumni Questionnaire (1-7) 7.Defend use of professional, ethical, and legal principles while implementing the roles of the registered nurse as provider, designer/manager/coordinator of care, and member of the profession. Employer Feedback (1-7) S. R. Wallace © 2014 Leading with Results

Predictive Validity Definition –The degree to which scores on an assessment predict future performance on another measure Process –Administer Assessment A, wait a period of time, administer Assessment B, calculate 1.Degree of relationship between assessments - or - 2.Accuracy of A predicting B S. R. Wallace © 2014 Leading with Results

Answering the Question Replication –Langford & Young (2013), Predicting NCLEX-RN Success With the HESI Exit Exam: Eighth Validity Study NCLEX-RN is P/F → Predict accuracy Used highest HESI Exit Exam score and whether student passed NCLEX-RN S. R. Wallace © 2014 Leading with Results

Answering the Question Predictive Validity Results Predictive Accuracy of HESI a (Langford & Young (2013)) HESI Scoring Categoryn b Pass NCLEX-RN c n% 900 and higher1,5601, Note. a n = 3,758. b Accuracy in predicting NCLEX-RN success regardless of whether the student was required to take up to three versions of the HESI before achieving a score in either scoring category. c NCLEX-RN scores are only reported as pass/fail. Predictive Accuracy of HESI a (Nursing, B.S. program) HESI Scoring Categoryn b Pass NCLEX-RN c n% 900 and higher and below Note. a Accuracy in predicting NCLEX-RN success based on student’s highest score regardless of which of three versions of the HESI the student took before achieving a score in any scoring category. b n = 643. c NCLEX-RN scores are only reported as pass/fail. HESI accurately predicts NCLEX S. R. Wallace © 2014 Leading with Results

Deepening the Discussion Through Guiding Questioning –Guided Discovery approach –Scaffold strategic questions –Extend Wait-Time I and II Examples –Did you answer the original question? –What are program improvement implications? –Do you see additional patterns in the data? What trends do you see? S. R. Wallace © 2014 Leading with Results

Deepening the Discussion Predictive Accuracy of HESI (Nursing, B.S. program) HESI Scoring Categoryn Pass NCLEX-RN n% 900 and higher and below Is there a cut score for remediation? S. R. Wallace © 2014 Leading with Results

Deepening the Discussion Combined Pass Rate and Potential Cut Scores Percent Passing NCLEX-RN (Nursing, B.S. program) HESI Scoring Categoryn Pass NCLEX-RNFail NCLEX-RN Combined Pass Rate n%n% 963 and higher and below and below S. R. Wallace © 2014 Leading with Results

Deepening the Discussion Discussion leads to new discoveries –University Writing Project AAC&U VALUE Rubrics written communication and critical thinking –VSA CLA written communication and analytical reasoning –HESI critical thinking S. R. Wallace © 2014 Leading with Results

Deepening the Discussion What could we do with this new information? What questions could be answered? Think – Pair – Share for a moment S. R. Wallace © 2014 Leading with Results

New Questions Arise Are the assessments measuring the same things? Can the Nursing program capitalize on the authentic assessments they are already using? Answer the questions through –Correlational study –Similar to a predictive validity study S. R. Wallace © 2014 Leading with Results

Correlation Coefficient, r S. R. Wallace © 2014 Leading with Results

VALUE Rubric – Critical Thinking VALUE Rubric – Written Communication CLA – Analytical Reasoning CLA – Written Expression CLA – Writing Mechanics HESI – Critical ThinkingACTGPANCLEX VALUE Rubric – Critical Thinking – VALUE Rubric – Written Communication.88**– CLA – Analytical Reasoning.24.18– CLA – Written Expression **– CLA – Writing Mechanics.40**.38**.74**.72**– HESI – Critical Thinking.48**.55** – ACT.29* *.43**– GPA.37**.33*.19.30*.22.49**.38**– NCLEX.21.31*.15.33*.68**.45**.25– Note. ** p < 0.01, * p < 0.05 Correlation Matrix Note. ** p < 0.01, * p < ** S. R. Wallace © 2014 Leading with Results

Correlation Matrix (Shading) VALUE Rubric – Critical Thinking VALUE Rubric – Written Communication CLA – Analytical Reasoning CLA – Written Expression CLA – Writing Mechanics HESI – Critical ThinkingACTGPANCLEX VALUE Rubric – Critical Thinking – VALUE Rubric – Written Communication.88**– CLA – Analytical Reasoning.24.18– CLA – Written Expression **– CLA – Writing Mechanics.40**.38**.74**.72**– HESI – Critical Thinking.48**.55** – ACT.29* *.43**– GPA.37**.33*.19.30*.22.49**.38**– NCLEX.21.31*.15.33*.68**.45**.25– Note. ** p < 0.01, * p < 0.05 S. R. Wallace © 2014 Leading with Results

VALUE Rubric – Critical Thinking VALUE Rubric – Written Communication CLA – Analytical Reasoning CLA – Written Expression CLA – Writing Mechanics HESI – Critical ThinkingACTGPANCLEX VALUE Rubric – Critical Thinking – VALUE Rubric – Written Communication.88**– CLA – Analytical Reasoning.24.18– CLA – Written Expression **– CLA – Writing Mechanics.40**.38**.74**.72**– HESI – Critical Thinking.48**.55** – ACT.29* *.43**– GPA.37**.33*.19.30*.22.49**.38**– NCLEX.21.31*.15.33*.68**.45**.25– Note. ** p < 0.01, * p < 0.05 Correlation Matrix (Same Constructs).24.48**.11.72**.20.38** S. R. Wallace © 2014 Leading with Results

VALUE Rubric – Critical Thinking VALUE Rubric – Written Communication CLA – Analytical Reasoning CLA – Written Expression CLA – Writing Mechanics HESI – Critical ThinkingACTGPANCLEX VALUE Rubric – Critical Thinking – VALUE Rubric – Written Communication.88**– CLA – Analytical Reasoning.24.18– CLA – Written Expression **– CLA – Writing Mechanics.40**.38**.74**.72**– HESI – Critical Thinking.48**.55** – ACT.29* *.43**– GPA.37**.33*.19.30*.22.49**.38**– NCLEX.21.31*.15.33*.68**.45**.25– Note. ** p < 0.01, * p < 0.05 Correlation Matrix (Different Constructs) **.40**.55**.85**.74**.18 S. R. Wallace © 2014 Leading with Results

VALUE Rubric – Critical Thinking VALUE Rubric – Written Communication CLA – Analytical Reasoning CLA – Written Expression CLA – Writing Mechanics HESI – Critical ThinkingACTGPANCLEX VALUE Rubric – Critical Thinking – VALUE Rubric – Written Communication.88**– CLA – Analytical Reasoning.24.18– CLA – Written Expression **– CLA – Writing Mechanics.40**.38**.74**.72**– HESI – Critical Thinking.48**.55** – ACT.29* *.43** – GPA.37**.33*.19.30*.22.49**.38**– NCLEX.21.31*.15.33*.68**.45**.25– Note. ** p < 0.01, * p < 0.05 Correlation Matrix (Outcomes) S. R. Wallace © 2014 Leading with Results

Correlation Conclusions Written Communication and Critical Thinking appear to be related CLA appears to relate more to itself than other important program outcomes VALUE Rubric assessment correlates with important program outcomes VALUE Rubrics appears more valid than CLA at the program and university level S. R. Wallace © 2014 Leading with Results

Program Assessment What should the Nursing program do? And the university? Why? Think – Pair – Share for a moment S. R. Wallace © 2014 Leading with Results

Program Assessment Integrating the VALUE Rubrics into program assessment requires an alignment between Outcomes Portfolio Rubric VALUE Rubric S. R. Wallace © 2014 Leading with Results

Program Assessment Are they aligned? Crosswalk the: –Program Outcomes (p.1) –Adapted VALUE Rubric (p. 9) –Portfolio Rubric (p. 10) Is anything missing in the Outcomes? Outcomes Portfolio Rubric VALUE Rubric S. R. Wallace © 2014 Leading with Results

Program Assessment Written Communication is NOT one of the student learning outcomes Written Communication IS emphasized throughout the curriculum Steps Nursing program is taking S. R. Wallace © 2014 Leading with Results

Program Assessment Program assessment is effective and efficient when assessments and corresponding rubrics –Are aligned with Course-level objectives Program student learning outcomes Broader university outcomes –Are used for multiple purposes Formatively Summatively Internally Externally S. R. Wallace © 2014 Leading with Results

Lessons Learned How have you used assessment results to foster more engaged and proactive assessment on your campus? Have you –Put results in the hands of users? –Asked probing questions? –Guided improvement efforts? S. R. Wallace © 2014 Leading with Results

Workshop Outcomes 1.Describe how to use a results-driven activity to lead a program through a deeper analysis of student data, with the goal of fostering more engaged and proactive assessment. 2.Describe a process for determining the predictive validity of an assessment. 3.Interpret a correlation matrix by identify assessments that are measuring something in common (or not in common). S. R. Wallace © 2014 Leading with Results

Contact Stephen R. WallaceAssociate Director Office of Assessment Services Northern Illinois University DeKalb, IL Anne-Marie KuchinskiUndergraduate NCLEX-RN Coordinator Nursing Program Northern Illinois University DeKalb, IL Mary Ellaine KorenAssociate Professor and Area Coordinator Nursing Program Northern Illinois University DeKalb, IL S. R. Wallace © 2014 Leading with Results

S. R. Wallace © 2014 Leading with Results