1. Receive your continuing education certificate 1.Log onto the Wifi: “Wifi @ OSU” – No password required 2.Go to atinursingconference.com/now – Complete the form – Enter the conference code: ROBERTS115 If you don’t have the ability to complete this process on a laptop, tablet, or smart phone, please see an ATI representative for the form to complete manually.

2. Educator Enrichment Conference | Columbus, OH Fostering Excellence and Distinction in Nursing Education Karin K Roberts, PhD, RN, CNE Using Data to Drive Evidence-Based Program-Related Decisions Roberts, 2015

3. Objectives

4. Program Evaluation

5. Key Words Assessment Systematic process Continuous Evidence-based decisions Quality improvement

6. Program Evaluation

7. Blueprint for program evaluation Terms used to describe plan – Systematic Evaluation Plan (SEP) – Program Evaluation Plan (PEP) – Comprehensive Evaluation Plan (CEP) Master Plan of Program Evaluation

8. Standard 6.1 – The systematic plan for evaluation of the nursing education unit emphasizes the ongoing assessment and evaluation of each of the following: Student learning outcomes; Program outcomes; Role-specific graduate competencies; and The ACEN Standards. – The systematic plan of evaluation contains specific, measurable expected levels of achievement; frequency of assessment; appropriate assessment methods; and a minimum of three years of data for each component within the plan. (ACEN, 2013) Systematic Evaluation Plan - ACEN

9. Standard IV.A – A systematic process is used to determine program effectiveness Elaboration: Written, ongoing, exists to determine achievement of program outcomes Comprehensive Identifies qualitative and quantitative data to be collected Includes timelines for collection, review of expected and actual outcomes, and analysis Periodically reviewed and revised (CCNE, 2013) Systematic Evaluation Plan - CCNE

10. Standard 6.2 –Evaluation findings are aggregated and trended by program option, location, and date of completion and are sufficient to inform program decision-making for the maintenance and improvement of the student learning outcomes and the program outcomes. (ACEN, 2013) Systematic Evaluation Plan - ACEN

11. Standard IV-H – Data analysis is used to foster ongoing program improvement Actual outcomes compared to expected outcomes Discrepancies inform areas of improvement Changes to program are deliberate, ongoing, and analyzed for effectiveness Faculty are engaged in program improvement process (CCNE, 2013) Systematic Evaluation Plan - CCNE

12. Key Words Systematic process Ongoing assessment (trended/aggregate data – 3 years) Levels of achievement/Outcomes (actual compared to expected) Program effectiveness Maintenance/improvement of SLOs and program outcomes

13. Outcomes – Program: Indicators that reflect the extent to which a program is achieving their mission and goals. NCLEX pass rates; Retention rates; Employment rates; Graduate and Employer Satisfaction rates. (ACEN, 2013; CCNE, 2013) – Student Learning Outcomes (SLOs) Statements of the characteristics or attributes of students upon graduation from a program of nursing. They are measurable, learner-oriented, and are attained as a result of program activities. (ACEN, 2013; CCNE, 2013) Program vs. Student Learning Outcomes

14. Format of SEP

15. ACEN - http://www.acenursing.net/resources/SampleSEP.pdfhttp://www.acenursing.net/resources/SampleSEP.pdf CCNE – does not provide recommended format PROGRAM EVALUATION ACEN Standard or Program Outcome or Student Learning Outcome PLAN IMPLEMENTATION ComponentExpected Level of Achievement (or program terminology) Frequency of Assessment Assessment MethodsResults of Data Collection and Analysis Including actual level/s of achievement Actions for Program Development, Maintenance, or Revision SEP Matrix Format - ACEN

16. Program Outcomes Summary Table - ACEN PROGRAM OUTCOME SUMMARY Required Program Outcomes Expected Level of Achievement (or program’s terminology) Actual Level of Achievement Resulting Actions Taken/To Be Taken with Timeline Action(s)Timeframe Performance on NCLEX and/or certifying examination Program Completion Program Satisfaction ---Graduate Satisfaction ---Employer Satisfaction Job Placement Other Outcome(a); identified by program (optional-not required by ACEN) ACEN - http://www.acenursing.net/resources/SampleSEP.pdfhttp://www.acenursing.net/resources/SampleSEP.pdf

17. SEP Matrix Format

18. Data Analysis Feedback Loop Unmet LOA/ Outcome Analyze Data/ Analyze New Data Develop Action Plan Implement Action Plan Collect New Data Roberts, 2015

19. Data Analysis Feedback Loop Unmet LOA/ Outcome Analyze Data/ Analyze New Data Develop Action Plan Implement Action Plan Collect New Data Roberts, 2015

20. Compare Actual with Expected LOAs/Outcomes Most important part of SEP – Data is collected, aggregated, trended (>/= to 3 yrs)

21. Compare Actual with Expected LOAs/Outcomes Most important part of SEP – Data is collected, aggregated, trended (>/= to 3 yrs) – Analyzed

22. Compare Actual with Expected LOAs/Outcomes Most important part of SEP – Data is collected, aggregated, trended (>/= to 3 yrs) – Analyzed (actual vs expected LOAs/Outcomes) – Evidence-based decisions are made based on data analysis

23. LOA/Outcome met – Maintain current state of program LOA/Outcome not met – Revise a component of the program LOA/Outcome not met – Develop a new component of the program Use strategy such as “root cause analysis” to analyze why there is a need for revision or development of a program component Evidence-Based Decisions Unmet LOA/ Outcome Analyze Data/ Analyze New Data Develop Action Plan Implement Action Plan Collect New Data

24. Structured collaborative process that uses data analysis to determine underlying/contributing factors of an adverse event with the goal of implementing corrective action(s). (adapted from CMS.gov) Problem-solving technique that attempts to get to the root of a problem. Using the” Five Whys” can help drill down to the root cause (sixsigma.com, 2015) Root Cause Analysis Strategy Unmet LOA/ Outcome Analyze Data/ Analyze New Data Develop Action Plan Implement Action Plan Collect New Data

25. Root Cause Analysis – 5 “Whys” UNMET LOA/ OUTCOME One sentence description of unmet LOA/outcome Why? Root Cause(s) (adapted from isixsigma.com, 2015)

26. Root Cause Analysis – 5 “Whys” UNMET LOA/ OUTCOME Review of curriculum reveals that the credit hour requirement of 76 credits is above the state limit and DOE recommendations (ACEN Standard 4.8). Why? Root Cause(s) Curriculum needs this much time to address content. New knowledge requires addition of new content. No content has been removed from curriculum to make room for new content. Content mapping has not been done to accommodate for the change in content. It has been 9 years since the curriculum was revised It has been 9 years since the curriculum was revised and content was mapped to specific courses. Subsequently, new content has been layered onto old content. An increase in credit hours was necessary to accommodate old and new content. (adapted from isixsigma.com, 2015)

27. Using the 5 Whys document provided, conduct a root cause analysis of your program issue. Document the following on your flip chart paper the issue; summary of possible underlying/ contributing factors; conclusion(s) regarding root cause of issue. Using this information and paper, present the results of your root cause analysis to the group Root Cause Analysis - Practice

28. When possible cause of unmet LOA/outcomes has been determined: – Develop corrective action plan – Set timeline for implement action plan and collection of new data (timeline) – Analyze actual vs expected LOA/outcome using new data according to timeline Development of Action Plan Unmet LOA/ Outcome Analyze Data/ Analyze New Data Develop Action Plan Implement Action Plan Collect New Data

29. Unmet SLO/OutcomeRoot Cause AnalysisAction Plan Minutes Committee/Month/ Date ACEN Standard 4.8- Length of time and credit hours for a program are consistent with…state and national standards, and best practices. It has been 9 years since the curriculum was revised and content was mapped to specific courses. Subsequently, new content has been layered onto old content. An increase in credit hours was necessary to accommodate old and new content. Jan 15, 2016-May 15, 2016 Conduct a literature review to determine appropriate number of credit hours for ADN curriculum and type of curricula that are currently being developed. Aug 15, 2016-May 15, 2017 Revise curriculum, consider revision to a concept-based curriculum. Send BON new curriculum’s required docs and submit substantive change report when BON approval is received. Aug 15, 2017 Implement new curriculum May 15, 2018 - Assess success of new curriculum’s first level of courses – edit as needed. May 15, 2019 - Assess success of new curriculum’s second level of courses – edit as needed Curriculum Committee January 15, 2016 Action Plan Developed Sample Action Plan Template

30. Using the results of the root cause analysis conducted on your program issue, develop an action plan and timeline for re-evaluation of LOAs/outcome(s). Document on your flip chart paper a brief description of the plan and timeline Using your paper, present your plan of action to the group Development of Action Plan - Practice

31. Using Data to Drive Evidence-Based Program- Related Decisions Assess LOAs/Outcomes – expected vs. actual —Analyze data and DOCUMENT (minutes) —Develop and implement evidence-based action plans —Collect new data according to timeline —Analyze new data —Determine if LOA/outcomes met —Develop new action plan… CONTINUOUS CYCLE Unmet LOA/ Outcome Analyze Data/ Analyze New Data Develop Action Plan Implement Action Plan Collect New Data Roberts, 2015

32. Accreditation Commission for Education in Nursing (2013). Accreditation manual. Atlanta, GA: ACEN Billings, D.M. & Halstead, J.A. (2012). Teaching in nursing: A guide for faculty. St. Louis, MO; Elsevier. Centers for Medicare and Medicaid Services (2015) Retrieved from https://www.cms.gov/medicare/provider-enrollment-and- certification/qapi/downloads/guidanceforrca.pdf Commission on Collegiate Nursing Education (2013) Standards of accreditation. Washington, DC: CCNE Escallier, L.A. & Fullerton, J.T. (2012). An innovation in design of a school of nursing evaluation protocol. Nurse Educator, 37(5), 187-191. References

33. iSixsigma (2015) Determine the root cause; The 5 whys. Retrieved from http://www.isixsigma.com/tools- templates/cause-effect/determine-root-cause-5-whys/. Keating, S. (2010). Curriculum development and evaluation in nursing. New York, NY: Springer. 11-23. Roberts. K., (2015). Data-action plan graphic. In Using Data to Drive Evidence-Based Program-Related Decisions [PowerPoint slides]. Educator Enrichment Conference, Columbus, OH. Stavropoulous, A. & M. Kelesi (2013). Concepts and methods of evaluation in nursing education – a methodological challenge. Health Science Journal, 6(1), 11-23. References