The Quest for Quality: Monitoring and Measuring Outcomes in Continuing Nursing Education Session 3: Monitoring and Measuring Outcomes for Provider Unit Effectiveness Pamela S. Dickerson, PhD, RN-BC, FAAN Director of Continuing Education, Montana Nurses Association President, PRN Continuing Education Presented by Ohio Nurses Association and Montana Nurses Association

Disclosures Purpose: Participants will effectively develop learning and provider unit processes that lead to measurable outcomes demonstrating quality patient care and/or enhanced professional development of registered nurses. Criteria for Successful Completion: Participate in the entire webinar. Complete the evaluation and registration information and return to ONA as directed. Conflict of Interest: There is no conflict of interest for any planner or presenter for this activity.

Objective Examine approaches to data collection and analysis related to the effectiveness of a provider unit in improving patient care and enhancing nursing professional development.

Reflection If you have participated in sessions 1 & 2, please rate your current ability to do the following: Strong = 3 Moderate = 2 Weak = 1 Conduct a needs assessment Analyze a gap Measure an activity outcome Write a provider unit outcome Your Score:

Reflection Data My score is closest to: 4 7 9 12

Self-Assessment Your Score = I know how to Strong = 3 Moderate = 2 Weak = 1 Develop goals and outcome measures for my provider unit Engage stakeholders in provider unit evaluation Evaluate provider unit data Measure provider unit effectiveness in relation to improved patient care or enhanced professional development of registered nurses Your Score =

My total score was closest to a: 4 7 9 12

Educational Design Process Accreditation Conceptual Framework ANCC 2013 – used with permission Educational Design Process Assessment of Learning Needs Planning and Faculty Design Principles Achievement of Objectives Structural Capacity Commitment Accountability Leadership Resources Nursing Professional Development: Continuing Nursing Education Quality Outcomes Evaluation Process Evaluation Participation Goals for Improvement Value/Benefit to Nursing Professional Development

Provider Unit Outcomes Measures Outcome: “The impact of structure and process on the organization…and the value/benefit to nursing professional development.” (ANCC, 2013). Outcome Measurement: “The process of observing, describing, and quantifying predefined indicators of outcomes of performance.” (ANA/NNSDO, 2010)

Evaluation in Context Outcome Measurement for Provider Unit Goals for Provider Unit Outcome Measures Provider Unit Structure & Function Learning Activities Activity Evaluation Outcome Measurement for Provider Unit PSD, 2013

Why Does Your Provider Unit Exist? Single- focused organization: Why is continuing education important enough to make it your business? What does “success” for your organization look like? Multi-focused organization: Why is continuing education part of what your organization does? How does continuing education benefit the organization?

Let’s Look at the Organization What is its mission? What are its goals? Who are its stakeholders? (Who cares about the organization, its success, and its functions?)

Your Provider Unit What are the goals for your provider unit? How do your goals match those of organization? (If they don’t, it’s time for some serious reassessment.)

Think Like A Reporter Who What Why Where How When

What Is The Best Time To Evaluate Your Provider Unit? Quarterly Annually When requested by administration When the provider application is due

What is the best way to frame Provider Unit Evaluation? How effectively do our structure and processes support our efforts in achieving our goals? Did we accomplish what we planned in our learning activities as a whole? Do we have administrative support and funding for our next provider application? When is our next provider application due?

Evaluating Your Provider Unit: Who Structural Capacity People Right mix of people? Right qualifications of people? Enough people? People doing the right things? Do people know how to do what’s needed as the system evolves?

Evaluating Your Provider Unit: What Structural Capacity: Physical and Material Resources Office space Office supplies Classroom space Equipment Up-to-date? Functional?

Evaluating Your Provider Unit - What Structural Capacity – Financial and Administrative Support Does primary nurse planner have authority to ensure adherence to criteria? Is there administrative support for the work of the provider unit? Are there financial resources for support of structure and processes of the provider unit as well as the educational activities?

Evaluating Your Provider Unit - When It depends… Some factors may need to be evaluated monthly or quarterly It may take time to collect data for some outcomes measures, ranging from several weeks to a year or more Organization may require an annual report and annual goals Provider renewal time is an excellent opportunity for a comprehensive review and analysis

Evaluating Your Provider Unit - Where Within the context of the provider unit itself Within the context of the organization Each will require unique perspectives, data, and analyses

Evaluating Your Provider Unit – How Look at components of structural capacity Look at components of processes – provider unit functions AND educational design processes Measure against your goals, which are supported by your predetermined outcomes measures Engage appropriate stakeholders

Stakeholders Who are the people who care? Learners Faculty Provider Unit Nurse Planners & Key Personnel Organizational Personnel Administration Finance Risk Management Others Community representatives Advisory Committee or Board Stockholders or owners, Board of Directors

Evaluate Your Provider Unit: Why Prove that you make a difference Validate your contribution to quality patient care Demonstrate your commitment to professional development of registered nurses

Example: Provider Unit Effectiveness (how stakeholder input helped improve a quality outcome measure) Outcome Measure: diversity of learning activities. Provider Unit Goal: Implement at least 5 web based learning opportunities for nurses in our hospital during calendar year 2013.

Stakeholder Input Learners requested opportunity for web-based learning activities they could access via the hospital’s intranet and complete at their convenience. Managers requested that contact hours be awarded for activities that did not require nurses to be away from the practice setting for the time required to attend classes.

Data In calendar year 2012, we offered 53 continuing education classes for our nurses – all were “real-time” learning opportunities. In calendar year 2013, we offered 3 web-based independent study activities. Our goal was 5, but the IT person resigned and has not been replaced.

Your Thoughts? Does this evidence address the question of how stakeholder input has helped to improve a quality outcome measure? Yes No

What Could Make It Stronger? Did learners feel that their needs were met with these 3 activities? What is the plan to move forward? Need to advocate for IT resources? What feedback has been received from managers in relation to their original request?

Example: Professional Development Outcome Measure: leadership development Provider Unit Goal: Enhance effectiveness of charge nurses and managers through a series of leadership classes and organizational initiatives.

Background Reason for this goal: Fits with organizational goal of improving staff satisfaction and retention through better relationships with front-line leadership. Evidence: Research demonstrating that new nurses leave 1st positions in significant numbers, primarily due to unsatisfactory relationships with front-line leaders. In our facility in 2012, 36% of new RNs left their positions within 18 months of hire. In our facility in 2012, 64% of RN staff rated their satisfaction as moderate to low and indicated relationship with management a key factor

Action Plan Survey staff nurses to get their perspectives on expectations of charge nurses and managers Survey charge nurses and managers to get their perspectives on their expectations of their roles Survey nursing administration to get their perspectives on expectations of charge nurses and managers

Action Plan Based on survey results, meet with nursing administration to develop a plan for role clarification, position description revisions, and other organizational initiatives to support more effective functioning of charge nurses and nurse managers. Plan coordinated educational initiatives. Establish planning committee to develop a series of leadership classes for front-line managers. Timeline for combined interventions – March through June, 2013

Educational Interventions Six half-day workshops for front-line managers over a three month period of time. Content from each session built on previous sessions Content addressed key areas of concern noted in surveys: staff team communication, interprofessional communication, conflict resolution, time management, workflow management, crisis intervention, dealing with bullying/intimidation, feedback & evaluation

Outcomes In the last 6 months of 2013, only 8% of new nurses (defined as within 18 months of hire) left their positions. In the staff satisfaction survey conducted in October of 2013, 72% of RNs ranked their satisfaction as moderate to high. Anecdotal comments indicated that the concentrated effort to better prepare and support front-line managers was a key factor.

Demonstrating the Evidence

More Evidence

So What? Has the provider unit contributed to the professional development of the front-line managers? Has the provider unit contributed to the professional development of ALL nurses in the organization?

Leadership: Another Example A college of nursing provider unit is aware of national data, supported by the IOM report on the future of nursing, that there is increased need for leadership education for registered nurses. A focus group of CNOs from area healthcare facilities supports the need at the local level.

Leadership Example Provider Unit Goal: Increase leadership capability of alumni and other participants in our CNE activities Outcome Measure: Professional Development - Leadership

Framing Your Plan For Goal Achievement Use of an evaluation model Revised Kirkpatrick Logic Moore Have a clear plan in place for what you’re going to do, what resources will be employed, and what outcome measurement data you will collect – begin with the end in mind!

Using the Revised Kirkpatrick Model What outcomes do we want? What behaviors are required of these nurse to increase their leadership capability? Where are the gaps? What are current capabilities and what are the desired capabilities? What KSAs do learners need to fill these gaps? What is the best learning strategy to address these needs?

Using the Logic Model What do we need? (better prepared leaders) To address this need, we must: If we do this effectively, the outcome will be: When leaders implement this outcome, we will see: When this happens, organizations will be better because:

Using The Moore Model Does the learner know? (is there a knowledge deficit?) Does the learner know how? (is there a skill deficit?) Can the learner show that he/she can use the knowledge? (is there a performance deficit?) Is the learner able to apply the knowledge? (is there a critical thinking/clinical judgment/application deficit in practice?)

The Outcome for The Provider Unit Provider Unit Goal: Increase leadership capability of alumni and other participants in our CNE activities Provider Unit Outcome Measure: professional development – leadership Stakeholders: CON leadership, CNO focus group, alumni, other learners in related activities Where’s the evidence? Self-report from participants Data from CNOs regarding leadership evidence in their organizations among nurses who participated in these activities Number of nurses engaged in leadership roles in practice or professional settings

Self-Assessment Your Score: I know how to Strong = 3 Moderate = 2 Weak = 1 Develop goals and outcome measures for my provider unit Engage stakeholders in provider unit evaluation Evaluate provider unit data Measure provider unit effectiveness in relation to improved patient care or enhanced professional development of registered nurses Your Score:

My total score was closest to a: 4 7 9 12

The Bottom Line Evaluation of your provider unit helps you identify strengths, contributions, and areas of opportunity. Providing evidence of accomplishments of your goals and related outcome measures helps you validate your worth to the organization and the nurses you serve.

Questions and Discussion

For Additional Information Pam Dickerson, PhD, RN-BC, FAAN pdickerson@aol.com pam@mtnurses.org Zandra Ohri, MA, MS, RN zohri@ohnurses.org

References American Nurses Association & National Nursing Staff Development Organization (2010). Nursing Professional Development: Scope and Standards of Practice. Silver Spring, MD. Nursesbooks.org American Nurses Credentialing Center. (2013). 2013 ANCC Primary Accreditation Application Manual for Providers and Approvers. Revision 3.0. Silver Spring, MD. Author. Commission on Accreditation, ANCC. (2012). The Value of Accreditation for Continuing Nursing Education: Quality Education Contributing to Quality Outcomes. Retrieved from: http://www.nursecredentialing.org/Accreditation/ResourcesServices/Accreditation-WhitePaper2012.pdf Dickerson, P. (2012). Rocking the boat: challenges ahead for continuing education providers. Journal of Continuing Education in Nursing 43(10), 443-450.

References Kirkpatrick, D. (1984). Evaluating Training Programs: The Four Levels. San Francisco: Barrett-Koehler. Kirkpatrick’s Four Level Evaluation Model. (2012). Retrieved from: http://www.nwlink.com/~donclark/hrd/isd/kirkpatrick.html Moore, D., Green, J., & Gallis, H. (2009). Achieving desired results and improved outcomes: integrating planning and assessment throughout learning activities. Journal of Continuing Education in the Health Professions 29(1), 1-15. W.K. Kellogg Foundation. (2004). Logic Model Development Guide. Battle Creek, MI; Author.