1. The Quest for Quality: Monitoring and Measuring Outcomes in Continuing Nursing Education Session 1: Outcomes for Learning Activities Pamela S. Dickerson, PhD, RN-BC, FAAN Director of Continuing Education, Montana Nurses Association President, PRN Continuing Education Presented by Ohio Nurses Association and Montana Nurses Association 1

2. Disclosures Purpose: Participants will effectively develop learning and provider unit processes that lead to measurable outcomes demonstrating quality patient care and/or enhanced professional development of registered nurses. Criteria for Successful Completion: Participate in this entire webinar. Complete the evaluation and registration information and return to ONA as directed. Conflict of Interest: There is no conflict of interest for any planner or presenter for this activity. 2

3. Objective Examine strategies for development of learning activities that produce measurable outcomes. 3

4. Self-Assessment I know how toStrong = 3 Moderate = 2Weak = 1 Conduct a meaningful needs assessment Analyze gaps Develop an outcome statement Measure outcomes 4 Total Score =

5. Why Do Outcomes Matter? Evidence related to patient safety Evidence related to quality of care Evidence related to life-long learning Accountability o To nurses o To patients o To employers o To the public 5

6. Benefits of Outcomes Evaluation for Providers of Continuing Education Credibility Sustainability Provide evidence to support the value of continuing nursing education (Read: How to keep your job!) 6

7. What is an Outcome? Webster: “A final consequence”, “result” NPD Scope & Standards, 2010: “something that follows, is the result of, or is the consequence of a project, program, or event” Outcome Measurement: “The process of observing, describing, and quantifying predefined indicators of outcomes of performance.” 7

8. What Do We Mean By “Measurable”? Webster – Able to be measured; of distinguished importance – SIGNIFICANT Webster – “Measure” = basis of comparison Quantifiable EVIDENCE Observable 8

9. Considering Evaluation Strategies: A Traditional Kirkpatrick Model Process – what are we doing? Product – what did we do? Outcomes – what difference did it make? Impact – how is our organization better as a result? 9

10. A Different Perspective Consider turning this upside down: o How do we want to make our organization better? (Where are the gaps?) o What outcomes do we want to achieve in our learning activity to help that happen? o What behaviors do we want to see in our learners to achieve that outcome? o What learning objectives and strategies will help learners achieve these behaviors? 10

11. Evaluation in Context Target Audience Needs Assessment Gap Analysis PurposeObjectives Content/ Teaching Methods Short-term Evaluation Outcome (long term evaluation) 11

12. Educational Design Components Target Audience: who needs this information? Gap Analysis: why do they need it? Learning Styles: how can they learn it best? Purpose: what will learners be able to do as a result of participating in the learning activity? Outcome: what measurable evidence will show that the gap has been filled? 12

13. Planning for Outcomes Starts at the Beginning With the given target audience, what is the gap in knowledge, skill, or practice? Gap Analysis Worksheet 13 What is the current state? What is the desired state? What evidence supports the need? What do we need to measure? How do we capture the data?

14. Gap Analysis In 2012, the Joint Commission released a sentinel event alert regarding opioid use in hospitalized patients. We need to teach this to our nurses. Is this a strong gap analysis statement? o Yes o No 14

15. 15

16. Gap Analysis What nurses need this information? Among this group, is there currently a problem in the way opioid medication is administered? o Too much? o Too frequently? o Not enough attention to alternatives o Not thorough assessment of patient need o Other What data do we have that shows a gap? If there is a gap, is it in knowledge, skill, or application in practice? What do we need to do to fill the gap? 16

17. A Stronger Statement In light of the 2012 Joint Commission Sentinel Event Alert regarding appropriate use of opioid medications among acute care patients and the fact that, in our medical-surgical unit, 80% of patients have received opioid medication in the last 14 days without evidence of assessment for other potentially effective types of analgesic support, there is a gap in … 17

18. Next Questions Do the nurses know about the recommendations from TJC? Are prescribers aware of the recommendations? How are pain assessments currently done? Are they thorough enough? Do they consider non- pharmacological and non-opioid options? What are the documentation options in the EMR? 18

19. Planning for Outcomes Starts at the Beginning How do you decide on a purpose for a learning activity? How does the purpose provide the learner with information about what he/she is expected to do as a result of participating in this activity (as opposed to what the provider will offer)? 19

20. Purpose The purpose of this activity is to provide information about safe opioid use among patients in the acute care hospital. Is this a strong purpose statement? o Yes o No 20

21. 21

22. Purpose The fact that information is being provided may be of value to the provider but is of no benefit to the learner. A better approach – the nurse will use opioid analgesia judiciously in accordance with TJC recommendations when caring for medical-surgical patients. 22

23. Outcome What is the desired outcome of our educational intervention? How will we be able to know when “success” has been achieved? (the gap has been filled – or lessened) 23

24. Outcome Statement Medical-surgical patients will receive pain management in accordance with TJC recommendations regarding opioid use. Is this a strong outcome statement? o Yes o No 24

25. 25

26. Outcome Statement We know the population – medical-surgical patients We know the standard – TJC recommendation We don’t know what “success” looks like – how many patients? What assessment data will be collected? Documented? Is success based on the assessment or the intervention? 26

27. A Stronger Statement From April 1-14, 2014, 90% of patients on the medical-surgical unit will have documentation reflective of nursing assessment related to type of pain and options for non-pharmacologic and/or non-opioid pain management. Previous data is 20% - gap to close is between 20 and 90 Why this outcome? The nurse cannot control the prescribing, but can advocate for the patient by collecting and documenting relevant data 27

28. For Consideration Is self-report a valid “measure” of accomplishment? 28

29. Yes No 29

30. For Consideration Self-report is a valid way to capture measurable data regarding a person’s perceptions This is the basis of phenomenological research May want to consider other ways of validating that data 30

31. Other Options Pre-test / Post-test / Follow-up test Surveys of managers Nurse educator observations of practice Chart reviews Monitoring of incidents, e.g. medication errors Other 31

32. External Sources of Data: Examples National Database for Nursing Quality Indicators: o Clinical: falls, pressure ulcers, pain assessments, healthcare- acquired infections, etc. o Non-clinical: workforce data, practice environment, etc. AHRQ: Extensive data base related to health care quality initiatives Centers for Disease Control and Prevention 32

33. Internal Sources of Data: Examples Risk management Quality / Performance Improvement Incident reports Recruitment / Retention Data Finance – cost of care, cost of equipment & supplies, etc. Readmission data 33

34. You’re Not Alone You don’t have to reinvent the wheel – or keep spinning yours. Seek help in data collection and analysis – both to support the need for the activity (gap analysis) and to measure the results (your outcome). 34

35. Does Your Educational Plan Meet the Assessed Need? Issue / gap clearly identified Desired outcome clear and measurable Supported by others It’s physically, mentally, emotionally “do-able” Supplies, equipment, resources available Value identified and shared Educational plan specific and targeted 35

36. Self-Assessment I know how toStrong = 3Moderate = 2Weak = 1 Conduct a meaningful needs assessment Analyze gaps Develop an outcome statement Measure outcomes 36 Total Score =

37. Your Own Measurable Outcome What was your score at the beginning of the session? What is your score now? If your score has increased, you have narrowed the gap between what you knew and what you now know. The best score is a 12. My pre-session score was a 5, my post- session score is a 9. Therefore, I have decreased the knowledge gap by 4 points. Now, how will you use this information? (Retitle the tool “I am able to” and assess yourself in 60-90 days.) – note: if ONA/MNA wanted to collect outcomes data, we could have you submit this additional information. 37

38. Questions, Ideas, and Inspirations 38

39. For Further Information Pam Dickerson, PhD, RN-BC, FAAN pam@mtnurses.org pdickerson@aol.com 39

40. References / Resources American Nurses Association and National Nursing Staff Development Organization (2010). Nursing Professional Development: Scope and Standards of Practice. Silver Spring, MD: Nursebooks.org. DeSilets, L. (2009). Connecting the dots of evaluation. Journal of Continuing Education in Nursing 40(12), 532- 533. Dickerson, P., and Chappell, K. (2012). Purpose, objectives, and outcomes: tying the pieces together. Journal of Continuing Education in Nursing 43(9), 389-390. 40

41. References / Resources Dickerson, P. (2011). Evaluating an activity: Beyond the “form”. Journal of Continuing Education in Nursing 42(7), 292-293. Dickerson, P. (2008). Addressing barriers to enhance outcomes. Journal of Continuing Education in Nursing 39(10), 437-438. Gibbs, J. and Reissour, S. (2010). Evaluating Outcomes: Evidence of Success, Journal of Continuing Education in Nursing, 41(11), 484-485. How to evaluate learning: The Kirkpatrick model for the 21 st century. http://www.dashe.com/blog/informal-learning/how- to-evaluate-learning-the-kirkpatrick-model-for-the-21st-centuryhttp://www.dashe.com/blog/informal-learning/how- to-evaluate-learning-the-kirkpatrick-model-for-the-21st-century 41