Design and Analysis of Open Response Surveys: Lessons Learned Dr. Joan Burtner Associate Professor of Industrial Engineering Mercer University

Design and Analysis of Open Response Surveys Designing Survey Administering Survey Conducting Survey Analysis Reporting Survey Results IIE/SHS FEB 2012 Mercer University School of EngineeringSlide 2

Open Response Surveys Ask participants to respond to a question such as “What can we do to improve patient care? Excellent method for collecting “Voice of the Customer” data within a Quality Mgt. System Allows analysts to interpret qualitative characteristics (tone, frustration, pride) Opportunity to discover unanticipated responses IIE/SHS FEB 2012 Mercer University School of EngineeringSlide 3

Designing and Administering the Survey Purpose of survey Delivery method ( paper, electronic, phone, in-person, etc.) Length of survey, allowable response time Incentives for participation Anonymity, confidentiality Who will collate and/or analyze data? How will data be analyzed? IIE/SHS FEB 2012 Mercer University School of EngineeringSlide 4

Design of Survey Questions Structure, specificity of written questions ◦ What can we do to improve the process? ◦ What is the one thing that will make your job easier? ◦ What is the “one idea” that will improve the registration process? Text-based questionnaires vs. oral interviews ◦ Limited, prescribed, uniform questions ◦ Non-uniform follow on questions in addition to prescribed questions IIE/SHS FEB 2012 Mercer University School of EngineeringSlide 5

Example 1: On-line Survey Question Please describe challenges that make it difficult to deliver patient care. Over 700 responses submitted anonymously over a two week period Data downloaded into an Excel file Many rows of data contained no text, just time and date of submission Two 2-person teams reviewed the data and developed preliminary codes IIE/SHS FEB 2012 Mercer University School of EngineeringSlide 6

Example 1: Initial Coding Coding done electronically by labeling each response in Excel Initial decision to allow only one major code for each entry to simplify analysis. Twenty-four categories emerged between the two teams. Overlap and clarification of labels resulted in the elimination of four categories. IIE/SHS FEB 2012 Mercer University School of EngineeringSlide 7

Example 1: Revised Coding Difficult to gain consensus among all four coders on the “one” main concept for each response Realization that multiple codes increased validity of work Excel spreadsheet expanded to four columns to accommodate multiple codes for each response IIE/SHS FEB 2012 Mercer University School of EngineeringSlide 8

Code - Process Change? Patients should not be transferred to med surg during change of shift. There needs to be a window of time that patients cannot be transferred. It is very unfair to the patient to be transferred at that time as going off nurses are trying to get caught up and give report; and coming on nurses are trying to get report. For example; no transfers between 6:00 and 07:15pm IIE/SHS FEB 2012 Mercer University School of EngineeringSlide 9

Code - Work Load? Policy? If a patient is computer literate, why do we not have some of the admission forms on line so that patients can fill out the questions at home? They would certainly have access to their medications & we might have a more exact list of what they take. They would be in a more comfortable & less stressful environment when filling out the forms & the older folks would most likely have a family member to assist in remembering when they had previous procedures, etc. IIE/SHS FEB 2012 Mercer University School of EngineeringSlide 10

Audience Participation Exercise Form teams of two or three Consider the following responses (handout) How would you code the data? Do multiple codes increase validity? IIE/SHS FEB 2012 Mercer University School of EngineeringSlide 11

Example 2: Nursing Interruptions Research agreement with a local hospital Students observed nurses at work Data collected orally and transcribed by observers Anecdotal reports from nurses collected periodically as time permitted IIE/SHS FEB 2012 Mercer University School of EngineeringSlide 12

Example 2: Nursing Interruptions Data Excerpt 1 April 15th Comments by Nurse - Interruptions Helping with code lift or code response Family Telephone calls Doctors When technician needs help Comments by Observer 1 Tech was using chart station when Nurse was trying to chart Having room in another section causes a lot of extra walking This could be a system failure interruption IIE/SHS FEB 2012 Mercer University School of EngineeringSlide 13

Example 2: Nursing Interruptions Data Excerpt 2 April 16th Comments by Nurses - Interruptions Family Lab Calls asking about results or talking about results Doctors calling Patients – getting called by other patients and helping other patients Unexpected events – Patients getting out of bed, codes Helping nurses, covering nurses when they are out for lunch Comments by Observer 2 Quiet day with not many interruptions Nurse didn’t have much to do IIE/SHS FEB 2012 Mercer University School of EngineeringSlide 14

Coding Nursing Interruptions Nature of the interruption ◦ Phone call, call light, patient’s family, healthcare professional, etc. Urgency of the interruption ◦ Avoidable, justifiable, etc. Consequence of the interruption ◦ Delay, error, etc. IIE/SHS FEB 2012 Mercer University School of EngineeringSlide 15

Example 3: On-line Survey Question Please list and describe ideas that will contribute to cost efficiency within our system. Over one hundred responses submitted Data downloaded into an Excel file Single person reviewed the data and categorized each response Fifty-eight usable responses Data organized into a pivot table IIE/SHS FEB 2012 Mercer University School of EngineeringSlide 16

Reporting Analysis Results Example 1 ◦ Ten-page written report ◦ Power-Point presentation to five-person committee ◦ Follow-up submission: frequency distribution models overall and departmentalized Example 2 ◦ In-person feedback to nurse manager ◦ Brief report with Pareto diagrams Example 3 ◦ Electronic feedback to client ◦ Pivot table demonstration IIE/SHS FEB 2012 Mercer University School of EngineeringSlide 17

Code Frequency - All Locations IIE/SHS FEB 2012 Mercer University School of EngineeringSlide 18

Frequency - Aberdeen IIE/SHS FEB 2012 Mercer University School of EngineeringSlide 19

Frequency - Dallas IIE/SHS FEB 2012 Mercer University School of EngineeringSlide 20

Frequency - Greenville IIE/SHS FEB 2012 Mercer University School of EngineeringSlide 21

Example 3: Pivot Table 1 IIE/SHS FEB 2012 Mercer University School of EngineeringSlide 22

Example 3: Pivot Table 2 IIE/SHS FEB 2012 Mercer University School of EngineeringSlide 23

Example 3: Pivot Table 3 IIE/SHS FEB 2012 Mercer University School of EngineeringSlide 24

Lessons Learned 1 Nurses are willing to spend time completing surveys. Many are eager to have their ideas heard. It is important to give feedback, when possible, with respect to how the survey results may change the process. IIE/SHS FEB 2012 Mercer University School of EngineeringSlide 25

Lessons Learned 2 The process of analyzing and quantifying open- ended responses is very time-consuming. It is very difficult to gain consensus on some responses. System-wide data should also be reported at the unit level if possible. In the case of “one best idea” it is difficult to prioritize. IIE/SHS FEB 2012 Mercer University School of EngineeringSlide 26

Lessons Learned 3 Due to the somewhat subjective nature of the coding and frequency tabulation, it may be difficult to use open-ended survey responses to measure the effectiveness of process changes. However, changes in the culture and attitudes of the respondents may be apparent. IIE/SHS FEB 2012 Mercer University School of EngineeringSlide 27

Questions / Contact Information Dr. Joan Burtner Associate Professor of Industrial Engineering and Industrial Management (478) Mercer University School of Engineering 1400 Coleman Avenue Macon, GA IIE/SHS FEB 2012 Mercer University School of EngineeringSlide 28