Training & Instructional Design

Training & Instructional Design
Needs Analysis Pt II: Data Collection Methods. This is the second presentation of the two-parting unit on Needs Analysis. We are still in the first stage of the ADDIE process in which you collect information to be used in the design, development, and implementation of a training program. Building on Part One of this unit, you will now learning how to select the appropriate data collection tools to meet the needs of the organization, learner, as well as the specific job or task. Needs Analysis Unit 2b This material (Comp20_Unit2b) was developed by Columbia University, funded by the Department of Health and Human Services, Office of the National Coordinator for Health Information Technology under Award Number 1U24OC

Training & Instructional Design Needs Analysis Unit 2b
Lecture Objectives Compare the methods used to collect data in a needs assessment Select appropriate data collection methods to meet different needs analysis situations In this unit we will explore how to assess the training needs of the organization and individual learners. By the end of this unit learners will be able to: Compare the methods used to collect data in a needs assessment. Select appropriate data collection methods to meet different needs analysis situations. Health IT Workforce Curriculum Version 3.0/Spring 2012 Training & Instructional Design Needs Analysis Unit 2b

ADDIE Analysis Design Develop Implement Evaluate In Lecture 1, we learned that needs analysis is the first step in designing training programs, and is required to determine if training is needed, where training is needed, what needs to be taught, and who needs to be trained, and explored a systematic way of gathering information to answer those questions. Health IT Workforce Curriculum Version 3.0/Spring 2012 Training & Instructional Design Needs Analysis Unit 2b

Data Collection The needs analysis stage of your training is likely to be successful if quality methods are used to collect data Different workplace and training situations require different data collection methods In essence, the needs analysis stage of instructional design is a series of data collection steps. The success of a needs analysis will depend on the quality of the methods used to collect the data. Not all methods will be appropriate or effective in each workplace and training situation, but there are steps you can take to ensure that the methods you choose are most appropriate for your audience and the types of information to be collected. Health IT Workforce Curriculum Version 3.0/Spring 2012 Training & Instructional Design Needs Analysis Unit 2b

Strategy Identify the purpose and audience Select your method and develop instrument Pilot test your instrument Revise and retest if needed Implement your instrument Data collection involves administering instruments, as well as gathering and organizing responses for analysis. A well-planned data collection strategy is critical to obtaining reliable, consistent and useful information. Throughout the process you will: Identify the purpose and audience Develop or select your instrument Pilot test your instrument Revise and retest if needed, and Implement your instrument. Health IT Workforce Curriculum Version 3.0/Spring 2012 Training & Instructional Design Needs Analysis Unit 2b

Identify the Purpose & Audience
For example, organizational analysis Need information about organizational goals, available resources, constraints, and support By assessing the attitudes and perceptions of staff and clinicians, and reviewing company documents Audience to target Senior clinicians who can speak accurately about the organization’s goals All employees? A cross-section of the workforce to get a holistic view Important first steps would be to answer the questions: What are we assessing? And, from whom will we get this information? Let’s say we’re conducting an organizational analysis. What type of information would we collect? We will want to focus on collecting information that will tell us about the organizational goals, available resources, constraints, and support. Some of this information would come from assessing the attitudes and perceptions of staff, but some of it will probably exist in company documents. Next, decide which employees to target for this data, and how many persons will give us a representative sample. In this example, it’ll be tempting to target senior personnel alone, because it seems likely that they would be able to speak accurately about the organization’s goals. But, shouldn’t we be also interested in how those goals are perceived by all employees? If so, we may consider targeting a cross-section of the workforce to get a holistic view. Every data collection task in a needs analysis should begin by taking this kind of critical look at what goals you are trying to achieve goals and the audience. Health IT Workforce Curriculum Version 3.0/Spring 2012 Training & Instructional Design Needs Analysis Unit 2b

Data Collection Methods
Survey Interview Focus/Discussion Group Observation Document Review The next step is to select a method that is appropriate for your audience and the kind of data to be collected. Would the same methods be used to collect data from both senior managers and office administrators? Or, would the methods used to assess staff perceptions be also used to complete an accurate job task analysis? In the next few slides we will take an in depth look at the following four data collection methods and discuss some advantages and disadvantages of each. Whether you use surveys, interviews, focus groups, observations, and document reviews, the choice will depend upon the type and size of the audience, the purpose of the assessment, timeline, budget and staff available. Health IT Workforce Curriculum Version 3.0/Spring 2012 Training & Instructional Design Needs Analysis Unit 2b

Surveys A way to get lots of information quickly and/or easily from a group of people individually Can be in-person, written, or by telephone Description and Purpose Let’s start with surveys. Surveys or questionnaires are best used when you want to collect lots of information on perceptions, beliefs, attitudes, or behaviors quickly and/or easily from a group of people. They can be administered in person, written, or by telephone. Health IT Workforce Curriculum Version 3.0/Spring 2012 Training & Instructional Design Needs Analysis Unit 2b

Surveys Advantages Can reach large numbers of people Questions can focus responses as either very specific or open-ended Can establish relationships among indicators. Can complete anonymously Inexpensive to administer Easy to compare and analyze Many sample questionnaires already exist Disadvantages No in-depth responses on issues Does not allow respondents to exchange ideas Limited to specific questions May not reach people who are hard to reach Wording can bias client's responses Are impersonal May not get full story The survey is one of the most commonly used data collection methods and is your best option when you want to collect information anonymously, and when you are trying to reach large numbers of people. They are also versatile and the questions in your survey can be written to elicit very specific or open-ended responses. No other option is better if numerical data is to be collected, such as time, frequency, rating, and so on. They are also inexpensive to administer; easy to compare and analyze; and many sample questionnaires already exist, so you often do not have to create your own. Additionally, many web-based survey tools are available today that make it easy to create, administer surveys, tally and often analyze survey responses. Even with these many advantages there are some challenges. As surveys are limited to specific questions, they tend to offer little in-depth responses on issues and you may not get the entire picture. You may be able to reach a large number of people, but the response rate can be low, and there won’t be an opportunity for respondents to exchange ideas. Survey questions can also be very difficult to write and if questions are misunderstood there is no recourse. Health IT Workforce Curriculum Version 3.0/Spring 2012 Training & Instructional Design Needs Analysis Unit 2b

Survey Writing Tips Use simple language Make the survey short Avoid double negatives Avoid open-ended questions Don't write leading questions Avoid double-barreled questions Offer an “out” for questions that don't apply Let’s discuss ways to improve the quality of surveys. Use simple language: Make your questions easy to understand by using simple, neutral language. Your goal is to write questions that recipients with a variety of backgrounds can easily understand. For example, the question “what is the frequency of your calls to technical support in the last month?” is better understood if it is written as, “About how many times in the last month have you called technical support?” Make the survey short: refer to the goals of your assessment when writing questions and only include questions that are essential to reaching those goals. If you cannot think of a way to actually use the answer to a question, it probably should be eliminated. Avoid double negatives: respondents can easily be confused deciphering the meaning of a question that uses two negative words. Avoid open-ended questions: an open-ended question is a written response. For example: “explain why your find your job difficult”. Too many written response questions, reduces the quality of the responses and are difficult to analyze. You may include a space at the end of the survey for additional, optional comments. Don't write leading questions: leading questions prompts the respondent to answer in a particular way. For example, look at the question “do you think that the new EHR system manages patients as well as the old one?” A better question would be, “how do you feel about the new EHR system compared to the old one?” Avoid double-barreled questions. Make sure you are asking only one question at a time. Here's an example of one a double-barreled question: “do technical support personnel respond in a timely and efficient way?” In this question it is not clear if the question is asking about timeliness or efficiency. By asking two different questions, you will get a much more accurate answer. Offer an “out” for questions that don't apply. When assessing a diverse audience there will be some questions that simply do not apply to everyone, or some respondents can't or won't answer certain questions because they aren't really sure how they want to respond. Include “Does Not Apply” or “Don't Know” options in these situations. Health IT Workforce Curriculum Version 3.0/Spring 2012 Training & Instructional Design Needs Analysis Unit 2b

Interviews Formal or informal discussion with an individual to gather in-depth information on specific topics When want to fully understand someone's impressions or experiences or learn more about their answers to questionnaires Description and Purpose Interviews are formal or informal discussions with an individual to gather in depth information on specific topics. They can be used when you want to fully understand someone's impressions or experiences, or to get more in depth answers to a pre-administered survey. Health IT Workforce Curriculum Version 3.0/Spring 2012 Training & Instructional Design Needs Analysis Unit 2b

Interviews Advantages Can get in-depth information Questions can be flexible and be tailored to each person Can get spontaneous responses Can get honest opinions, individual perspective Can get to questions that people are uncomfortable answering in a group Can get ideas or opinions that never occurred to you Can be done in a setting familiar to participant Disadvantages Time consuming to administer Interviewers needed and can be costly Interviewer can bias client's responses Not efficient for reaching a large number of people Not useful if trying to get answers to specific questions from people or get consistent answers With interviews you are likely to get ideas or opinions that never occurred to you. Here you have the opportunity to tailor questions to each person, and get spontaneous, honest opinions from an individual perspective. This is a good alternative to the focus group for persons who are uncomfortable answering in a group, or prefer to be interviewed in a familiar setting. However, interviews are not efficient for reaching a large number of people, and are not useful if you are trying to get answers to specific questions. In addition, they are time-consuming to administer and it may be costly to hire and train interviewers. There is always the risk of the interviewer biasing the interviewee’s responses, often inadvertently. There is also the well-documented phenomenon of subjects offering socially desirable answers and trying to please the interviewer. Health IT Workforce Curriculum Version 3.0/Spring 2012 Training & Instructional Design Needs Analysis Unit 2b

Interview Guide Types of Questions Description Experience Questions Deals with actions or experiences Opinion Questions Deals with opinions or beliefs Feeling Questions Deals with emotions, feelings, like happiness, fear, anxiety, and confidence Knowledge Questions Deals with factual information, not opinion or feeling Sensory Questions Deals with what respondents see, smell, hear, touch, or taste Background Questions Standard questions that describe the respondent's background Interview questions fall into a number of categories: Experience questions that deal with actions or experiences Opinion questions that deal with opinions or beliefs Feeling questions that deal with deal with emotions, feelings, like happiness, fear, anxiety, and confidence Knowledge questions for factual information, not opinion or feeling Sensory questions to assess for what respondents see, smell, hear, touch, or taste, and Background questions, which are standard questions that describe the respondents’ background Health IT Workforce Curriculum Version 3.0/Spring 2012 Training & Instructional Design Needs Analysis Unit 2b

Focus groups Small group discussions around a set of questions guided by a moderator To explore a new topic, get in-depth knowledge, ask questions about “how and why,” understand a gap in understanding between groups of people, or prepare for a survey Description and Purpose Focus groups are small group discussions facilitated by a moderator. They are useful when exploring a new topic, to get in-depth knowledge, to ask questions about “how and why”, to understand a gap in understanding between groups of people, to prepare for, or to follow-up a survey. Health IT Workforce Curriculum Version 3.0/Spring 2012 Training & Instructional Design Needs Analysis Unit 2b

Focus Groups Advantages Participants can build on each other’s comments Can be quicker than a survey Can be less costly Can reach more people than interviews Can hear a variety of ideas and opinions Participants can take advantage of the shared experience Disadvantages Need a good facilitator for safety and closure Difficult to schedule 6-8 people together A group or an individual can dominate the session Information gathered may not represent the larger community Smaller numbers reached Can be hard to analyze responses Focus groups are preferred in situations where it is important to hear a variety of ideas and opinions. In these groups, participants can build on each other’s comments, and you can take advantage of their shared experience. A focus group can be quicker and less costly than a survey, and will definitely reach more people than interviews. But, for focus groups to be successful, they require a good facilitator for safety and closure; someone who can effectively manage the conversation and prevent a group or individual from dominating the session. Compared to the other methods we discussed so far, focus groups are the most difficult to administer. In the average workplace it can be difficult to schedule 6-8 people together, so you can expect to reach smaller numbers, which means that the information gathered may not represent the larger community. Also, the data you collect will have to be transcribed and can be difficult to analyze. Health IT Workforce Curriculum Version 3.0/Spring 2012 Training & Instructional Design Needs Analysis Unit 2b

Observation Observer records an expert performing job natural or structured setting To gather accurate information about how a program actually operates, particularly about processes Description and Purpose In observational methods, the observer records an expert performing job natural or structured setting, and they are used to gather accurate information about how a program actually operates, particularly about processes. Health IT Workforce Curriculum Version 3.0/Spring 2012 Training & Instructional Design Needs Analysis Unit 2b

Observation Advantages Can view operations of a program as they are actually occurring Can adapt to events as they occur Disadvantages Can be difficult to train observers Can be difficult to interpret seen behaviors Can be complex to categorize observations Can influence behaviors of program participants Can be expensive This is the most effective method if you want to view the operations of a workplace as they are actually occurring. The observer is there to record behaviors and can adapt to events as they occur. But, there’re disadvantages to this method. Not only is this method expensive, it can also be difficult to train observers and to interpret seen behaviors, as it can be complex to categorize observations. And, being observed can influence the behavior of participants. Health IT Workforce Curriculum Version 3.0/Spring 2012 Training & Instructional Design Needs Analysis Unit 2b

Document Review When want impression of how program operates without interrupting the program Analyze budget documents, quality control documents, goal statements, evaluation reports, scheduling and staffing reports, or other documents for existing problems, such as testimonials from other organizations Description and Purpose The document review option is best used when you want the impression of how a program operates without interrupting the program. You can analyze budget documents, quality control documents, goal statements, evaluation reports, scheduling and staffing reports, or other documents for existing problems, such as testimonials from other organizations. Health IT Workforce Curriculum Version 3.0/Spring 2012 Training & Instructional Design Needs Analysis Unit 2b

Document Review Advantages Can get comprehensive and historical information Doesn't interrupt program or client's routine in program Information already exists Disadvantages Often takes a lot of time Information may be incomplete Need to be clear about what looking for Not a flexible means to get data Data restricted to what already exists You can use this method if you can get comprehensive and historical information that already exists. Also, this is the only method that doesn't interrupt an employee’s routine. But, this process can take a lot of time, especially if you are not clear about what you are looking for. This is also not a flexible means to get data, as you are limited to what already exists and information may be incomplete. Health IT Workforce Curriculum Version 3.0/Spring 2012 Training & Instructional Design Needs Analysis Unit 2b

Pilot Test Your Instrument
Identify and eliminate problems that might occur Keep practice exercises realistic Try out survey or interview tool on a few persons Pilot test if possible Make changes prior to actual data collection If changes are drastic, another pilot test again and/or re-train data collectors Pilot test results It is good practice to pilot your data collection methods and instruments. By doing a practice data collection exercise, you can identify and eliminate problems that might occur. Keep your practice exercises as realistic as possible. For example, you will try out your survey or interview on a few persons from the larger group you plan to assess. If your population is small or you have little time, a pilot test may not be feasible. If the pilot test results indicate that changes need to be made to data collection approaches or instruments, these should be made prior to actual data collection. If changes are drastic, another pilot test of instruments and/or another practice or training session with data collectors may be appropriate. Health IT Workforce Curriculum Version 3.0/Spring 2012 Training & Instructional Design Needs Analysis Unit 2b

Implement Your Instrument
Here are some guidelines to keep in mind during implementation Keep survey instruments short, one or two pages if possible Provide directions on how to return the survey. Prepare online surveys is possible Always let people how you will use the data Always find a way to report the information back to the participants Offer incentives, such as like lunch for focus groups and interviews Here are some guidelines to keep in mind during implementation: Keep survey instruments short, one or two pages if possible. Provide directions on how to return the survey. Prepare online surveys if possible. Always let people know how you will use the data. Always find a way to report the information back to the participants. This will not only improve their cooperation, but may improve your data (e.g., getting feedback on observations and other qualitative data to validate it). Offer incentives, such as lunch for focus groups and interviews. Health IT Workforce Curriculum Version 3.0/Spring 2012 Training & Instructional Design Needs Analysis Unit 2b

Implement Your Instrument
Confidentiality should always be foremost in your mind Inform respondents about your confidentiality guidelines when you begin data collection Make surveys anonymous Assign fictitious names to respondents or provide answers in categories rather than by respondent Confidentiality should always be foremost in your mind. Always inform respondents about your confidentiality guidelines when you begin data collection, and always provide the confidentiality you promise. Make surveys anonymous. When analyzing data collected from interviews, focus groups, or observations, assign fictitious names to respondents or provide answers in categories rather than by respondent. Health IT Workforce Curriculum Version 3.0/Spring 2012 Training & Instructional Design Needs Analysis Unit 2b

Case Example Meaningful Use Clinical Quality Measure: The percentage of patients 18 years of age and older who were current smokers or tobacco users, who were seen by a practitioner during the measurement year and who received advice to quit smoking or tobacco use or whose practitioner recommended or discussed smoking or tobacco use cessation medications, methods or strategies. One of the Meaningful Use Clinical Quality Measures is: The percentage of patients 18 years of age and older who were current smokers or tobacco users, who were seen by a practitioner during the measurement year and who received advice to quit smoking or tobacco use or whose practitioner recommended or discussed smoking or tobacco use cessation medications, methods or strategies. For this case we will assume that the EHR has been properly configured to collect the required data and document the clinical interaction, as well as produce reports so individual clinicians can attest to this measure. The Analysis Phase outcome, of the instructional system design model, is defined by the Meaningful Use Measure, but you need to collect some additional data for the analysis. You could observe what takes place now in a clinical interaction between the patients and the doctors. Conducting a few small focus groups to collect information about how the patient/doctor clinical interview would be an effective data collection method too. From your data collection you should be able to define what procedures the doctor needs to do and what data she needs to collect in order to achieve the outcome or objective. This information will then trigger you to think about appropriate activities or learning experiences that will help the doctors learn and change their practice pattern. Remember that knowledge is not sufficient to change ingrained practice patterns and behaviors. You must also understand what will motivate the learner to successfully perform and keep doing the new clinical procedure . This should also be part of your data collection in the Assessment Phase. Use the assessment information to define the business need ( the Meaningful Use Criteria), the job performance needs (this new clinical procedures and protocols), the training needs (skills, knowledge and attitudes), and the individual needs (what will motivate the learner to implement the change). Health IT Workforce Curriculum Version 3.0/Spring 2012 Training & Instructional Design Needs Analysis Unit 2b

Summary Choose the method that is most appropriate for your audience and the type of information to be collected. Ultimately, the decision of data collection method will take into account the validity of the responses collected, balanced with budget and efficiency concerns. To summarize, choose the method that is most appropriate for your audience and the type of information to be collected. Ultimately, the decision of data collection method will take into account the validity of the responses collected, balanced with budget and efficiency concerns. Health IT Workforce Curriculum Version 3.0/Spring 2012 Training & Instructional Design Needs Analysis Unit 2b

References Clark DR. Bloom’s Taxonomy [monograph on the Internet]. Big Dog & Little Dog’s Performance Juxtaposition; c2004 [cited 2010 Jun 21]. Available from: Clark DR. Instructional System Design (ISD). Big Dog & Little Dog’s Performance Juxtaposition; c2004 [cited 2010 Jun 21]. Available from: Carkhuff RR, Fisher SG. Instructional systems design: volumes I & II. Amherst, MA: Human Resource Development Press; 1984. Carliner S. Training Design. Danvers, MA: American Society for Training and Development; 2003. Gagne RM, Wager WW, & Golas K. Principles of Instructional Design (5th ed.). California: Wadsworth Publishing, 2004. Reigeluth CM. Instructional-design Theories and Models: A new paradigm of instructional theory. New Jersey: Lawrence Erlbaum Associates, Inc, 1999. No audio Health IT Workforce Curriculum Version 3.0/Spring 2012 Training & Instructional Design Needs Analysis Unit 2b

Training & Instructional Design

Similar presentations

Presentation on theme: "Training & Instructional Design"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Training & Instructional Design

Similar presentations

Presentation on theme: "Training & Instructional Design"— Presentation transcript:

Similar presentations

About project

Feedback