Methodology Tips for Constructing Instruments

Matching methods to research paradigm MethodQuantitativeQualitative Written Instrument Standardized Instrument (validity and reliability established) Constructed survey or interview guide (Use some questions from other surveys; construct new questions). May determine validity and reliability after administration No predeveloped questions in survey or interview guide Predeveloped questions – may be administered in no established order; questions may be added depending on circumstances. Predeveloped questions – established order ObservationStructured observation – predetermined items are counted in terms of frequency, duration, occurrence, permanent product, etc. Open-ended observations – trying to identify themes and patters Content analysis Structured counting of a pre- determined phenomena in terms of frequency, etc. Open-ended analysis; themes and patterns are identified.

Quantitative/Descriptive Studies involve specific Research Designs Designs structure how the interviews are conducted. Allow different groups and time periods to be compared. Minimize threats to internal and external validity.

Components of Designs Sampling (Random; Other probability sampling or nonprobability – such as convenience, purposive, or snowball) Groups – one group; two, or multiple group comparisons. Time intervals – pre and post tests or multiple observations.

Common design types Experimental Design (pre-post; random sample, two or more groups compared. Quasi Experimental with nonequivalent comparisons groups- pre and post. Quasi Experiments with post-test only. One group - pre and post test only. One group – multiple time periods measured (time series or longitudinal studies) One group – one measure – usually called cross-sectional survey.

Random designs help ensure internal/external validity Internal validity – attribute of people included or way the research was conducted may limit our ability to establish a real connection between the independent and dependent variables. External validity – can we generalize results to other people, settings, and times. (Note for some types of studies – qualitative research and needs assessments and some types of evaluations – we do not need generalizability. We are only interested in finding things out about a particular situation).

Exceptions to Experimental/Quasi- experimental Designs: One group studies Case studies (combination of methods looking at one individual, group, family, organization, or community) Survey research Consumer satisfaction surveys Needs Assessments

Use these alternative methods For descriptive studies When you do not have an intervention When you want to look at a specific situation or group and you do not intend to generalize. When it would be difficult or unethical to create an experimental or control group. When you do not have time or money to conduct an experiment.

Threats to Internal Validity Include: History Maturation Testing Instrumentation Regression to the mean Selection Interaction between Selection and the other threats to Internal Validity.

Qualitative Research Usually one group Comparisons may be made after data analysis – decision to compare made based on obvious patterns in the data. Sampling – nonprobability – purposive, convenience, snowball Uses observation, interviews, and content analysis.

Focus groups may also be used to conduct qualitative research. Focus groups: Are used to assess people’s perceptions about programs or services. Are used to generate recommendations for changing programs and services. Are often used in conjunction with needs assessments or to generate ideas or explore themes that can be used in descriptive studies. Are used to answer questions about how or why programs work. May be used in conjunction with other research methods. Include six to eight participants who are selected using purposive sampling methods. Utilize a semi-structured interview guide with 6-8 questions. Require that a facilitator ask each of the questions and solicit responses from members. Are used to generate a common response or a consensus from group members.

In any study: A mixture of methods may be used. Researchers may seek to increase reliability and validity by using both quantitative and qualitative approaches. This is called triangulation! Methods used should match the research question: a)What happens/what is the outcome? quantitative) b)Why does something happen? How does it happen? (qualitative) c)A description of what exists (Descriptive)

Descriptive studies Generally involve surveys, standardized interviews, or the use of pre-existing (secondary data). Include needs assessments. Can include both qualitative and quantitative data collection. In some cases, one instrument can include a mix of qualitative and quantitative data collection methods.

Survey and Interview Construction Tips Only measure one concept per question. Don’t write the question to solicit a specific response Response categories (if used) should be mutually exclusive and exhaustive Questions should be tactful and should not invoke a social stigma. May be filter questions – respondents can be directed to successive questions based on one response. Demographic questions are generally put at the beginning or the end of the instrument. Be aware that respondents do not like to answer some types of questions (age, income, organization budgets, etc.)

Example of closed-ended (response category) questions 1.What is your gender 1.Male2.Female 2.What is your ethnic identification? 1.African American 2.American Indian 3.Asian/Pacific Islander 4.European American 5.Mexican American/Latino 6.Multicultural or multi-racial 7.Other ____________________

More examples 3. What is your age? 1.18 or under – 64 5.Over 65 4.How satisfied are you with Social Work 292? 1.Very satisfied 2.Satisfied 3.Neutral 4.Unsatisfied 5.Very Unsatisfied

Sample question: How would you assess your own practice ability in the following areas: HighMediumLow Write Case Notes 123 Write Reports123 Use Computers 123 Apply Theories 123 Evaluate Practice 123

SAMPLE QUALITATIVE INTERVIEW QUESTIONS Interview Protocol: Perspectives on Deafness 1.1. Can you describe how you first became aware of your deafness 2.2. How do you see yourself today, in terms of your deafness? 3.3. What does your deafness mean to you? 4.4. Can you describe any particularly difficult or traumatic experiences in your life related to your deafness? 5.5. Can you describe how you fit into deaf culture? 6.6. To what extent do you consider yourself active in both the deaf world and the hearing world?

Each type of question corresponds to a level of measurement. These levels of measurement will determine what type of data analysis we use: All qualitative research is nominal. All categorical data in which there is no difference in value among the categories is nominal. All categorical data in which there is an implied ranking (for example, high-medium-low) is ordinal. Any question that asks respondents to rank order responses is also ordinal. Any question in which the response is a number or can be interpreted as a number with equal values among the data points is ratio.

For example, ratio data can include: What is your age? _______ Five point likert scales (1=Very likely, 2 = likely, etc. Usually likert scales are illustrated on the survey instrument as Respondents are asked to circle the best response. Scores on standardized instruments are also interpreted as ratio data.