Elspeth Slayter, Ph.D. Assistant Professor, Salem State University Lecture notes on threats to validity, threats to trustworthiness and the optimization of rigor in social work research

Administrative matters/Check-in Tools for critical thinking in “research- y language” Qualitative work: Threats to trustworthiness; Optimization of rigor Quantitative work: Threats to validity

QUALITATIVEQUANTITATIVE Concerned with how people think and feel about the topics of concern to the research Gather broader, more in-depth information from fewer respondents (micro-analysis) Open questions for greater depth and personal detail Use a structured survey instrument that asks all respondents the same questions in the same order to allow for statistical analysis Gather a narrow amount of information from a large number of respondents (macro- analysis Closed questions for quantification, can be coded and processed quickly

Not “opposite” of QN – a different way to answer different questions Different underlying assumptions about how individual/group behavior is best studied Reflects a systematic approach to the conduct of research

Rejection of threats to validity vs. evaluative standards, strategies for rigor Flexibility of design - need for transparency As subjectivity is embraced – need to document Careful reporting on implementation of approaches Note-taking, auditing, memo-ing REFLEXIVITY – self-awareness in process

Threats to trustworthiness Reactivity: Distorting effects of researcher’s presence on respondent beliefs/behaviors Researcher biases: Observations/Interpretations clouded by researcher’s worldview Respondent biases: Social desirability response to withholding information

1. Member Checking: Empowerment of respondents, keeps you honest 2. Negative Case Analysis: Testing and challenging interpretations to date 1. Fear management paper example 2. Searching for disconfirming evidence 3. Audit Trail: Replicability, accountable for interpretive decisions

4. Triangulation: Theory, method, observer, data 5. Prolonged Engagement: Goal is saturation 6. Peer Debriefing and Support (PDS) 1. Keeping us honest in interpretation 2. Emperor’s new clothes check

Interviews followed by memos Individual Listening to tapes Reading of transcripts Coding of memos, transcripts for themes Team Comparison of each transcript Write up of themes identified Member checking Review of transcripts, write-up

1. How can this study be generalizable with low n (sample size)? 2. Where are your hypotheses? 3. Why isn’t this study more objective? 4. Will attrition be a problem with so few in the study? 5. Why can’t you be more forthcoming about what your findings will look like in advance? 6. How is this different from journalism?

Tools for critical thinking, names

 Must specify type  Refers to the accuracy and trustworthiness of: instruments (surveys, variables) data (and how it was gathered) findings

1) The utility of the device or method that measures the issue at hand 2) The collective judgment of the research community that a concept, measure and method are valid

 Are results of the study applicable to settings other than the one in which the study was conducted?  Can be addressed by using: Large population samples Many control or comparison groups Replicating the study in a different setting or with a different population

Why?

 What are the challenges to external validity in your project?  Who is your study generalizable to?

 Looking at how a concept was operationalized and deciding whether or not “on the face of it” the measurement makes sense  Generally based upon consensus by researchers Example: Measuring prevalence of child abuse in families through parent interviews

 Achieved when a measurement has the appropriate content for “getting at” all the issues present in a complex construct  Think “inner mechanics”  Examples: Socioeconomic status Quality of life Client satisfaction

 Close fit between the construct being measured and the actual observations made  D  o your questions (and therefore variables) adequately “get at” the  construct your are measuring?  Examples: Does the IQ test measure intelligence? Does income alone measure SES?

Content  Technical presence of all the theory-pieces of the measure in question  Does it have all of the parts?  Presence of the parts Construct  Overall ‘vibe’ about whether the measure really gets at what it intends to  Do the parts make sense as a whole?  Quality of the parts  Representativeness

 Mostly an issue in longitudinal research  Concerned with reducing or maximizing errors in research design in several ways  Must say which type of threat to internal validity

 History (something happens to effect study’s process)  Maturation (something naturally happens along the way)  Testing effect (people get used to the testing, questions)  Measurement issues/instrumentation error (questions, inter-rater reliability)  Regression to the mean (over time, people will score close to the average)

 Selection bias/differential selection of participants (can happen even with randomization)  Attrition/mortality (subjects leave the study one way or another)  Reactive effects of participants (behavioral response to being in treatment/control group) “Hawthorne effect”  Diffusion of treatment (control or comparison groups end up experiencing some of the treatment effect…)  Interaction effect (of any of the above)

 Critical consumer of research What the heck the terms mean! What the authors don’t say  Comment on potential threats in end-of- semester proposal (limitations) Quantiative – threats to validity Qualitative – threats to trustworthiness, optimization of rigor