1. VALIDITY AND RELIABILITY
© LOUIS COHEN, LAWRENCE MANION AND KEITH MORRISON
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

2. STRUCTURE OF THE CHAPTER
Defining validity
Validity in quantitative, qualitative and mixed methods research
Types of validity
Triangulation
Ensuring validity
Reliability
Reliability in quantitative and qualitative research
Validity and reliability in interviews, experiments, questionnaires, observations, tests, life histories and case studies
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

3. WHAT IS VALIDITY?
A demonstration that a particular instrument measures what it intends, purports or claims to measure.
A demonstration that an account accurately represents those features that it is intended to describe, represent, explain or theorise.
The extent to which interpretations of data are warranted by the theories and evidence used.
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

4. QUANTITATIVE RESEARCH
BASES OF VALIDITY IN
QUANTITATIVE RESEARCH
QUALITATIVE RESEARCH
Controllability
Natural
Isolation, control, manipulation of
variables
Thick description
Replicability
Uniqueness
Predictability
Emergence, unpredictability
Generalizability
Context-freedom
Context-boundedness
Fragmentation and atomization
Holism
Randomization of samples
Purposive sample/no sampling
Neutrality
Value-ladenness of observations
Objectivity
Confirmability
Observability
Observable and non-observable
meanings/ intentions
Inference
Description, inference, explanation
‘Etic’ research
‘Emic’ research
Observations
Meanings
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

5. BASES OF RELIABILITY IN QUANTITATIVE RESEARCH
QUALITATIVE RESEARCH
Reliability
Dependability
Demonstrability
Trustworthiness
Stability and replicability
Parallel forms
Context-freedom
Context-specificity
Objectivity
Authenticity
Coverage of domain
Comprehensiveness of situation
Verification of data and analysis
Honesty and candour
Answering research questions
Depth of response
Meaningfulness to the research
Meaningfulness to respondents
Parsimony
Richness
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

6. BASES OF RELIABILITY IN QUANTITATIVE RESEARCH
QUALITATIVE RESEARCH
Objectivity
Confirmability
Internal consistency
Credibility
Generalizability
Transferability
Inter-rater reliability
Triangulation
Accuracy and precision
Accuracy and comprehensiveness
Neutrality
Multiple interests represented
Consistency
Alternative forms (equivalence)
Split-half
Inter-item correlation
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

7. VALIDITY IN QUANTITATIVE AND QUALITATIVE RESEARCH
Validity in quantitative research often concerns: objectivity, generalizability, replicability, predictability, controllability, nomothetic statements.
Validity in qualitative research often concerns: honesty, richness, authenticity, depth, scope, subjectivity, strength of feeling, catching uniqueness, idiographic statements.
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

8. TYPES OF VALIDITY Catalytic Concurrent Consequential Construct Content
Criterion-related
Convergent and discriminant
Cross-cultural
Cultural validity
Descriptive
Ecological
Evaluative
External
Face
Internal
Interpretive
Jury
Predictive
Systemic
Theoretical
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

9. VALIDITY IN QUANTITATIVE RESEARCH
Concurrent
Construct
Content
Criterion-related
Convergent and discriminant
Cross-cultural
Evaluative
External
Face
Internal
Jury
Predictive
Theoretical
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

10. VALIDITY IN QUALITATIVE RESEARCH
Credibility
The truth value (replacing the quantitative concepts of internal validity)
Transferability
Generalizability (replacing the quantitative concept of external validity)
Dependability
Consistency (replacing the quantitative concept of reliability)
Confirmability
Neutrality (replacing the quantitative concept of objectivity)
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

11. MAXWELL’S VALIDITY IN QUALITATIVE RESEARCH (MAXWELL)
Descriptive validity
the factual accuracy of the account, that it is not made up, selective, or distorted; objectively factual) and credible
Interpretive validity
the ability of the research to catch the meaning, terms, interpretations, intentions that situations and events, i.e. data, have for the participants/ subjects themselves, in their terms
Theoretical validity
the theoretical constructions that the researcher brings to the research theoretical validity is the extent to which the research explains phenomena; construct validity
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

12. MAXWELL’S VALIDITY IN QUALITATIVE RESEARCH (MAXWELL)
Generalizability
the view that the theory generated may be useful in understanding other similar situations; generalizing within specific groups or communities, situations or circumstances and, beyond, to outsider communities, situations or circumstance
Evaluative validity
the application of an evaluative, judgmental stance towards that which is being researched, rather than a descriptive, explanatory or interpretive framework.
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

13. VALIDITY IN MIXED METHODS RESEARCH
Representation
Legitimation
Sample integration
Inside–outside
Weakness minimization
Sequential
Conversion
Paradigmatic mixing
Commensurability
Multiple validities
Political
Integration (of methods)
Integration
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

14. INTERNAL VALIDITY
Demonstration that the explanation of a particular event, issue or set of data which a piece of research provides can actually be sustained by the data and the research.
The findings must describe accurately the phenomena being researched.
Truth value and credibility of interpretations and conclusions.
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

15. TESTING THREATS TO VALIDITY AND RELIABILITY MATURATION HISTORY
DIRECTION
OF CAUSALITY
INSTRUMENT-
ATION
THREATS TO
VALIDITY AND
RELIABILITY
TYPE I AND
TYPE II
ERRORS
EXPERIMENTAL
MORTALITY
OPERATIONAL-
IZATION
CONTAMIN-
ATION
REACTIVITY
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

16. INTERNAL VALIDITY IN QUALITATIVE RESEARCH
Confidence in the data
Authenticity of the data (the ability of the research to report a situation through the eyes of the participants)
Cogency of the data
Soundness of the research design
Plausibility of the data and interpretation
Credibility of the data
Auditability of the data
Dependability of the data
Confirmability of the data
Clarity on the kinds of claim made from the research (e.g. definitional, descriptive, explanatory, theory generative)
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

17. EXTERNAL VALIDITY
The degree to which the results can be general­ized to the wider population, cases, settings, times or situations, i.e. the transferability of the findings.
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

18. ESTABLISHING VALIDITY IN QUALITATIVE RESEARCH
Prolonged engagement in the field
Persistent observation
Triangulation
Leaving an audit trail
Respondent validation
Weighting the evidence (giving priority)
Checking for representativeness
Checking for researcher effects
Making contrast/comparisons
Theoretical sampling
Checking the meaning of outliers
Using extreme cases
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

19. ESTABLISHING VALIDITY IN QUALITATIVE RESEARCH
Ruling out spurious relations
Replicating a finding
Referential adequacy
Following up surprises
Structural relationships
Peer debriefing
Rich and thick description
Looking for possible sources of invalidity
Assessing rival explanations
Negative case analysis
Confirmatory data analysis
Effect sizes
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

20. THREATS TO VALIDITY IN QUANTITATIVE RESEARCH
History
Maturation
Statistical regression
Testing
Instrumentation
Selection bias
Experimental mortality
Instrument reactivity
Selection–maturation interaction
Type I and Type II errors
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

21. VALIDITY PROBLEMS IN CROSS-CULTURAL RESEARCH
Failure to operationalize elements of cultures.
Whose construction of ‘culture’ to adopt: ‘emic’/‘etic’.
False attribution of causality to cultural factors rather than non-cultural factors.
Directions of causality.
Ecological fallacy.
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

22. VALIDITY PROBLEMS IN CROSS-CULTURAL RESEARCH
Sampling and instrumentation.
Convergent and discriminant validity.
Response bias and preparation of participants.
Language problems.
Problems of equivalence (conceptual, psychological, meaning, instrument, understanding, significance, relevance, measurement, linguistic).
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

23. THREATS TO EXTERNAL VALIDITY IN QUANTITATIVE RESEARCH
Failure to describe independent variables explicitly.
Lack of representativeness of available and target populations.
Hawthorne effect.
Inadequate operationalizing of dependent variables.
Sensitization/reactivity to experimental/research conditions.
Interaction effects of extraneous factors and experimental/ research treatments.
Invalidity or unreliability of instruments.
Ecological validity.
Multiple treatment validity.  
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

24. THE HAWTHORNE EFFECT Date
Between 1927 and 1932 researchers carried out experiments at the Western Electric Company’s Hawthorne plant.
Purposes
To examine the effects of changes to working conditions on output of workers
Sample
Six women, chosen as average workers
Method
Women worked in a test room. Output measured under different conditions (e.g. no change → change to method of payment → introduce two rest periods → introduce six rest periods → changes in lighting conditions, early clocking-off, five-day working week → return to initial conditions)
Duration
15 weeks
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

25. THE HAWTHORNE EFFECT Results: Conclusion: Implications:
Output rose steadily during test period and after the test period.
Conclusion:
Output did not seem to depend on test conditions. Increased output seemed to be due to the fact that the people had been involved in the experiment itself, i.e. the act of research had affected the results. The results were a research of the research itself.
Implications:
The act of being involved in research itself affects the results.
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

26. THREATS TO EXTERNAL VALIDITY IN QUALITATIVE RESEARCH
Selection effects
Setting effects
History effects
Construct effects
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

27. ENSURING VALIDITY AT THE DESIGN STAGE
Choose an appropriate timescale
Ensure adequate resources for the research
Select appropriate methodology
Select appropriate instruments
Use an appropriate sample
Ensure reliability
Select appropriate foci
Avoid having biased researcher(s)
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

28. ENSURING VALIDITY AT THE DATA-COLLECTION STAGE
Reduce the Hawthorne effect
Minimize reactivity
Avoid dropout rates amongst respondents
Take steps to avoid non-return of questionnaires
Avoid too long or too short an interval between pre-tests and post-tests
Ensure inter-rater reliability
Match control and experimental groups
Ensure standardized procedures for gathering data
Build on the motivations of respondents
Tailor instruments to situational factors
Address researcher characteristics
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

29. ENSURING VALIDITY AT THE DATA-ANALYSIS STAGE
Use respondent validation
Avoid subjective interpretation of data
Reduce the halo effect
Use appropriate statistical treatments
Recognize extraneous factors which may affect data
Avoid poor coding of qualitative data
Avoid making inferences/generalizations beyond the data
Avoid equating correlations and causes
Avoid selective use of data
Avoid unfair aggregation of data
Avoid degrading the data
Avoid Type I and/or Type II errors
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

30. ENSURING VALIDITY AT THE REPORTING STAGE
Avoid using data selectively and unrepresentatively
Indicate the context and parameters of the research
Present the data without misrepresenting the message
Make claims which are sustainable by the data
Avoid inaccurate or wrong reporting of data
Ensure that the research questions are answered
Release research results neither too soon nor too late
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

31. RELIABILITY
Reliability is an umbrella term for dependability, consistency and replicability over time, over instruments and over groups of respondents.
Can we believe the results?
Can we trust the results?
Reliability is concerned with precision and accuracy.
Reliability is concerned with consistency.
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

32. RELIABILITY IN QUANTITATIVE AND QUALITATIVE RESEARCH
Reliability in quantitative research
consistency (stability), accuracy, predictability, equivalence, replicability, concurrence, descriptive and causal potential.
Reliability in qualitative research
accuracy, fairness, dependability, comprehensiveness, respondent validation, ‘checkability’, empathy, uniqueness, explanatory and descriptive potential, confirmability.
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

33. RELIABILITY IN QUANTITATIVE RESEARCH
Reliability as stability
Consistency over time and samples
Reliability as equivalence
Equivalent forms of same instrument;
Inter-rater reliability;
Reliability as internal consistency
Split half reliability (e.g. for test items)
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

34. TRIANGULATION Paradigms Methodologies Instruments Researchers Time
Location
Theories
Samples
Participants
Data
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

35. SPLIT-HALF RELIABILITY (Spearman-Brown)
r = the actual correlation between the two halves of the instrument;
Reliability = = = 0.919
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

36. RELIABILITY IN QUALITATIVE RESEARCH
Credibility
Neutrality
Confirmability
Dependability
Consistency
Applicability
Trustworthiness
Transferability
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

37. RELIABILITY AND REPLICATION IN QUALITATIVE RESEARCH
Repeat:
The status position of the researcher
The choice of informants/respondents
The social situations and conditions
The analytic constructs used
The methods of data collection and analysis
Address:
Stability of observations
Parallel forms
Inter-rater reliability
Respondent validation
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

38. IMPROVING RELIABILITY
Minimise external sources of variation
Standardise conditions under which measurement occurs
Improve researcher consistency
Broaden the sample of measurement questions by:
adding similar questions to the instrument;
increasing the number of researchers (triangulation);
increasing the number of occasions in an observational study.
Exclude extreme responses (outliers)
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors

39. RELIABILITY AND VALIDITY AT ALL STAGES
Design and methodology
Sampling
Instrumentation
Timing
Data collection
Data analysis
Data interpretation
Reporting
© 2018 Louis Cohen, Lawrence Manion and Keith Morrison; individual chapters, the contributors