RELIABILITY AND VALIDITY

Slides:



Advertisements
Similar presentations
Cross Cultural Research
Advertisements

Copyright © Allyn & Bacon (2007) Hypothesis Testing, Validity, and Threats to Validity Graziano and Raulin Research Methods: Chapter 8 This multimedia.
Experimental Research Designs
Culture and psychological knowledge: A Recap
Reliability, Validity, Trustworthiness If a research says it must be right, then it must be right,… right??
MSc Applied Psychology PYM403 Research Methods Validity and Reliability in Research.
SOWK 6003 Social Work Research Week 4 Research process, variables, hypothesis, and research designs By Dr. Paul Wong.
SOWK 6003 Social Work Research Week 4 Research process, variables, hypothesis, and research designs By Dr. Paul Wong.
© 2005 The McGraw-Hill Companies, Inc., All Rights Reserved. Chapter 5 Making Systematic Observations.
9 Quantitative Research Designs.
Research Methods in MIS
Classroom Assessment A Practical Guide for Educators by Craig A
Validity Lecture Overview Overview of the concept Different types of validity Threats to validity and strategies for handling them Examples of validity.
Formulating the research design
Basic Principles of Research Design
Chapter 8 Experimental Research
Experimental Design The Gold Standard?.
© 2013 Cengage Learning. Outline  Types of Cross-Cultural Research  Method validation studies  Indigenous cultural studies  Cross-cultural comparisons.
Experimental and Quasi-Experimental Designs
Qualitative Research.
Chapter 3 The Research Design. Research Design A research design is a plan of action for executing a research project, specifying The theory to be tested.
Day 6: Non-Experimental & Experimental Design
Research Methods in Psychology (Pp 1-31). Research Studies Pay particular attention to research studies cited throughout your textbook(s) as you prepare.
VALIDITY, RELIABILITY, and TRIANGULATED STRATEGIES
Bryman: Social Research Methods, 4 th edition What is a concept? Concepts are: Building blocks of theory Labels that we give to elements of the social.
Validity & Reliability Trustworthiness
Copyright © 2012 Wolters Kluwer Health | Lippincott Williams & Wilkins Chapter 14 Measurement and Data Quality.
V ALIDITY IN Q UALITATIVE R ESEARCH. V ALIDITY How accurate are the conclusions you make based on your data analysis? A matter of degree Non-reification.
Evaluating a Research Report
WELNS 670: Wellness Research Design Chapter 5: Planning Your Research Design.
Copyright ©2008 by Pearson Education, Inc. Pearson Prentice Hall Upper Saddle River, NJ Foundations of Nursing Research, 5e By Rose Marie Nieswiadomy.
Reliability and Validity Why is this so important and why is this so difficult?
The Basics of Experimentation Ch7 – Reliability and Validity.
The Nature of Quantitative Research
Slides to accompany Weathington, Cunningham & Pittenger (2010), Chapter 3: The Foundations of Research 1.
EDU 8603 Day 6. What do the following numbers mean?
Basic Qualitative Strategies Cap Peck, Research Question Beginning with less specific research questions Start with interests and a sense of the.
CDIS 5400 Dr Brenda Louw 2010 Validity Issues in Research Design.
1 Copyright © 2011 by Saunders, an imprint of Elsevier Inc. Chapter 8 Clarifying Quantitative Research Designs.
1 Lecture 3 Theory and Measurement: Causation, Validity and Reliability.
Issues in Validity and Reliability Conducting Educational Research Chapter 4 Presented by: Vanessa Colón.
META-ANALYSIS, RESEARCH SYNTHESES AND SYSTEMATIC REVIEWS © LOUIS COHEN, LAWRENCE MANION & KEITH MORRISON.
Copyright © Allyn & Bacon 2008 Intelligent Consumer Chapter 14 This multimedia product and its contents are protected under copyright law. The following.
 Descriptive Methods ◦ Observation ◦ Survey Research  Experimental Methods ◦ Independent Groups Designs ◦ Repeated Measures Designs ◦ Complex Designs.
Research Methodology and Methods of Social Inquiry Nov 8, 2011 Assessing Measurement Reliability & Validity.
Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins Chapter 17 Assessing Measurement Quality in Quantitative Studies.
Research Design ED 592A Fall Research Concepts 1. Quantitative vs. Qualitative & Mixed Methods 2. Sampling 3. Instrumentation 4. Validity and Reliability.
Experimental Research Methods in Language Learning Chapter 5 Validity in Experimental Research.
Experimental & Quasi-Experimental Designs Dr. Guerette.
Research Design Quantitative Study Design - B. Back to Class 9.
 Measuring Anything That Exists  Concepts as File Folders  Three Classes of Things That can be Measured (Kaplan, 1964) ▪ Direct Observables--Color of.
Reliability and Validity Themes in Psychology. Reliability Reliability of measurement instrument: the extent to which it gives consistent measurements.
Chapter Eight: Quantitative Methods
Measurement Experiment - effect of IV on DV. Independent Variable (2 or more levels) MANIPULATED a) situational - features in the environment b) task.
CJ490: Research Methods in Criminal Justice UNIT #4 SEMINAR Professor Jeffrey Hauck.
Reliability and Validity Why is this so important and why is this so difficult?
Can you hear me now? Keeping threats to validity from muffling assessment messages Maureen Donohue-Smith, Ph.D., RN Elmira College.
CRITICALLY APPRAISING EVIDENCE Lisa Broughton, PhD, RN, CCRN.
ESTABLISHING RELIABILITY AND VALIDITY OF RESEARCH TOOLS Prof. HCL Rawat Principal UCON,BFUHS Faridkot.
Issues in Evaluating Educational Research
Reliability and Validity in Research
Hypothesis Testing, Validity, and Threats to Validity
Ron Sterr Kim Sims Heather Cruz aka “The Carpool”
A Bit More About Qualitative Analysis
Chapter Eight: Quantitative Methods
Experimental Design.
RELIABILITY AND VALIDITY AT ALL STAGES
VALIDITY AND RELIABILITY
Experimental Design.
Meta-analysis, systematic reviews and research syntheses
Presentation transcript:

RELIABILITY AND VALIDITY © LOUIS COHEN, LAWRENCE MANION & KEITH MORRISON

STRUCTURE OF THE CHAPTER Defining validity Validity in quantitative research Validity in qualitative research Types of validity Triangulation Validity in mixed methods research Ensuring validity Reliability Reliability in quantitative research Reliability in qualitative research

STRUCTURE OF THE CHAPTER Validity and reliability in interviews Validity and reliability in experiments Validity and reliability in questionnaires Validity and reliability in observations Validity and reliability in tests Validity and reliability in life histories

QUANTITATIVE RESEARCH BASES OF VALIDITY IN QUANTITATIVE RESEARCH QUALITATIVE RESEARCH Controllability Natural Isolation, control, manipulation of Variables Thick description Replicability Uniqueness Predictability Emergence, unpredictability Generalizability Context-freedom Context-boundedness Fragmentation and atomization Holism Randomization of samples Purposive sample/no sampling Neutrality Value-ladenness of observations Objectivity Confirmability Observability Observable and non-observable meanings/ intentions Inference Description, inference, explanation ‘Etic’ research ‘Emic’ research Observations Meanings

BASES OF RELIABILITY IN QUANTITATIVE RESEARCH QUALITATIVE RESEARCH Reliability Dependability Demonstrability Trustworthiness Stability and replicability Parallel forms Context-freedom Context-specificity Objectivity Authenticity and confirmability Coverage of domain Comprehensiveness of situation Verification of data and analysis Honesty and candour Answering research questions Depth of response Meaningfulness to the research Meaningfulness to respondents Parsimony Richness Internal consistency Credibility Generalizability Transferability Inter-rater reliability & triangulation Inter-rater reliability and triangulation Accuracy and precision Accuracy and comprehensiveness Neutrality Multiple interests represented Consistency Alternative forms (equivalence) Split-half and inter-item correlation

VALIDITY IN QUANTITATIVE AND QUALITATIVE RESEARCH Validity in quantitative research often concerns: objectivity, generalizability, replicability, predictability, controllability, nomothetic statements. Validity in qualitative research often concerns: honesty, richness, authenticity, depth, scope, subjectivity, strength of feeling, catching uniqueness, idiographic statements.

TYPES OF VALIDITY Catalytic Concurrent Consequential Construct Content Criterion-related Convergent & discriminant Cross-cultural Cultural validity Descriptive Ecological Evaluative External Face Internal Interpretive Jury Predictive Systemic Theoretical

VALIDITY IN QUANTITATIVE RESEARCH Concurrent Construct Content Criterion-related Convergent & discriminant Cross-cultural Evaluative External Face Internal Jury Predictive Theoretical

VALIDITY IN MIXED METHODS RESEARCH Representation Legitimation Sample integration Inside-outside Weakness minimization Sequential Conversion Paradigmatic mixing Commensurability Multiple validities Political Integration (of methods)

TESTING THREATS TO VALIDITY AND RELIABILITY MATURATION HISTORY DIRECTION OF CAUSALITY INSTRUMENT- ATION THREATS TO VALIDITY AND RELIABILITY TYPE 1 AND TYPE 2 ERRORS EXPERIMENTAL MORTALITY OPERATIONAL- IZATION CONTAMIN- ATION REACTIVITY

ESTABLISHING VALIDITY IN QUALITATIVE RESEARCH Prolonged engagement in the field Persistent observation Triangulation Leaving an audit trail Respondent validation Weighting the evidence (giving priority) Checking for representativeness Checking for researcher effects Making contrast/comparisons Theoretical sampling Checking the meaning of outliers Using extreme cases

ESTABLISHING VALIDITY IN QUALITATIVE RESEARCH Ruling out spurious relations Replicating a finding Referential adequacy Following up surprises Structural relationships Peer debriefing Rich and thick description Looking for possible sources of invalidity Assessing rival explanations Negative case analysis Confirmatory data analysis Effect sizes

THREATS TO VALIDITY IN QUANTITATIVE RESEARCH History Maturation Statistical regression Testing Instrumentation Selection Bias Experimental mortality Instrument reactivity Selection-maturation interaction Type I and Type II errors

VALIDITY PROBLEMS IN CROSS-CULTURAL RESEARCH Failure to operationalize elements of cultures Whose construction of ‘culture’ to adopt: ‘emic’/‘etic’ False attribution of causality to cultural factors rather than non-cultural factors Directions of causality Ecological fallacy Sampling and instrumentation Convergent and discriminant validity Response bias and preparation of participants Language problems Problems of equivalence (conceptual, psychological, meaning, instrument, understanding, significance, relevance, measurement, linguistic)

THREATS TO EXTERNAL VALIDITY IN QUANTITATIVE RESEARCH Failure to describe independent variables explicitly Lack of representativeness of available and target populations Hawthorne effect Inadequate operationalizing of dependent variables Sensitization/reactivity to experimental/research conditions Interaction effects of extraneous factors and experimental/ research treatments Invalidity or unreliability of instruments Ecological validity Multiple treatment validity  

THE HAWTHORNE EFFECT Between 1927 and 1932 researchers carried out experiments at the Western Electric Company’s Hawthorne plant. Purposes: To examine the effects of changes of working conditions on output of workers Sample: Six women, chosen as average workers Method: Women worked in a test room. Output measured under different conditions (e.g. no change → change to method of payment → introduce two rest periods → introduce six rest periods → changes in lighting conditions, early clocking-off, five-day working week → return to initial conditions Duration: 15 weeks

THE HAWTHORNE EFFECT Results: Output rose steadily during test period and after the test period. Conclusion: Output did not seem to depend on test conditions. Increased output seemed to be due to the fact that the people had been involved in the experiment itself, i.e. the act of research had affected the results. The results were a research of the research itself. Implications: The act of being involved in research itself affects the results.

THREATS TO EXTERNAL VALIDITY IN QUALITATIVE RESEARCH Selection effects Setting effects History effects Construct effects

ENSURING VALIDITY AT THE DESIGN STAGE Choose an appropriate time scale; Ensure adequate resources for the research Select appropriate methodology Select appropriate instruments Use an appropriate sample Ensure reliability Select appropriate foci Avoid having biased researcher(s)

ENSURING VALIDITY AT THE DATA COLLECTION STAGE Reduce the Hawthorne effect Minimize reactivity Avoid drop-out rates amongst respondents Take steps to avoid non-return of questionnaires Avoid too long or too short an interval between pre-tests and post-tests Ensure inter-rater reliability Match control and experimental groups Ensure standardized procedures for gathering data Build on the motivations of respondents Tailor instruments to situational factors Address researcher characteristics

ENSURING VALIDITY AT THE DATA ANALYSIS STAGE Use respondent validation; Avoid subjective interpretation of data Reduce the halo effect Use appropriate statistical treatments Recognize extraneous factors which may affect data Avoid poor coding of qualitative data Avoid making inferences/generalizations beyond the data Avoid equating correlations and causes Avoid selective use of data Avoid unfair aggregation of data Avoid degrading the data; Avoid Type I and/or Type II errors

ENSURING VALIDITY AT THE DATA REPORTING STAGE Avoid using data selectively and unrepresentatively Indicate the context and parameters of the research Present the data without misrepresenting the message Make claims which are sustainable by the data Avoid inaccurate or wrong reporting of data Ensure that the research questions are answered Release research results neither too soon nor too late

RELIABILITY IN QUANTITATIVE AND QUALITATIVE RESEARCH Reliability in quantitative research: consistency (stability), accuracy, predictability, equivalence, replicability, concurrence, descriptive and causal potential. Reliability in qualitative research: accuracy, fairness, dependability, comprehensiveness, respondent validation, ‘checkability’, empathy, uniqueness, explanatory and descriptive potential, confirmability.

TYPES OF RELIABILITY IN QUANTITATIVE RESEARCH Reliability as stability: Consistency over time and samples; Reliability as equivalence: Equivalent forms of same instrument; Inter-rater reliability; Reliability as internal consistency: Split half reliability (e.g. for test items)

TRIANGULATION Methodologies Instruments Researchers Time Location Theories Samples Participants Data

SPLIT-HALF RELIABILITY (Spearman-Brown) r = the actual correlation between the two halves of the instrument (e.g. 0.85); Reliability = = = 0.919

RELIABILITY IN QUALITATIVE RESEARCH Credibility Neutrality Confirmability Dependability Consistency Applicability Trustworthiness Transferability

RELIABILITY AND REPLICATION IN QUALITATIVE RESEARCH Repeat: The status position of the researcher The choice of informants/respondents The social situations and conditions The analytic constructs used The methods of data collection and analysis Address: Stability of observations Parallel forms Inter-rater reliability Respondent validation

IMPROVING RELIABILITY Minimise external sources of variation; Standardise conditions under which measurement occurs; Improve researcher consistency; Broaden the sample of measurement questions by: adding similar questions to the instrument; increasing the number of researchers (triangulation); increasing the number of occasions in an observational study. Exclude extreme responses (outliers).

RELIABILITY AND VALIDITY AT ALL STAGES Design and methodology Sampling Instrumentation Timing Data collection Data analysis Data reporting