CDIS 5400 Dr Brenda Louw 2010 Validity Issues in Research Design

Objectives  Demonstrate :  an understanding of the importance of internal and external validity issues in experimental research  use this knowledge to critically evaluate research articles for writing a literature review and for planning EBP

Overview  Introduction  Terminology  Internal validity  Credibility  External Validity  Transferability and dependability

Readings  Schiavetti, et al.,2011, Chapter 4 Pp

Introduction  Validity : accurate representation; soundness and strength  Major consideration: have factors that can have an effect on the validity of the data been accounted for and controlled ?  Therefore research designs should address possible threats to validity so that the results can be trusted

Evaluation of research design  Internal validity: concerns the degree to which the design meets two purposes, i.e., providing answers to research questions and controlling variability.  External validity: concerns the degree to which generalizations can be made outside of the scope of the study.

Terminology issues  Internal validity  External validity  Credibility  Transferability, dependability Quantitative research Qualitative research

Control of variability  Researcher and critical reader:  Is change in DV caused by the experimental treatment and not by factors that can mimic the effects of the treatment

Internal validity  The ability of a study to rule out competing explanations for the results other than the effect of the IV on the DV by using experimental control to compensate for confounding variables in a study the (Haynes & Johnson,2009)

Attributes of studies with High and Low degrees of Internal validity  Low internal validity  Weak design that does not fully control for extraneous variables  Vague definitions of variables  Results accounted for by alternative explanations other than the IV  Unsystematic  High internal validity  Strong design with strong control of extraneous variables  Strong definitions of variables  Results explained by effects of IV on the DV with alternative explanations limited by the design  Systematic, attention to detail

Threats to internal validity  History: has some event occurred between 1 st and 2 nd measurements in addition to the DV  Maturation: changes in Subjects themselves that cannot be controlled by the researcher  Reactive pretest: practice effects –exposure to any type of experience in the early part of the study may influence behavior later in the experiment any type of experience in the early part of the study may influence behavior later in the experiment

Continued…  Instrumentation:  Study is only as good as the DV  Types of instrumentation: Electronic devices  Rating scales, questionnaires, standardized tests  Sources of error Quality of the instrument, condition, calibration, interpretive skills

Continued…  Statistical regression:  Ss selected on basis of atypically low or high scores change on a subsequent test so that their scores are somewhat better/poorer than originally  Change not due to treatment but due to scores regressing to more typical mean score  Differential subject selection:  Selection of experimental and groups important  Differences in Ss in 2 groups may account for treatment effects rather than treatment itself

Continued…  Differential Ss selection  Attrition  Withdrawal of participants  IC allows for withdrawing  Problematic in follow-up studies  Common in survey research : 50% adequate for analysis and reporting

Credibility –Qualitative research  Qualitative designs assure internal validity by establishing credibility  Researcher justified in describing impact of one variable on another or conclusions that relationship is causal  Threats:  Researcher bias :researcher instrument of data collection employ reflexivity  Researcher reactivity: preconceived notions affect data interpretation Use cross comparisons

External validity  The generalizability of research results from the laboratory to the real world (Haynes & Johnson, 2009)

External validity  Can results from a sample of individuals be applied or generalized to the entire population from which the sample was selected  Generalization grows with cumulative research on a given topic  Limit until evidence indicates validity beyond confines of an individuals study e.g. systematic replication studies

Threats to external validity  Subject selection  Are Ss representative of the population to which researcher wants to generalize?  Especially important in between subject designs  To counteract: Develop specific selection criteria Use random assignment Use intentional matching of Ss Include summary of Ss characteristics

Transferability- Qualitative research  Used in qualitative research to describe to which extent findings are externally valid  Dependability : consistency with which the same finding can be observed in similar circumstances

Reliability  Extent to which a measuring instrument or procedure yields consistent or repeatable results  4 ways of establishing reliability : Inter observer reliability Test-retest reliability Parallel forms reliability Split-half reliability

Inter observer reliability  2 different examiners/observers use the procedure to test the same persons  E.g. Group of participants tested 1x by each SLP on PPVT-3 Obtain 2 scores for each participant Compute correlation coefficient to determine degree of reliability Strong correlation :different examiners obtained similar results e.g. high score for 1 st examiner and high score when tested by 2 nd examiner

Test-retest reliability  Participants tested 2 different times but by same examiner  Strong test-retest reliability : consistent results from 1x to the next when administered by the same person  May compute correlation coefficient for each participant  Important when studying treatment outcomes- change due to treatment and not measurement instability

Parallel forms reliability  Construct/select 2 different but equivalent forms of a measure, comparable test items  Conduct 1 st test on all participants, then alter 2 nd test on all participants  Each participant has 2 scores-determine correlation coefficient to determine the degree of reliability between the 2 measures

E.g.  EI study : developed Surveillance tool for Communication Development (STCD) ( Decker, Louw, Kritzinger, 2009) for 0-6 month old infant population  Tested 55 infants in a rural hospital in South Africa  0- 3 mos and 3-6 mos screening intervals STCD Rossetti Infant-Toddler Language Scale ( Rossetti, 1990)  Results: High sensitivity score for both intervals % and 96.25% Use of STCD validate for use in rural hospital

Split-half reliability l  Administer test 1x to participants  Then divide items into 2 equivalent forms and compare participants scores on each form  Problematic and depends on how test is constructed

Conclusion  Reliability is a matter of degree  Correlation coefficient provides quantitative information re the level of reliability  Level of reliability depends on how scores are to be used  High scores needed for making decisions re intervention  Lower scores needed when mean of group scores are used

Conclusion  Both degree of validity and reliability reflect the degree to which researcher may have error in measurements  Validity errors reflect bias in the instrument itself  Reliability errors reflect use of the instrument