Ch 9 Internal and External Validity

Validity  The quality of the instruments used in the research study  Will the reader believe what they are readying  It’s not be the instrument that needs to be validated but how the instrument is to be used  Questions --- Is the data accurate? --- Is the data meaningful? --- Is the data creditable? --- Is the data useful? --- Is the data appropriate?

External validity  Can the researcher use what have been observed in the research to make global generalizations beyond the research itself?  Does the instrument measure what it is suppose to?  Is the research sample representative of the population?  Using evidence to support external validity  Does the study have sufficient controls to insure that the conclusions or inferences are truly warranted? 1. Face validity: looks like it measures a particular characteristic, concepts, or phenomena

External validity (Continued) 2. Content validity: measures the expected outcomes. Are the results similar to what others familiar with the study’s contents might contend? 3. Criterion validity: the relationship of scores on an instrument to other similar instruments 4. Construct validity: measures characteristics, concepts, theories, phenomenon that cannot be directly observed but inferred, such as to what is inferred in the review of literature

Threats to Internal Validity  The extent to which the research design, instrument, and the data collected allow the researcher to draw accurate conclusions about the relationship within the data.  The research has not been affected by any interference with the research environment that would give the results of the research the appearance of something else other than what the research originally intended.  Threats to internal validity 1. Subject characteristics: too much or too little representation of a characteristic: gender, age, race, beliefs, attitudes, intelligence, etc.

Threats to Internal Validity (Continued 1) 2. Mortality: Participants dropping out of the study 3. Location: Comfort, distance, resources, need consistency 4. Instrumentation: Consistent use of the instrument a. Instrument decay: nature of the instrument causes change in interpretation b. Data collector characteristics: may interpret things differently, need consistency

Threats to Internal Validity (Continued 2) c. Data collector bias: have the possibility of distorting information, need standards or ignorance 5. History: Unexpected evens occur, 9/11, 921earthquake, death in a family, accidents, storms, etc. 6. Maturation: Too much time taken for the completion of the instrument, the passage of time 7. Regression: Increased performance may occur due to practice or intervention, such as special ed assistance

Threats to Internal Validity (Continued 3) 8. Attitude: Special treatment of participants, special favors, control vs. experimental group 9. Implementation: Consistency among participants 10. Testing: Difference between pre-test & post-test, practice 11. Sources of bias: The assessment, the administration of the assessment, the interpretation of the assessment

Threats to Internal Validity (Continued 4) a. Assessment bias: Offensive questions Inaccurate and unfair penalties related to scores b. Bias in administration: Examiner Students Setting C. Bias in interpretation: Criterion referenced Norm referenced

Ways to overcome treats to internal validity  Prepare alternative strategies for the unexpected  Be consistent in the treatment and relationship with all participants  Use an appropriate research design that will assist the researcher to gather, collect, and analyze the data in the most efficient manner  Examining items for bias: Panel review Empirical analysis What is the preferred method to examine assessment bias?

Triangulation  Compare three forms of information for similar results 1. Observation 2. Interviews 3. Data gathering 4. Focus groups

Is the test appropriate to the population?  What is the composition of the test-taking population?  To what extent can the assessment be administered without encumbrance to all members of the population?  Is there a translated version, adapted version or accommodated version of the test?  Are there recommendations for alternative testing procedures?  Has the planned accommodation been assessed in terms of it’s impact on validity and reliability of test scores?

Conclusion  The more valid and reliable research instruments are the more likely one will draw the appropriate conclusions from the collected data and solve the research problem in a creditable fashion.