Chapter 8 VALIDITY AND RELIABILITY Fraenkel & Wallen (2006). How to design and evaluate research in education (6th ed.). Boston: McGraw-Hill. Prof. Dr. Sabri KOÇ Spring 2010 Spring 2010 Prof. Dr. Sabri Koç

The Importance of Valid Instruments The quality of instruments used in research is very important because conclusions drawn are based on the information obtained by these instruments. Researchers follow certain procedures to make sure that the inferences they draw based on the data collected are valid and reliable. Researchers should keep in mind these two terms, validity and reliability, in preparing data collection instruments. Spring 2010 Prof. Dr. Sabri Koç

VALIDITY & RELIABILITY Validity refers to the appropriateness, meaningfulness, correctness, and usefulness of the inferences a researcher makes. Reliability refers to the consistency of scores or answers from one administration of an instrument to another. Spring 2010 Prof. Dr. Sabri Koç

VALIDITY Researchers want the information obtained from participants through an instrument to serve the purposes of the study. Validity refers to the degree to which evidence supports any inferences the researcher makes based on the data collected using a particular instrument. Spring 2010 Prof. Dr. Sabri Koç

VALIDITY Validation is the process of collecting and analyzing evidence to support such inferences. A meaningful inference is one that says something about the meaning of the information obtained through an instrument. For example: What does a high score on a particular test mean? Spring 2010 Prof. Dr. Sabri Koç

VALIDITY In relation to the validity the crucial question is: Do the results of the assessment provide useful information about the topic or variable being measured? What kinds of evidence might a researcher collect? Researchers collect three main types of evidence: Spring 2010 Prof. Dr. Sabri Koç

VALIDITY Content-related evidence of validity, Criterion-related evidence of validity, and Construct-related evidence of validity. Spring 2010 Prof. Dr. Sabri Koç

VALIDITY Content-related evidence of validity: Content and format of the instrument How appropriate is the content? How comprehensive is the content? Does the content get at the intended variable? How adequately does the sample of items or questions represent the content to be assessed? Is the format of the instrument appropriate? Spring 2010 Prof. Dr. Sabri Koç

VALIDITY Criterion-related evidence of validity: The relationship between the scores obtained by an instrument and the scores obtained by another instrument (a criterion). How strong is this relationship? How well do such scores estimate the present or future performance of a certain type? Spring 2010 Prof. Dr. Sabri Koç

VALIDITY Construct-related evidence of validity: The nature of the psychological construct or characteristic being measured by the instrument. How well does a measure of the construct explain the differences in the behavior of individuals or their performance on certain tasks? Spring 2010 Prof. Dr. Sabri Koç

VALIDITY Content-related evidence: Effects of a new math program on mathematics ability of fifth-graders: Adequacy of sampling: The nature of the psychological construct or characteristic being measured by the instrument. Format of the instrument: The clarity of printing, size of type, adequacy of work space, appropriateness of language, clarity of directions etc. Spring 2010 Prof. Dr. Sabri Koç

VALIDITY How can we obtain content-related evidence of validity? Have someone who knows enough about what is being measured to be a competent judge. In other words, the researcher should get one or two judges’ opinions about the content and format of the instrument to be applied before administering it. The researcher evaluates the feedback from the judges and makes necessary modifications in the instrument. Spring 2010 Prof. Dr. Sabri Koç

VALIDITY Two forms of criterion-related validity: Criterion-related evidence: Compare performance on one instrument with performance on some other, independent criterion. Academic ability scores of the students on the instrument compared with students’ grade point averages. High scores on the instrument will correspond to high grade point averages. Two forms of criterion-related validity: Predictive validity: Student scores on a science aptitude test administered at the beginning of the semester are compared with the end-of-the-semester grades. Concurrent validity: Instrument data and criterion data are collected at nearly the same times and the results are compared to obtain evidence of concurrent validity. Spring 2010 Prof. Dr. Sabri Koç

VALIDITY In both types of criterion-related validity the degree of relationship existing between the scores of individuals is called the correlation coefficient and symbolized by the letter r. All correlation coefficients fall between +1.00 and -1.00. An r of .00 indicates that no relationship exists. We will deal with this topic in detail in Chapter 10 Descriptive Statistics. Spring 2010 Prof. Dr. Sabri Koç

VALIDITY When a correlation coefficient is used to describe the relationship between a set of scores obtained by the same group of individuals on a particular instrument (the predictor) and their scores on some criterion measure (the criterion), it is called a validity coefficient. Expectancy table: See Table 8.1, p. 156 Spring 2010 Prof. Dr. Sabri Koç

VALIDITY Construct-related evidence: In obtaining construct-related evidence of validity, there are three steps involved. the variable being measured is clearly defined hypotheses, based on a theory underlying the variable, are formed about how people who possess a lot versus a little of the variable will behave in a particular situation, and the hypotheses are tested both logically and empirically. Spring 2010 Prof. Dr. Sabri Koç

VALIDITY Construct validation involves a wide variety of procedures and many types of evidence, including both content-related and construct-related evidence. Spring 2010 Prof. Dr. Sabri Koç

RELIABILITY Reliability refers to the consistency of scores obtained from one administration of an instrument to another and from one set of items to another. If the test is reliable, we would expect a student who receives a high score on the test for measuring typing ability at first instance, to receive a high score the next time he takes the test. The scores may not be identical, but they should be close. Spring 2010 Prof. Dr. Sabri Koç

RELIABILITY The scores obtained from an instrument can be quite reliable, but not valid. The test on the Constitution of the US versus success in the physical education. If the data are unreliable, they cannot lead to valid inferences. (See Figure 8.2, p. 158) Spring 2010 Prof. Dr. Sabri Koç

ERRORS OF MEASUREMENT Whenever people take the same test twice, they will seldom perform exactly the same, that is, their scores or answers will not be identical. It is inevitable due to a variety of factors such as motivation, energy, anxiety, a different testing situation etc. Such factors result in errors of measurement. Because errors of measurement are always present, variation in reliability measurements is estimated. Such estimates can be expressed as another application of the correlation coefficient known as a reliability coefficient. Spring 2010 Prof. Dr. Sabri Koç

Validity & Reliability Coefficients A validity coefficient expresses the relationship between scores of the same individuals on two different instruments. A reliability coefficient expresses the relationship between the scores of the same individuals on the same instrument at two different times or between two parts of the same instrument. Reliability coefficients must range from .00 to 1.00, that is, with no negative values. Spring 2010 Prof. Dr. Sabri Koç

Validity & Reliability Coefficients The test-retest method involves administering the same test to the same group after a certain time has elapsed. A reliability coefficient is then calculated to indicate the relationship between the two sets of scores obtained. For most educational research, stability of scores over a two- to three-month period is usually viewed as sufficient evidence of test-retest reliability. Spring 2010 Prof. Dr. Sabri Koç

Validity & Reliability Coefficients Equivalent-forms method When the equivalent-forms method is used, two different but equivalent (parallel or alternate) forms of an instrument are administered to the same group of individuals during the same time period. A high coefficient would indicate strong evidence of reliability – that two forms are measuring the same thing. Spring 2010 Prof. Dr. Sabri Koç

INTERNAL CONSISTENCY METHODS The methods we have seen so far require two administrations or testing sessions. There are several internal-consistency methods of estimating reliability that require only a single administration of an instrument. Split-half procedure: This procedure involves scoring two halves (odd items versus even items) of a test separately for each person and then calculating a correlation coefficient for the two sets of scores. The coefficient indicates the degree to which two halves of the test provide the same results and hence describes the internal consistency of the test. In this case, the Spearman-Brown prophecy formula is used to calculate the reliability coefficient. Spring 2010 Prof. Dr. Sabri Koç

INTERNAL CONSISTENCY METHODS Reliability of 2 x reliability for ½ test scores on r = ------------------------------ total test 1 + reliability for ½ test Reliability of a test can be increased by adding more items similar to the original ones. Kuder-Richardson Approaches: The most frequently used method for determining internal consistency is the Kuder-Richardson approach, particularly formulas KR20 and KR21. KR21 requires (1) the number of items on the test, (2) the mean, and (3) the standard deviation if the items on the test are of equal difficulty. (See KR21 formula in your textbook.) Spring 2010 Prof. Dr. Sabri Koç

INTERNAL CONSISTENCY METHODS Alpha Coefficient – Another check on the internal consistency of an instrument is to calculate an alpha coefficient frequently called Cronbach alpha symbolized as . We will deal with this formula later. Spring 2010 Prof. Dr. Sabri Koç

INTERNAL VALIDITY When a study has internal validity, it means that any relationship observed between two or more variables should be unambiguous as to what it means rather than being due to “something else” – (alternative hypothesis). Spring 2010 Prof. Dr. Sabri Koç

INTERNAL VALIDITY Please read this chapter on your own, which is about confounding variables (erroneous variables, “something else” – alternative hypothesis) that are threats to internal validity of your research. Threats to Internal Validity Subject characteristics threat Maturity threat Location threat Instrumentation threat Testing threat History threat Maturation threat Subject attitude threat (Hawthorne effect) Regression threat Implementation threat Spring 2010 Prof. Dr. Sabri Koç

THANK YOU ALL FOR YOUR ATTENTION AND PATIENCE! Spring 2010 Prof. Dr. Sabri Koç