Reliability and Validity Chapter 4 Copyright ©2006 by Pearson Education, Inc. Upper Saddle River, New Jersey 07458 All rights reserved. Terry Overton Assessing Learners with Special Needs, 5e

Reliability-Having confidence in the consistency of the test results. Validity-Having confidence that the test is measuring what it is supposed to measure. Copyright ©2006 by Pearson Education, Inc. Upper Saddle River, New Jersey 07458 All rights reserved. Terry Overton Assessing Learners with Special Needs, 5e

Correlation- A statistical method of observing the degree of relationship between two sets of data or two sets of variables. Correlation coefficient -The numerical representation of the strength and direction of the relationship between two sets of variables or data (0 to 1.0; can be positive or negative). Copyright ©2006 by Pearson Education, Inc. Upper Saddle River, New Jersey 07458 All rights reserved. Terry Overton Assessing Learners with Special Needs, 5e

General Principles of Research Correlational Studies A positive correlation (+) means that as one variable increases, so does the other. A zero or near zero correlation means that the variables have no relationship

The direction of the line plotted on the scatter plot provides a clue about the relationship. If the relationship is positive, the direction of the line looks like this: Copyright ©2006 by Pearson Education, Inc. Upper Saddle River, New Jersey 07458 All rights reserved. Terry Overton Assessing Learners with Special Needs, 5e

General Principles of Research Correlational Studies The value of the correlation coefficient can range from –1.00 to +1.00. The higher the absolute value, the stronger the relationship is, regardless of the direction. A negative correlation (-) means that as one variable increases, the other decreases.

If the data represent a negative correlation, the direction of the line in the scatter plot looks like this: Copyright ©2006 by Pearson Education, Inc. Upper Saddle River, New Jersey 07458 All rights reserved. Terry Overton Assessing Learners with Special Needs, 5e

Concept Check The greater the score on a depression inventory, the lower the score on a memory test Negative

Concept Check Which relationship is stronger? +. 85 or -.90 or 1.2 - .90 +. 85 or -.90 or 1.2 .25 or - .60 or + .63 +.63

Figure 2.9 A strong correlation between depression and impaired sleep does not tell us whether depression interferes with sleep, poor sleep leads to depression, or whether another problem leads to both depression and sleep problems.

Positive Correlation * In investigating how data are related, it is important to determine if the two sets of data represent positive, negative, or no correlation. *In a positive correlation, when a student scores high on the first variable or test, the student will also score high on the second measure. *The data below illustrate a positive correlation: Student 1- 75 Student 2- 88 Student 3- 90 Student 4- 63 Student 1- 72 Student 2- 89 Student 3- 93 Student 4- 64 Set 2 Set 1 Copyright ©2006 by Pearson Education, Inc. Upper Saddle River, New Jersey 07458 All rights reserved. Terry Overton Assessing Learners with Special Needs, 5e

Negative Correlation In a negative correlation, when a student scores high on one variable or test, the student will score low on the other variable or test. The data below illustrate a negative correlation: Student 1- 88 Student 2- 99 Student 3- 56 Student 4- 97 Student 1- 32 Student 2- 45 Student 3- 15 Student 4- 12 Set 1 Set 2 Copyright ©2006 by Pearson Education, Inc. Upper Saddle River, New Jersey 07458 All rights reserved. Terry Overton Assessing Learners with Special Needs, 5e

Height and Temperament of 20 Men Correlation Height and Temperament of 20 Men 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 80 63 61 79 74 69 62 75 77 60 64 76 71 66 73 70 68 90 42 81 39 48 72 57 30 84 Subject Height in Inches Temperament

Scatterplot of Height and Temperament Correlation 95 90 85 80 75 70 65 60 55 50 45 40 35 30 25 Temperament scores Height in inches 55 60 65 70 75 80 85 Scatterplot of Height and Temperament

Pearson correlation coefficient r = the Pearson coefficient r measures the amount that the two variables (X and Y) vary together (i.e., covary) taking into account how much they vary apart Pearson’s r is the most common correlation coefficient; there are others.

Computing the Pearson correlation coefficient To put it another way: Or

Sum of Products of Deviations Measuring X and Y individually (the denominator): compute the sums of squares for each variable Measuring X and Y together: Sum of Products Definitional formula Computational formula n is the number of (X, Y) pairs

Correlation Coefficent: the equation for Pearson’s r: expanded form:

Example What is the correlation between study time and test score:

Calculating values to find the SS and SP:

Calculating SS and SP

Calculating r

Correlation Coefficient Interpretation Range Strength of Relationship 0.00 - 0.20 Very Low 0.20 - 0.40 Low 0.40 - 0.60 Moderate 0.60 - 0.80 High Moderate 0.80 - 1.00 Very High

Methods of Studying Reliability Test-retest reliability- A study that employs the readministration of a single instrument to check for consistency across time. Equivalent forms reliability-Consistency of a test using like forms that measure the same skill, domain, or trait; also known as alternate forms reliability. Copyright ©2006 by Pearson Education, Inc. Upper Saddle River, New Jersey 07458 All rights reserved. Terry Overton Assessing Learners with Special Needs, 5e

Methods of Studying Reliability, Continued Internal consistency-Methods to study the reliability across the items of the test. Examples include split-half reliability, K-R 20, and coefficient alpha. Split-half reliability-A method of measuring internal consistency by studying the reliability across items by comparing the data of the two halves of the test. Copyright ©2006 by Pearson Education, Inc. Upper Saddle River, New Jersey 07458 All rights reserved. Terry Overton Assessing Learners with Special Needs, 5e

Methods of Studying Reliability, Continued Interrater Reliability- The consistency of a test to measure a skill, trait, or domain across examiners. This type of reliability is most important when responses are subjective or open-ended. Reliability coefficients may vary across age and grade levels of a specific instrument. Copyright ©2006 by Pearson Education, Inc. Upper Saddle River, New Jersey 07458 All rights reserved. Terry Overton Assessing Learners with Special Needs, 5e

Methods of Studying Reliability, Continued Kuder-Richardson 20- (K-R 20)- A formula used to check consistency across items of an instrument that has items scored as 1 or 0 or right/wrong. Coefficient Alpha- A formula used to check the consistency across items of instruments with responses of varying credit. For example, items may be scored as 0,1,2,or 3 points. Copyright ©2006 by Pearson Education, Inc. Upper Saddle River, New Jersey 07458 All rights reserved. Terry Overton Assessing Learners with Special Needs, 5e

Standard Error of Measurement Each test score is made of two parts: true score and error. A student’s true score may only be estimated. The standard error of measurement is a method used to estimate the amount of error of a test. It represents the typical amount of error of any obtained score. The standard error of measurement is used to estimate a range of scores within which the student’s true scores exists. Obtained Score = True Score + Error Copyright ©2006 by Pearson Education, Inc. Upper Saddle River, New Jersey 07458 All rights reserved. Terry Overton Assessing Learners with Special Needs, 5e

Standard Error of Measurement, Continued The standard error of measurement is calculated using the following formula: SEM = SD 1-.r Where SEM = the standard error of measurement SD = the standard deviation of the norm group of scores obtained during the development of the instrument r = the reliability coefficient Copyright ©2006 by Pearson Education, Inc. Upper Saddle River, New Jersey 07458 All rights reserved. Terry Overton Assessing Learners with Special Needs, 5e

Example of Calculating SEM For a specific test, the standard deviation is 4. The reliability coefficient is .89. The SEM would be: 4 1- .89 .11 This represents the amount of error on this test instrument. 4 4 x .33 = 1.32 SEM = 1.32 Copyright ©2006 by Pearson Education, Inc. Upper Saddle River, New Jersey 07458 All rights reserved. Terry Overton Assessing Learners with Special Needs, 5e

The range of possible scores is Applying the SEM The SEM was 1.32. A student’s obtained score was 89. To determine the range of possible true scores, add and subtract the SEM from the obtained score of 89. 89 + 1.32 = 90.32 89 - 1.32 = 87.68 The range of possible scores is 87.68 - 90.32 Copyright ©2006 by Pearson Education, Inc. Upper Saddle River, New Jersey 07458 All rights reserved. Terry Overton Assessing Learners with Special Needs, 5e

Selecting the Best Test Instruments When considering which tests will be the most reliable, it is important to select a test that has the highest reliability coefficient and the smallest standard of error. This will mean that the results obtained are more likely to be more consistent with the student’s true ability. The obtained score will contain less error. Copyright ©2006 by Pearson Education, Inc. Upper Saddle River, New Jersey 07458 All rights reserved. Terry Overton Assessing Learners with Special Needs, 5e

Test Validity Validity indicates the degree of quality of the test instrument. This is a measure of how accurately the test measures what it is designed to measure. Some of the methods that indicate validity of a test include criterion-related validity, content validity, construct validity. Copyright ©2006 by Pearson Education, Inc. Upper Saddle River, New Jersey 07458 All rights reserved. Terry Overton Assessing Learners with Special Needs, 5e

Criterion-related Validity There are two ways to study criterion-related validity. Concurrent-related validity- When a test is compared with another measure at the same time. Predictive validity-When a test is compared with a measure in the future. For example, when college entrance exams are compared with student performance in College (GPAs). Copyright ©2006 by Pearson Education, Inc. Upper Saddle River, New Jersey 07458 All rights reserved. Terry Overton Assessing Learners with Special Needs, 5e

Content Validity In order for a test to have good content validity, it must have items that are representative of the domain or skill being assessed. During the development of the test, items are selected after careful study of the items and the domain they represent. Copyright ©2006 by Pearson Education, Inc. Upper Saddle River, New Jersey 07458 All rights reserved. Terry Overton Assessing Learners with Special Needs, 5e

Content Validity In order for a test to have good content validity, it must have items that are representative of the domain or skill being assessed. During the development of the test, items are selected after careful study of the items and the domain they represent. Copyright ©2006 by Pearson Education, Inc. Upper Saddle River, New Jersey 07458 All rights reserved. Terry Overton Assessing Learners with Special Needs, 5e

Construct Validity Construct validity means that the instrument has the ability to assess the psychological constructs it was meant to measure. A construct is a psychological trait or characteristic such as creativity or mathematical ability. Copyright ©2006 by Pearson Education, Inc. Upper Saddle River, New Jersey 07458 All rights reserved. Terry Overton Assessing Learners with Special Needs, 5e

Studying Construct Validity Construct validity can be studied by Investigating: Developmental changes-If an ability is expected to change across time and that ability is the construct of the test, testing across time will show changes in the performance on that test as the ability responds to developmental change. Copyright ©2006 by Pearson Education, Inc. Upper Saddle River, New Jersey 07458 All rights reserved. Terry Overton Assessing Learners with Special Needs, 5e

Construct Validity Continued Correlation with other tests-If the construct has been measured successfully by other instruments, a correlation with those other instruments is evidence of construct validity. Factor analysis-Using a statistical method known as factor analysis, evidence can be gathered for construct validity. Factor analysis allows the investigator to see if items that are thought to measure a construct are answered in the same manner. Copyright ©2006 by Pearson Education, Inc. Upper Saddle River, New Jersey 07458 All rights reserved. Terry Overton Assessing Learners with Special Needs, 5e

Construct Validity Continued Internal Consistency-This measure also provides evidence of construct validity. Convergent or discriminant validity-These Measures indicate that the performance is consistent with like measures (convergent) or the performance is unlike measures thought to measure different constructs than the one being assessed by the specific instrument. Copyright ©2006 by Pearson Education, Inc. Upper Saddle River, New Jersey 07458 All rights reserved. Terry Overton Assessing Learners with Special Needs, 5e

Construct Validity Continued Experimental interventions-When the construct being assessed can be influenced by an intervention or treatment, assessment before and after the intervention provides evidence of construct validity. Copyright ©2006 by Pearson Education, Inc. Upper Saddle River, New Jersey 07458 All rights reserved. Terry Overton Assessing Learners with Special Needs, 5e