Assessing Learners with Special Needs: An Applied Approach, 6e © 2009 Pearson Education, Inc. All rights reserved. Chapter 4:Reliability and Validity

Assessing Learners with Special Needs: An Applied Approach, 6e Overton © 2009 Pearson Education, Inc. All Rights Reserved. 2 Reliability—Having confidence in the consistency of the test results. Validity—Having confidence that the test is measuring what it is supposed to measure.

Assessing Learners with Special Needs: An Applied Approach, 6e Overton © 2009 Pearson Education, Inc. All Rights Reserved. 3 Correlation Correlation—a statistical method of observing the degree of relationship between two sets of data or two sets of variables. Correlation coefficient—the numerical representation of the strength and direction of the relationship between two sets of variables or data. Pearson’s r—a statistical formula for determining strength and direction of correlations.

Assessing Learners with Special Needs: An Applied Approach, 6e Overton © 2009 Pearson Education, Inc. All Rights Reserved. 4 Positive Correlation In investigating how data are related, it is important to determine if the two sets of data represent positive, negative, or no correlation. In a positive correlation, when a student scores high on the first variable or test, the student will also score high on the second measure.

Assessing Learners with Special Needs: An Applied Approach, 6e Overton © 2009 Pearson Education, Inc. All Rights Reserved. 5 Student Student Student Student Student Student Student Student Set 1Set 2 The data below illustrate a positive correlation:

Assessing Learners with Special Needs: An Applied Approach, 6e Overton © 2009 Pearson Education, Inc. All Rights Reserved. 6 Negative Correlation In a negative correlation, when a student scores high on one variable or test, the student will score low on the other variable or test.

Assessing Learners with Special Needs: An Applied Approach, 6e Overton © 2009 Pearson Education, Inc. All Rights Reserved. 7 Student Student Student Student Student Student Student Student Set 1 Set 2 The data below illustrate a negative correlation:

Assessing Learners with Special Needs: An Applied Approach, 6e Overton © 2009 Pearson Education, Inc. All Rights Reserved. 8 Data Set Data Set Two sets of data are presented below. Determine if the data sets represent a positive, negative, or no relationship. One way to determine the direction of the relationship is to plot the scores on a Scatter plot. The data for set 1 and set 2 are plotted on the next slide.

Assessing Learners with Special Needs: An Applied Approach, 6e Overton © 2009 Pearson Education, Inc. All Rights Reserved. 9 * ** * ** * * * Set 1 Set 2

Assessing Learners with Special Needs: An Applied Approach, 6e Overton © 2009 Pearson Education, Inc. All Rights Reserved. 10 The direction of the line plotted on the scatter plot provides a clue about the relationship. If the relationship is positive, the direction of the line looks like this:

Assessing Learners with Special Needs: An Applied Approach, 6e Overton © 2009 Pearson Education, Inc. All Rights Reserved. 11 If the data represent a negative correlation, the direction of the line in the scatter plot looks like this:

Assessing Learners with Special Needs: An Applied Approach, 6e Overton © 2009 Pearson Education, Inc. All Rights Reserved. 12 Test-retest reliability—A study that employs the readministration of a single instrument to check for consistency across time. Equivalent forms reliability—Consistency of a test using like forms that measure the same skill, domain, or trait; also known as alternate forms reliability. Methods of Studying Reliability

Assessing Learners with Special Needs: An Applied Approach, 6e Overton © 2009 Pearson Education, Inc. All Rights Reserved. 13 Internal consistency—Methods to study the reliability across the items of the test. Examples include split-half reliability, K-R 20, and coefficient alpha. Split-half reliability—A method of measuring internal consistency by studying the reliability across items by comparing the data of the two halves of the test.

Assessing Learners with Special Needs: An Applied Approach, 6e Overton © 2009 Pearson Education, Inc. All Rights Reserved. 14 Interrater Reliability—The consistency of a test to measure a skill, trait, or domain across examiners. This type of reliability is most important when responses are subjective or open-ended. Reliability coefficients may vary across age and grade levels of a specific instrument.

Assessing Learners with Special Needs: An Applied Approach, 6e Overton © 2009 Pearson Education, Inc. All Rights Reserved. 15 Kuder-Richardson 20 (K-R 20)—a formula used to check consistency across items of an instrument that has items scored as 1 or 0 or right/wrong. Coefficient Alpha—a formula used to check the consistency across items of instruments with responses of varying credit. For example, items may be scored as 0, 1, 2, or 3 points.

Assessing Learners with Special Needs: An Applied Approach, 6e Overton © 2009 Pearson Education, Inc. All Rights Reserved. 16 Interpreting Scores True score—what a student would actually score if there were no error in the assessment process. Obtained score—obtained score = true score + error Error—a variety of factors that interfere with obtaining a student’s true score (e.g., student fatigue, bad lighting, poorly written questions)

Assessing Learners with Special Needs: An Applied Approach, 6e Overton © 2009 Pearson Education, Inc. All Rights Reserved. 17 Standard error of measurement—the amount of error determined to exist using a specific instrument, calculated using the instrument’s standard deviation and reliability. It is used to estimate a range of scores within which the student’s true score exists.

Assessing Learners with Special Needs: An Applied Approach, 6e Overton © 2009 Pearson Education, Inc. All Rights Reserved. 18 Confidence interval—the range of scores for an obtained score determined by adding and subtracting the standard error of measurement. Confidence interval = (obtained score – SEM) to (obtained score + SEM)

Assessing Learners with Special Needs: An Applied Approach, 6e Overton © 2009 Pearson Education, Inc. All Rights Reserved. 19 Estimated true score—a method of calculating the amount of error correlated with the distance of the score from the mean of the group. Estimated true score = M + r(X – M) where M = mean of distribution, r = reliability coefficient, X = obtained score

Assessing Learners with Special Needs: An Applied Approach, 6e Overton © 2009 Pearson Education, Inc. All Rights Reserved. 20 Standard Error of Measurement The standard error of measurement is calculated using the following formula: 1-.r Where SEM = the standard error of measurement SD = the standard deviation of the norm group of scores obtained during the development of the instrument r = the reliability coefficient SEM = SD

Assessing Learners with Special Needs: An Applied Approach, 6e Overton © 2009 Pearson Education, Inc. All Rights Reserved. 21 Example of Calculating SEM For a specific test, the standard deviation is 4. The reliability coefficient is.89. The SEM would be: x.33 = 1.32 SEM = 1.32 This represents the amount of error on this test instrument.

Assessing Learners with Special Needs: An Applied Approach, 6e Overton © 2009 Pearson Education, Inc. All Rights Reserved. 22 Applying the SEM The SEM was A student’s obtained score was 89. To determine the range of possible true scores, add and subtract the SEM from the obtained score of = = The range of possible scores is

Assessing Learners with Special Needs: An Applied Approach, 6e Overton © 2009 Pearson Education, Inc. All Rights Reserved. 23 Selecting the Best Test Instruments When considering which tests will be the most reliable, it is important to select a test that has the highest reliability coefficient and the smallest standard of error. This will mean that the results obtained are more likely to be more consistent with the student’s true ability. The obtained score will contain less error.

Assessing Learners with Special Needs: An Applied Approach, 6e Overton © 2009 Pearson Education, Inc. All Rights Reserved. 24 Test Validity Validity—the degree of quality of the test instrument. This is a measure of how accurately the test measures what it is designed to measure. Methods that indicate validity of a test include criterion-related validity, content validity, and construct validity.

Assessing Learners with Special Needs: An Applied Approach, 6e Overton © 2009 Pearson Education, Inc. All Rights Reserved. 25 Concurrent-related validity—When a test is compared with another measure at the same time. There are two ways to study criterion-related validity. Predictive validity—When a test is compared with a measure in the future. For example, when college entrance exams are compared with student performance in College (GPAs). Criterion-Related Validity

Assessing Learners with Special Needs: An Applied Approach, 6e Overton © 2009 Pearson Education, Inc. All Rights Reserved. 26 In order for a test to have good content validity, it must have items that are representative of the domain or skill being assessed. During the development of the test, items are selected after careful study of the items and the domain they represent. Content Validity

Assessing Learners with Special Needs: An Applied Approach, 6e Overton © 2009 Pearson Education, Inc. All Rights Reserved. 27 Construct validity means that the instrument has the ability to assess the psychological constructs it was meant to measure. A construct is a psychological trait or characteristic such as creativity or mathematical ability. Construct Validity

Assessing Learners with Special Needs: An Applied Approach, 6e Overton © 2009 Pearson Education, Inc. All Rights Reserved. 28 Construct validity can be studied by investigating: Developmental changes—If an ability is expected to change across time and that ability is the construct of the test, testing across time will show changes in the performance on that test as the ability responds to developmental change. Studying Construct Validity

Assessing Learners with Special Needs: An Applied Approach, 6e Overton © 2009 Pearson Education, Inc. All Rights Reserved. 29 Correlation with other tests—If the construct has been measured successfully by other instruments, a correlation with those other instruments is evidence of construct validity. Factor analysis—Using a statistical method known as factor analysis, evidence can be gathered for construct validity. Factor analysis allows the investigator to see if items that are thought to measure a construct are answered in the same manner.

Assessing Learners with Special Needs: An Applied Approach, 6e Overton © 2009 Pearson Education, Inc. All Rights Reserved. 30 Internal Consistency—This measure also provides evidence of construct validity. Convergent or discriminant validity— These measures indicate that the performance is consistent with like measures (convergent) or the performance is unlike measures thought to measure different constructs than the one being assessed by the specific instrument.

Assessing Learners with Special Needs: An Applied Approach, 6e Overton © 2009 Pearson Education, Inc. All Rights Reserved. 31 Experimental interventions— When the construct being assessed can be influenced by an intervention or treatment, assessment before and after the intervention provides evidence of construct validity.