Reliability & Validity

Reliability-Having confidence in the consistency of the test results. Reliability of a test refers to how well it provides a consistent (stable) set of results across similar test situations, time periods, and examiners or the extent to which measurements are repeatable.

Correlation- A statistical method of observing the degree of relationship between two sets of data or two sets of variables (e.g., hours spent studying and grades) Correlation coefficient -The numerical representation of the strength and direction of the relationship between two sets of variables or data. Represented by a real number between -/+ 1.00 (the stronger the correlation= the more reliable) * In investigating how data are related, it is important to determine if the two sets of data represent positive, negative, or no correlation.

Positive Correlation * In a positive correlation, when a student scores high on the first variable or test, the student will also score high on the second measure (e.g. calories eaten and weight) * The data below illustrate a positive correlation: Student 1- 75 Student 2- 88 Student 3- 90 Student 4- 63 Student 1- 72 Student 2- 89 Student 3- 93 Student 4- 64 Set 1 Set 2

Negative Correlation In a negative correlation, when a student scores high on one variable or test, the student will score low on the other variable or test (e.g. days absent and test scores) The data below illustrate a negative correlation: Student 1- 88 Student 2- 99 Student 3- 56 Student 4- 97 Student 1- 32 Student 2- 45 Student 3- 15 Student 4- 12 Set 1 Set 2

Two sets of data are presented below. Determine if the data sets represent a positive, negative, or no relationship. Data Set 1 94 78 89 58 62 77 75 45 95 Data Set 2 95 80 90 59 60 78 76 47 97 One way to determine the direction of the relationship is to plot the scores on a Scatter plot. The data for set 1 and set 2 are plotted on the next slide.

100 95 90 85 80 75 70 65 60 55 50 45 Set 1 * ** 45 50 55 60 65 70 75 80 85 90 95 100 Set 2

The direction of the line plotted on the scatter plot provides a clue about the relationship. If the relationship is positive, the direction of the line looks like this:

If the data represent a negative correlation, the direction of the line in the scatter plot looks like this:

No Correlation When there is little or no relationship between scores. The scattergram does not indicate a distinct line Figure 4.4 See scattergrams on p. 121 in Overton text

Methods of Studying Reliability Test-retest reliability- A study that employs the re-administration of a single instrument to check for consistency across time. (Sensitive to practice effects and effects of instruction!) Equivalent forms reliability-Consistency of a test using like forms that measure the same skill, domain, or trait; also known as alternate forms reliability. (Form A & B of a test)

Methods of Studying Reliability, Continued Internal consistency-Methods to study the reliability across the items of the test (within the test). Split-half reliability-studying the reliability across items by comparing the data of the two halves of the test. Kuder-Richardson 20- (K-R 20)- A formula used to check consistency across items of an instrument that has items scored as 1 or 0 or right/wrong. Coefficient Alpha- A formula used to check the consistency across items of instruments with responses of varying credit. For example, items may be scored as 0,1,2,or 3 points (e.g. a rubric).

Methods of Studying Reliability, Continued Interrater Reliability- The consistency of a test to measure a skill, trait, or domain across examiners. This type of reliability is most important when responses are subjective or open-ended. Reliability coefficients may vary across age and grade levels of a specific instrument!

Random Error Random error describes random events that have nothing to do with the quality being measured but impact the results Random error impacts the reliability of a test, and consequently all possible sources of random error must be addressed

Standard Error of Measurement Each test score is made of two parts: true score and error. A student’s true score may only be estimated. The standard error of measurement is a method used to estimate the amount of error of a test. It represents the typical amount of error of any obtained score. The standard error of measurement is used to estimate a range of scores within which the student’s true scores exists. Obtained Score = True Score + Error

Standard Error of Measurement, Continued The standard error of measurement is calculated using the following formula: SEM = SD 1-.r Where SEM = the standard error of measurement SD = the standard deviation of the norm group of scores obtained during the development of the instrument r = the reliability coefficient

Example of Calculating SEM For a specific test, the standard deviation is 4. The reliability coefficient is .89. The SEM would be: 4 1- .89 .11 This represents the amount of error on this test instrument. 4 4 x .33 = 1.32 SEM = 1.32

The range of possible scores is Applying the SEM The SEM was 1.32. A student’s obtained score was 89. To determine the range of possible true scores, add and subtract the SEM (a certain factor) from the obtained score of 89. 89 + 1.32(1) = 90.32 89 - 1.32(1) = 87.68 The range of possible scores is 87.68 - 90.32

Confidence Intervals Confidence intervals allow us to make statements about the proximity of the obtained score to the true score. A 68% confidence interval allows us to say that even though we don't know the true score, there is only a 32% chance that it is not within the range specified. A 95% confidence interval allows us to say that there is only a 5% chance that the person's true score is not within the range specified.

An Example Milly's obtained score on the Spelling Accuracy Test was 53. The SEM of the test is 1.85. We wish to be very confident in our estimate of Milly's score and choose a 95% confidence interval. See factor list…The factor to multiply the SEM by to achieve a 95% CI is 1.96 so 1.96 x 1.85=3.62. 53 + or – 3.62= Lower bound= 49.4 and Upper bound = 56.7 There is a 95% probability that Milly's true score on the Spelling Accuracy Test falls in the region between 49.38 and 56.62.

Selecting the Best Test Instruments When considering which tests will be the most reliable, it is important to select a test that has the highest reliability coefficient and the smallest standard of error. This will mean that the results obtained are more likely to be more consistent with the student’s true ability. The obtained score will contain less error.

Validity-Having confidence that the test is measuring what it is supposed to measure. Validity indicates the degree of quality of the test instrument. Validity coefficient ranges between -/+1.0. Validity is a relative rather than absolute notion.

Criterion-related Validity established by comparing performance on an accepted standard or criterion There are two ways to study criterion-related validity. Concurrent-related validity- When a test is compared with a similar measure administered within a short period of time. Predictive validity-When a test is compared with a measure in the future. For example, when college entrance exams are compared with student performance in College (GPAs).

Content Validity In order for a test to have good content validity, it must have items that are representative of the domain or skill being assessed. During the development of the test, items are selected after careful study of the items and the domain they represent. To ascertain content validity, submit the test to a panel of expert judges. The judges should examine the test in terms of: Completeness Appropriateness Format Bias

Construct Validity Construct validity means that the instrument has the ability to assess the psychological constructs it was meant to measure. A construct is a psychological trait or characteristic such as creativity or mathematical ability.

Studying Construct Validity Developmental changes Correlation with other tests Factor Analysis Internal consistency Convergent or discriminant validation Experimental interventions Overton, 138 & 139

External, Internal, and Social Validity External validity is concerned with generalizability (e.g., Do the results obtained in one particular situation apply in other situations?) Internal validity is the fundamental basis for interpreting the results of an intervention (e.g., Did the intervention make a difference in this specific instance?) Social validity refers to the social value and acceptability of an educational intervention.