Creating Assessments The three properties of good assessments

Validity: A case study using The Shakespeare Test “To ensure that my lessons were effective, I chose to implement a pre- and post-test on the three-class periods that would be participating in my Shakespeare unit... After administering the pre-test, I found that approximately 90% of the students scored an ‘F’ and the other 10% landed just above the “F” grade level.”

Validity: A case study using The Shakespeare Test “At the end of the 20-day unit, I administered the post-test. My assumption was that the majority of them would score high on the test. Fortunately, my assumption paid off and I found that there were zero “D” level students and nearly 80% of the students were in the “B” to “A” range. Much like the pre-test, I found this extremely satisfying since I felt I had a few complications here and there throughout the unit.”

Creating Assessments All good assessments have three key features: All good assessments have three key features: –Validity –Reliability –Usability

Validity val·id adj. 1. Well grounded; just. 2. Producing the desired results; efficacious. 3. Having legal force; effective or binding. 4. Logic. a. Containing premises from which the conclusion may logically be derived. b. Correctly inferred or deduced from a premise. (American Heritage Dictionary)

Validity

Validity Validity: an evaluation of the adequacy and appropriateness of the interpretations and uses of assessment results. (Linn & Gronlund, 2000). Validity: an evaluation of the adequacy and appropriateness of the interpretations and uses of assessment results. (Linn & Gronlund, 2000).

Validity Let us emphasize some important points: Let us emphasize some important points: Validity refers to the appropriateness of the interpretation of the results of an assessment procedure, not to the procedure itself. Consider the following example: Consider the following example:

Dilbert’s Medical Exam One day Dilbert was not feeling well. He went to the Doctor’s office for a check-up. While there he and the doctor had a discussion about who was smarter, doctors or engineers. Dilbert lost the argument when the doctor pointed out that doctors make more money; therefore they must be smarter. After this, the doctor informed Dilbert that his test results indicated he had a terminal condition and would die within a few weeks. This is what happened next.

Validity Four Primary Categories of Validity: Four Primary Categories of Validity: –Content –Construct –Criterion –Consequential

Validity Content Validity Content Validity –How well does the sample of assessment tasks represent the domain of tasks to be measured? “Math Content Universe” CountingGeometryAlgebra Trigonometry Calculus + - x ÷ MultivariateCalculus TopologyNumericalAnalysis

Validity Construct Validity Construct Validity –How well can performance on the assessment be explained in terms of psychological characteristics? –How does one measure effort? Love? Hate? Indifference? Intelligence? Situational Awareness?

Validity Criterion Validity Criterion Validity –How accurately does performance on the assessment predict future performance or estimate the present performance on some other valued measure called a criterion?

Validity Consequential Validity Consequential Validity –How well did use of the assessment serve the intended purpose and avoid adverse effects?

Validity Cautions to keep in mind: Cautions to keep in mind: –Validity is a unitary concept. In the gestalt tradition, the “whole” of validity is greater than the sum of its various “parts.” –Validity is always specific to some use or interpretation, and no assessment is valid for all purposes. An assessment may have high validity in some aspects, but low validity for others.

Validity Cautions to keep in mind: Cautions to keep in mind: –Though we often say “validity of the test”, it is important to recognize that what is really being validated is the interpretation and use of the test results. –Validity is a matter of degree. It exists along a continuum, the same way gray exists in the continuum between black and white. At best we can classify it as “high”, “moderate”, or “low”

Validity Cautions to keep in mind: Cautions to keep in mind: –Validity involves an overall evaluative judgment. This judgment must be based on sound assessment results that include accurate measurement. Errors in the assessment process can radically affect the interpretation of the results and the resulting evaluation.

Validity To help insure the validity of your assessments, items should be: To help insure the validity of your assessments, items should be: –Written based on the content defined in the objectives. –Assessed at the specified learning level. –Reviewed by competent colleagues.

Validity The following is a list of common mistakes that will LOWER validity: The following is a list of common mistakes that will LOWER validity: –Use too few items to assess the objective. –Provide tasks or write items that do not function as intended (e.g. ambiguous, clues that lead to answers, bias, too easy, too hard, etc.). –Arrange items from hardest to easiest.

Validity Mistakes that LOWER validity (cont): Mistakes that LOWER validity (cont): –Write items that do not match the learning level specified in the objectives. –Impose unrealistic time constraints –Provide an atmosphere that is poorly suited to the purpose (e.g. noisy, messy, malfunctioning equipment). –Provide unclear or confusing directions. –Unfair or unsystematic scoring.

Validity –It is an evil for a well-taught and well- trained student to fail in an examination. –It is an evil for an unqualified student, through some inefficiency of the test, to obtain credit in an examination. New York State Department of Education, 1906