1. Establishing the Reliability and Validity of Outcomes Assessment Measures Anthony R. Napoli, PhD Lanette A. Raymond, MA Office of Institutional Research & Assessment Suffolk County Community College http://sccaix1.sunysuffolk.edu/Web/Central/IT/InstResearch/

2. Validity defined zThe validity of a measure indicates to what extent items measure some aspect of what they are purported to measure

3. Types of Validity zFace Validity zContent Validity zConstruct Validity zCriterion-Related Validity

4. Face Validity zIt looks like a test of *#%* zNot validity in a technical sense

5. Content Validity z Incorporates quantitative estimates z Domain Sampling z The simple summing or averaging of dissimilar items is inappropriate

6. Construct Validity zIndicated by correspondence of scores to other known valid measures of the underlying theoretical trait z Discriminant Validity z Convergent Validity

7. Criterion-Related Validity zRepresents performance in relation to particular tasks of discrete cognitive or behavioral objectives z Predictive Validity z Concurrent Validity

8. Reliability defined zThe reliability of a measure indicates the degree to which an instrument consistently measures a particular skill, knowledge base, or construct zReliability is a precondition for validity

9. Types of Reliability zInter-rater (scorer) reliability zInter-item reliability zTest-retest reliability zSplit-half & alternate forms reliability

10. Validity & Reliability in Plain English zAssessment results must represent the institution, program, or course zEvaluation of the validity and reliability of the assessment instrument and/or rubric will provide the documentation that it does

11. Content Validity for Subjective Measures zThe learning outcomes represent the program/course (domain sampling) zThe instrument addresses the learning outcomes zThere is a match between the instrument and the rubric zRubric scores can be applied to the learning outcomes, and indicate the degree of student achievement within the program/course

12. Inter-Scorer Reliability consistently zRubric scores can be obtained and applied to the learning outcomes, and indicate the degree of student achievement within the program/course consistently

13. Content Validity for Objective Measures zThe learning outcomes represent the program/course zThe items on the instrument address specific learning outcomes zInstrument scores can be applied to the learning outcomes, and indicate the degree of student achievement within the program/course

14. Inter-Item Reliability consistently zItems that measure the same learning outcomes should consistently exhibit similar scores

15. Content Validity ( CH19 ) Objective I II III IV Description Write and decipher chemical nomenclature Solve both quantitative and qualitative problems Balance equations and solve mathematical problems associated w/ balanced equations Demonstrate an understanding intra-molecular forces A 12-item test measured students’ mastery of the objectives

16. Content Validity ( CH19 )

17. Content Validity ( SO11 ) Objective I II III Description Identify the basic methods of data collection Demonstrate an understanding of basic sociological concepts and social processes that shape human behavior Apply sociological theories to current social issues A 30-item test measured students’ mastery of the objectives

18. Content Validity ( SO11 )

20. Inter-Rater Reliability Fine Arts Portfolio zDrawing zDesign zTechnique zCreativity zArtistic Process zAesthetic Criteria zGrowth zPortfolio Presentation Scale: 5 = Excellent 4 = Very Good 3 = Satisfactory 2 = Unsatisfactory 1 = Unacceptable

21. Inter-Rater Reliability Fine Arts Portfolio

22. Inter-Item Reliability ( PC11 ) Objective Description Demonstrate a satisfactory knowledge of: 1. the history, terminology, methods, & ethics in psychology 2. concepts associated with the 5 major schools of psychology 3. the basic aspects of human behavior including learning and memory, personality, physiology, emotion, etc… 4. an ability to obtain and critically analyze research in the field of modern psychology A 20-item test measured students’ mastery of the objectives

23. Inter-Item Reliability ( PC11 ) zEmbedded-questions methodology  Inter-item or internal consistency reliability KR-20, r tt =.71 z Mean score = 12.478 z Std Dev = 3.482 z Std Error = 0.513 z Mean grade = 62.4%

24. Inter-Item Reliability ( PC11 ) Motivational Comparison z2 Groups Graded Embedded Questions Non-Graded Form & Motivational Speech zMundane Realism

25. Inter-Item Reliability ( PC11 ) Motivational Comparison zGraded condition produces higher scores (t(78) = 5.62, p <.001). zLarge effect size (d = 1.27).

26. Inter-Item Reliability ( PC11 ) Motivational Comparison z Minimum competency 70% or better zGraded condition produces greater competency (Z = 5.69, p <.001).

27. Inter-Item Reliability ( PC11 ) Motivational Comparison zIn the non-graded condition this measure is neither reliable nor valid KR-20 N-g = 0.29

28. Criterion-Related Concurrent Validity ( PC11 )

29. “I am ill at these numbers.” -- Hamlet --

30. “When you can measure what you are speaking about and express it in numbers, you know something about it; but when you cannot express it in numbers, your knowledge is of a meager and unsatisfactory kind.” -- Lord Kelvin -- “There are three kinds of lies: lies, damned lies, and statistics.” -- Benjamin Disraeli --