Developing General Education Course Assessment Measures Anthony R. Napoli, PhD Lanette A. Raymond, MA Office of Institutional Research & Assessment Suffolk County Community College

Why Validity & Reliability ?  Assessment results must represent student achievement of course learning objectives  Evaluation of the validity and reliability of the assessment measure provides the evidence that it does so

Types of Measures  ‘Performance’ Measures  ‘Objective’ Measures

Validity for Performance Measures  Identified learning outcomes represent the course (domain sampling)  The measure addresses the learning outcomes (content validity)  There is a match between the measure and the rubric (criteria for evaluating performance)  Rubric scores can be linked to the learning outcomes, and indicate the degree of student achievement within the course

Validity for Objective Measures  Identified learning outcomes represent the course (domain sampling)  The items on the measure address specific learning outcomes (content validity)  Scores on the measure can be applied to the learning outcomes, and indicate the degree of student achievement within the course

Content Validity ( MA23 )

Content Validity ( MA61 )

Content Validity ( SO11 ) ObjectiveIIIIIIDescription Identify the basic methods of data collection Demonstrate an understanding of basic sociological concepts and social processes that shape human behavior Apply sociological theories to current social issues A 30-item test measured students’ mastery of the objectives

Content Validity ( SO11 )

Reliability  Can it be done consistently?  Can the rubric be applied consistently across raters -- Inter-rater reliability  Can each of the items act consistently as a measure of the construct -- Inter-item reliability

Inter-Rater Reliability ( MA23 ) – The Rubric Item 1A

Inter-Rater Reliability ( MA23 ) – The Rubric Item 1B

Inter-Rater Reliability ( MA23 ) – The Data Set

Inter-Rater Reliability ( MA23 ) – Results ItemLowHighRange 1A B C A B A B

Inter-Item Reliability ( MA61 ) Objective 1 Demonstrate an understanding of a mathematical function (range and domain, symmetric, composite, inverses). Items 1a, 1b, 2, 3,4, 8a,10a Understand the zeros of quadratic functions and sketch graphs of quadratic functions. Items 5a-d Comprehend the significance of the fundamental theorem of algebra and be able to solve a polynomial and write a polynomial in its factored form. Items 6a-f, Sketch the graph of rational functions. Items 8b-d Sketch the graphs of exponential and logarithmic functions. Items 9a-c, 10a-d Solve exponential and logarithmic equations. Items 11, Understand and graph the trigonometric functions, and solve applications using right triangle relationships. Items 13a-c,

Face Validity and Reliability… Is this enough?  Measures with face validity & adequate levels of reliability can produce misleading/inaccurate results.  Even content valid measures cannot guarantee accurate estimates of student achievement MA23 – earlier pilot

Criterion-Related Validity (MA23 – earlier pilot)

Criterion-Related Validity (MA23)

Criterion-Related Validity (MA61)

Motivational Comparison ( PC11 )  2 Groups Graded Embedded Questions Non-Graded Form & Motivational Speech  Mundane Realism

Motivational Comparison ( PC11 )  Graded condition produces higher scores (t(78) = 5.62, p <.001).  Large effect size (d = 1.27). (d = 1.27).

Motivational Comparison ( PC11 )  Minimum competency 70% or better 70% or better  Graded condition produces greater competency (Z = 5.69, p <.001).

Motivational Comparison ( PC11 )  In the non-graded condition this measure is neither reliable nor valid KR-20 N-g = 0.29

Motivational Comparison ( PC11 )

Developing General Education Course Assessment Measures Anthony R. Napoli, PhD Lanette A. Raymond, MA Office of Institutional Research & Assessment Suffolk County Community College