Using Item Response Theory to Track Longitudinal Course Changes

Slides:

Advertisements

Similar presentations

Measurement, Evaluation, Assessment and Statistics

Advertisements

Advanced Methods and Analysis for the Learning and Social Sciences PSY505 Spring term, 2012 January 23, 2012.

Item Response Theory in a Multi-level Framework Saralyn Miller Meg Oliphint EDU 7309.

BCOR 1020 Business Statistics Lecture 15 – March 6, 2008.

Logical Line Fitting: One Step in the EDA Process by Shannon Guerrero Northern Arizona University NCTM 2008 Annual Meeting & Exposition Salt Lake City,

Item Response Theory. Shortcomings of Classical True Score Model Sample dependence Limitation to the specific test situation. Dependence on the parallel.

Assessment Data Brigham Young University Department of Civil and Environmental Engineering W. Spencer Guthrie April 25, 2008 Brigham Young University.

Intro to Statistics for the Behavioral Sciences PSYC 1900 Lecture 3: Central Tendency And Dispersion.

Edpsy 511 Homework 1: Due 2/6.

Why Scale -- 1 Summarising data –Allows description of developing competence Construct validation –Dealing with many items rotated test forms –check how.

12.3 – Measures of Dispersion

Classical Test Theory By ____________________. What is CCT?

Item Response Theory. What’s wrong with the old approach? Classical test theory –Sample dependent –Parallel test form issue Comparing examinee scores.

Descriptive Statistics Measures of Center. Essentials: Measures of Center (The great mean vs. median conundrum.)  Be able to identify the characteristics.

Stats 95 Statistical analysis without compelling presentation is annoying at best and catastrophic at worst. From raw numbers to meaningful pictures.

Measures of Central Location (Averages) and Percentiles BUSA 2100, Section 3.1.

The ABC’s of Pattern Scoring Dr. Cornelia Orr. Slide 2 Vocabulary Measurement – Psychometrics is a type of measurement Classical test theory Item Response.

Measuring Mathematical Knowledge for Teaching: Measurement and Modeling Issues in Constructing and Using Teacher Assessments DeAnn Huinker, Daniel A. Sass,

Copyright (C) 2002 Houghton Mifflin Company. All rights reserved. 1 Understandable Statistics Seventh Edition By Brase and Brase Prepared by: Lynn Smith.

Examples for the midterm. data = {4,3,6,3,9,6,3,2,6,9} Example 1 Mode = Median = Mean = Standard deviation = Variance = Z scores =

Instrumentation (cont.) February 28 Note: Measurement Plan Due Next Week.

Mean and Standard Deviation of Grouped Data Make a frequency table Compute the midpoint (x) for each class. Count the number of entries in each class (f).

Test Scaling and Value-Added Measurement Dale Ballou Vanderbilt University April, 2008.

ELA & Math Scale Scores Steven Katz, Director of State Assessment Dr. Zach Warner, State Psychometrician.

University of Georgia – Chemistry Department JExam - A Method to Measure Outcomes Assessment Charles H. Atwood, Kimberly D. Schurmeier, and Carrie G. Shepler.

A COMPARISON METHOD OF EQUATING CLASSIC AND ITEM RESPONSE THEORY (IRT): A CASE OF IRANIAN STUDY IN THE UNIVERSITY ENTRANCE EXAM Ali Moghadamzadeh, Keyvan.

Copyright (C) 2002 Houghton Mifflin Company. All rights reserved. 1 CHEBYSHEV'S THEOREM For any set of data and for any number k, greater than one, the.

The ABC’s of Pattern Scoring

What’s with all those numbers?.  What are Statistics?

Item Analysis: Classical and Beyond SCROLLA Symposium Measurement Theory and Item Analysis Heriot Watt University 12th February 2003.

WEIGHTED AVERAGE ALG114 Weighted Average: an average where every quantity is assigned a weight. Example: If a teacher thinks it’s more important, a final.

2. Main Test Theories: The Classical Test Theory (CTT) Psychometrics. 2011/12. Group A (English)

WELCOME TO MATH 3 Please begin reading the syllabus on your desk!

Item Response Theory Dan Mungas, Ph.D. Department of Neurology University of California, Davis.

Exam 1. Spring Comparison Spring 2007 A’s 12 8% B’s 25 17% C’s 46 32% Passing 83 57% D’s 34 23% Median 71% Spring 2008 A’s 25 14% B’s 45 27% C’s 38 22%

The Normal Distribution AS Mathematics Statistics 1 Module.

Mean, Median, and Mode Lesson 7-1. Mean The mean of a set of data is the average. Add up all of the data. Divide the sum by the number of data items you.

Utilizing Item Analysis to Improve the Evaluation of Student Performance Mihaiela Ristei Gugiu Central Michigan University Mihaiela Ristei Gugiu Central.

Using Psychometric Analysis to Drive Mathematics Standardized Assessment Decision Making Mike Mazzarella George Mason University.

Lesson 2 Main Test Theories: The Classical Test Theory (CTT)

Copyright © 2016 Brooks/Cole Cengage Learning Intro to Statistics Part II Descriptive Statistics Intro to Statistics Part II Descriptive Statistics Ernesto.

Characteristics of Normal Distribution symmetric with respect to the mean mean = median = mode 100% of the data fits under the curve.

STATISTICS Chapter 2 and and 2.2: Review of Basic Statistics Topics covered today:  Mean, Median, Mode  5 number summary and box plot  Interquartile.

Norm Referenced Your score can be compared with others 75 th Percentile Normed.

Copyright © Cengage Learning. All rights reserved. Section 3.1 Measures of Central Tendency: Mode, Median, and Mean.

Descriptive Statistics Measures of Center

Looking at the both ‘ends’ of the social aptitude dimension

Intro to Statistics Part II Descriptive Statistics

Introduction to the Validation Phase

Intro to Statistics Part II Descriptive Statistics

Do-Now-Day 2 Section 2.2 Find the mean, median, mode, and IQR from the following set of data values: 60, 64, 69, 73, 76, 122 Mean- Median- Mode- InterQuartile.

EVSC 1300—Spring, 2018 Exam 3 35 total points >28 A 22–27 B 17–21 C

الاختبارات محكية المرجع بناء وتحليل (دراسة مقارنة )

Organizing and Displaying Data

Stem & Leaf Plots How to make a Stem & Leaf Plot.

מדינת ישראל הוועדה לאנרגיה אטומית

Anthropology Graduate Program – Sample Timeline

Decimal Applications : Mean, Median, and Mode

Business Assessment Test Results

By ____________________

EVSC 1300—Spring, 2017 Exam 3 40 total points >35 A 29–34 B 23–28 C

CS3332(01) Course Description

Anthropology Graduate Program – Sample Timeline

EECS3030(02) Course Description

Class Greeting.

Common Exams: Fall Data Update

Chapter 6: Probability.

14.2 Measures of Central Tendency

Stem & Leaf Plots How to make a Stem & Leaf Plot.

Women’s Golf Scores (8 golfers)

Presentation transcript:

Using Item Response Theory to Track Longitudinal Course Changes Charles H. Atwood, Braden Ohlsen, and Brock Casselman University of Utah Department of Chemistry Salt Lake City, UT

Quick Description of Item Response Theory (IRT) Classical Test Theory (CTT) Mean, median, mode Discrimination Index comparison of top and bottom quartiles CTT is good for small samples CTT is more dependent upon the subject group Same assessment given to 2 groups will give different results

Quick Description of Item Response Theory (IRT) More modern psychometric analysis tool than CTT Works well for large sample sizes (> 200) Iterative process which fits the students responses to a mathematical formula

Quick Description of Item Response Theory (IRT) We use the program Bilog-MG3 to fit our data. Bilog-MG3 assigns to every test item values for a, b, and c and displays them on an item characteristic curve Plot of the probability of a student with a given IRT ability vs. probability that the student answered the question

Quick Description of Item Response Theory (IRT) Item Characteristic Curve

Description of Item Response Theory (IRT) Item Information Curve for every test question

Description of Item Response Theory (IRT) Test information curve Sum of Item Information Curves for an assessment

IRT Equating

CHEM 1210 Equated Average Scores 2014 - 2016 Fall 2014 2015 Equated to 2014 2016 Equated Midterm Exam 1 70.3% 74.2% 74.0% Midterm Exam 2 62.3% 68.1% 68.5% Midterm Exam 3 66.9% 79.0% 78.5%

Equated Test 1

Equated Test 2

Equated Test 3

ACS FINAL EXAM CHANGES FROM 2013 TO 2016

CHEM 1220 Equated Average Scores 2015 - 2016 Spring 2014 Spring 2015 Equated to 2014 Spring 2016 Equated to 2014 Midterm Exam 1 66.7% 75.9% 83.7% Midterm Exam 2 62.9% 73.4% 78.5%

Equated Test 1

Equated Test 2