Choosing tests for EEF evaluations – reliability and validity and other issues Steve Higgins & Carole Torgerson

Slides:

Advertisements

Similar presentations

Questionnaire Development

Advertisements

The Research Consumer Evaluates Measurement Reliability and Validity

Taking Stock Of Measurement. Basics Of Measurement Measurement: Assignment of number to objects or events according to specific rules. Conceptual variables:

1 COMM 301: Empirical Research in Communication Kwan M Lee Lect4_1.

Psychlotron.org.uk Why is it important that psychological research be valid and reliable?

Types of Reliability.

MEASUREMENT CONCEPTS © 2012 The McGraw-Hill Companies, Inc.

Psychometrics William P. Wattles, Ph.D. Francis Marion University.

VALIDITY AND RELIABILITY

Chapter Ten. Figure 10.1 Relationship of Noncomparative Scaling to the Previous Chapters and the Marketing Research Process Focus of This Chapter Relationship.

Reliability Analysis. Overview of Reliability What is Reliability? Ways to Measure Reliability Interpreting Test-Retest and Parallel Forms Measuring and.

Part II Sigma Freud & Descriptive Statistics

MEQ Analysis. Outline Validity Validity Reliability Reliability Difficulty Index Difficulty Index Power of Discrimination Power of Discrimination.

Chapter 1. Herbert Spencer v. Charles Darwin In the first volume of A System of Synthetic Philosophy, entitled First Principles (1862), Spencer argued.

CH. 9 MEASUREMENT: SCALING, RELIABILITY, VALIDITY

Measurement. Scales of Measurement Stanley S. Stevens’ Five Criteria for Four Scales Nominal Scales –1. numbers are assigned to objects according to rules.

Reliability and Validity of Research Instruments

RELIABILITY & VALIDITY What is Reliability? What is Reliability?What is Reliability?What is Reliability? How Can We Measure Reliability? How Can We Measure.

Reliability Analysis. Overview of Reliability What is Reliability? Ways to Measure Reliability Interpreting Test-Retest and Parallel Forms Measuring and.

Reliability and Validity Dr. Roy Cole Department of Geography and Planning GVSU.

RELIABILITY & VALIDITY

Definition & Measurement “measurement is the beginning of science, … until you can measure something, your knowledge is meager and unsatisfactory” Lord.

Lecture 7 Psyc 300A. Measurement Operational definitions should accurately reflect underlying variables and constructs When scores are influenced by other.

Measurement: Reliability and Validity For a measure to be useful, it must be both reliable and valid Reliable = consistent in producing the same results.

SOWK 6003 Social Work Research Week 5 Measurement By Dr. Paul Wong.

SELECTION & ASSESSMENT SESSION THREE: MEASURING THE EFFECTIVENESS OF SELECTION METHODS.

Validity, Reliability, & Sampling

Research Methods in MIS

 Rosseni Din  Muhammad Faisal Kamarul Zaman  Nurainshah Abdul Mutalib  Universiti Kebangsaan Malaysia.

Validity and Reliability

Reliability, Validity, & Scaling

Measurement in Exercise and Sport Psychology Research EPHE 348.

Instrumentation.

Educational Research: Competencies for Analysis and Application, 9 th edition. Gay, Mills, & Airasian © 2009 Pearson Education, Inc. All rights reserved.

MEASUREMENT CHARACTERISTICS Error & Confidence Reliability, Validity, & Usability.

Data Analysis. Quantitative data: Reliability & Validity Reliability: the degree of consistency with which it measures the attribute it is supposed to.

Copyright © 2012 Wolters Kluwer Health | Lippincott Williams & Wilkins Chapter 14 Measurement and Data Quality.

The Basics of Experimentation Ch7 – Reliability and Validity.

Validity and Reliability THESIS. Validity u Construct Validity u Content Validity u Criterion-related Validity u Face Validity.

Chapter Five Measurement Concepts. Terms Reliability True Score Measurement Error.

Tests and Measurements Intersession 2006.

METHOD in Personality Research. How do we gather data? 1. From whom??? 1. From whom??? A. Self A. Self B. Others B. Others Plus/Minus? Plus/Minus?

Research methods in clinical psychology: An introduction for students and practitioners Chris Barker, Nancy Pistrang, and Robert Elliott CHAPTER 4 Foundations.

Advanced Research Methods Unit 3 Reliability and Validity.

1 Measurement and Data Collection  What and How?  Types of Scales Nominal Nominal Ordinal Ordinal Interval Interval Ratio Ratio.

Validity Validity: A generic term used to define the degree to which the test measures what it claims to measure.

RELIABILITY AND VALIDITY OF ASSESSMENT

Evaluating Survey Items and Scales Bonnie L. Halpern-Felsher, Ph.D. Professor University of California, San Francisco.

Research Methodology and Methods of Social Inquiry Nov 8, 2011 Assessing Measurement Reliability & Validity.

Copyright © 2008 Wolters Kluwer Health | Lippincott Williams & Wilkins Chapter 17 Assessing Measurement Quality in Quantitative Studies.

Reliability: The degree to which a measurement can be successfully repeated.

MEASUREMENT. MeasurementThe assignment of numbers to observed phenomena according to certain rules. Rules of CorrespondenceDefines measurement in a given.

MEASUREMENT: PART 1. Overview  Background  Scales of Measurement  Reliability  Validity (next time)

Reliability and Validity Themes in Psychology. Reliability Reliability of measurement instrument: the extent to which it gives consistent measurements.

DENT 514: Research Methods

Reliability and Validity in Testing. What is Reliability? Consistency Accuracy There is a value related to reliability that ranges from -1 to 1.

Reliability a measure is reliable if it gives the same information every time it is used. reliability is assessed by a number – typically a correlation.

Validity & Reliability. OBJECTIVES Define validity and reliability Understand the purpose for needing valid and reliable measures Know the most utilized.

Measurement and Scaling Concepts

VALIDITY What is validity? What are the types of validity? How do you assess validity? How do you improve validity?

1 Measurement Error All systematic effects acting to bias recorded results: -- Unclear Questions -- Ambiguous Questions -- Unclear Instructions -- Socially-acceptable.

Ch. 5 Measurement Concepts.

Product Reliability Measuring

Measurement: Part 1.

Tests and Measurements: Reliability

پرسشنامه کارگاه.

مركز مطالعات و توسعه آموزش دانشگاه علوم پزشكي كرمان

Reliability and validity

Presentation transcript:

Choosing tests for EEF evaluations – reliability and validity and other issues Steve Higgins & Carole Torgerson & School of Education, Durham University EEF Evaluators Conference, June 2013

The perfect test! Highly reliable … Internal consistency Inter-item /Item total correlation Split-Half Reliability Cronbach's Alpha (α) Test/Re-test Parallel forms & Split half Inter-rater Wonderfully valid … Translation validity Face (weak and strong versions) Content (CVI) Criterion validity Concurrent Predictive Construct-related validity Convergent Discriminant.. And eminently practical Short Easy to administer to large groups Cheap Quick and easy to mark/ get the data

Other issues Intervention alignment Measures what is taught versus a good measure of school learning (predictive validity) Standardisation Availability of data Sampling (compared with intervention focus) Recency Poor reliability can increase Type II errors (false negatives)

Diamond ranking Most important Least important

Competing priorities ReliableValidPractical Number of items Time to administer Cost Ease of marking and data entry Aligned to intervention

Other issues (Carole T) Choice of test(s) Possible test effects Blinding teachers? intervention designers (developers)? markers Teaching to the test Treatment inherent measures Developers/teachers attitudes to tests

Discussion Discuss any of these or other testing issues that have arisen in your evaluation How can EEF help?