Lecture 7: reliability & validity Aims & objectives –This lecture will explore a variety of techniques for ensuring that research is conducted with reliable and valid measures –You should understand internal and external reliability and a variety if techniques for ensuring validity

Definitions ‘Reliability is the agreement between two efforts to measures the same trait through maximally similar methods. Validity is represented in the agreement between two attempts to measure the same trait through maximally different methods.’ (Campbell & Fiske, 1959) Reliability is a necessary but not sufficient condition for validity

Types of research/practice Clinical, educational, research, forensic, occupational Questionnaire development and use –Private, face to face Interviews –Face to face, telephone

Ethics/morals Need to ensure that the measures being used and reliable and valid Equal opportunities and cultural biases Stability over time

Classical theory of error measurement ACTUAL score = TRUE score + Error score Standard error of measurement Universe of items All items correlate to some extent with the true score Reliability s related to the average correlation between items and test length

Reliability Internal –Coefficient alpha –Split halves –Parallel forms External –Test-retest (correlations and ANOVA) Inter-rater –Kappa –Agreement does not imply accuracy Intra-rater

Sources of unreliability Guessing Ambiguous items Test length Instructions Temperature, illness Item order effects Response rate Social desirability

Item generation Past research From subject populations Unambiguous Address a specific issue Simple language No jargon Clear instructions

Context effects Affected by the nature of the questions, the order of questions and the type of response scale use. Example (availability & accessibility) –Control: General > dating (r =.12) –Prime: Dating > general (r =.66)

Validity Faith Face Content Construct Predictive

A model Task Global Specific Mood Retrieve info Use mood Select comparison Social aspect Private Public

Generalizability theory Basically, a test once shown to be reliable is not always going to be reliable –Culture changes –Time of day effects –Time of year effects Use ANOVA procedure to show under what circumstances a test is reliable