Reliability, Validity, Trustworthiness If a research says it must be right, then it must be right,… right??

Validity/Reliability and Trustworthiness Why do we need validity and reliability in quantitative studies and “trustworthiness” in qualitative studies? We can’t trust the results if we can’t trust the methods!

Reliability What is reliability? What are different ways to establish reliability?

Reliability of Measurement Reliability - The extent to which measures are free from error Error is measured by consistency

Reliability of Measurement Types of reliability evidence –Test-retest –Equivalence (i.e. parallel form) –Internal consistency –Agreement / Inter-rater reliability

Validity of Measurement Definition of validity –The extent to which inferences are appropriate, meaningful, and useful –The extent to which a measurement (survey, test, etc.) measures what it actually purports to measure Current example – content tests and teacher licensure

Validity Content Predictive (criterion-related) Concurrent Construct

Validity of Measurement For research results to have any value, validity of the measurement of a variable must exist –Use of established and “new” instruments and the implications for establishing validity –Importance of establishing validity prior to data collection (e.g. pilot tests)

Thought Question Criticisms of standardized tests like the SAT claim that they discriminate against particular groups of students (especially minorities) and do not represent a broad enough domain of knowledge to adequately assess a student’s academic potential. What issue of validity is operating in these arguments?

Thought Question Other arguments against the SAT state that the tests do not adequately estimate an individual’s ability to succeed in college. What issue of validity is operating here?

Validity Concurrent –When more than one measure is used with similar results Construct –When a broad concept like “school climate” is measured using data consistent with what is known about the construct

Reliability & Validity of Measurement What is the relationship of reliability to validity? – If a watch consistently gives the time at 1:10 when actually it is 1:00, it is ____ but not ____. –______ is necessary but not sufficient condition for _______. – To be _____, an instrument must be ______, but a ____ instrument is not necessarily _____.

Ways to Establish Trustworthiness Credibility –Prolonged engagement –Triangulation –Member checking –Peer debriefing Dependability –Triangulation –Multiple coders

Ways to Establish Trustworthiness Transferability –Thick description –Purposive sampling Confirmability –Member checking –Peer debriefing

Review Why is research/science an important way of knowing? What is the point of the “curious skeptic”? What is empirical research? How is empirical research different from theories? What are 2 different kinds of theories we talked about in class and what is the difference?

Review What is a hypothesis? What is the null hypothesis and why is there such a thing as a null hypothesis? What is a variable? In what kind of research will you see variables identified up front? What are some qualities of research problems? What are some major differences between qualitative and quantitative research?

Review Why is sampling important? How is sampling different in qualitative and quantitative research? What is a “target population”? What are different kinds of sampling in quantitative research? What are different kinds of sampling in qualitative research?

Midterm Triplett article critique (35%). Come to class with this done. Multiple choice/matching (45%) Short answer (20%)