Reliability, validity, and scaling

Basics What is reliability? What is validity? Can you have one without the other? Why or why not?

Reliability Observed score = True score + random error + systematic error. Explain. Why is this important? How can you decrease these errors? What does a reliability estimate of .85 tell you? What do we want our reliability estimates to be?

Types of Reliability What are each, how do you calculate them, when would you use them, and how can you increase them? Inter-rater reliability Test-retest reliability Parallel-forms reliability

Internal consistency reliability What are each, how do you calculate them, what do they tell you, where do you want values to be, and how can you increase them? Average inter-item correlation Average item-total Split-half reliability Cronbach’s alpha Kuder-Richardson Formula 20 (KR-20)

Cronbach’s alpha a Schmitt, 1996 Write down what you knew about alpha before reading the article List 2 things you learned from the article

Uses and abuses of alpha What is the difference between internal consistency and homogeneity? Which does alpha tell you? What increases alpha? What is a problem with using alpha to correct correlations for reliability? Why is a high alpha not necessarily a good thing?

Sample SPSS analyses

Construct Validity How does construct validity relate to internal and external validity? What are each of these, how would you calculate them, and what do they tell you? Translational validity Face validity Content validity Criterion-related validity Predictive validity Concurrent validity (aka known groups) Convergent validity Discriminant validity/Divergent validity How high/low should your correlations be?

Multitrait-multimethod matrix (MTMM) Nomological network Cronbach & Meehl, 1955 MTMM Campbell & Fiske, 1959 Look at example p. 69 What information does this give you? Pattern matching Advantages/disadvantages How can SEM be used to show this?

Design Threats to construct validity What are these, and how can the problem be decreased? Inadequate preoperational explication of constructs Mono-operation bias Mono-method bias Interaction of different treatments Interaction of testing and treatment Restricted generalizability across constructs Confounding constructs and levels of constructs

Social threats to CV What are these, and how can the problem be decreased? Hypothesis guessing Evaluation apprehension Experimenter expectancies Other threats Social desirability Response styles Demand characteristics

Method variance/Method bias Podsakoff, MacKenzie, & Podsakoff, 2012 What is it? What are some types and causes? What effects does it have? Why does it have these effects?

How can you deal with method bias? Use more than one method—get predictor and criterion from different sources Separate measures temporally, proximally, or psychologically Use different types and points on scales Label all points on scales, and make items less ambiguous Decrease tendencies for socially desirable responses Reverse score items Try to increase motivation and ability of participants Control for bias statistically (several options) Make it a manipulation or look for interaction effects

Controlling for other variables Westfall & Tarkoni, 2016 What is the main point of this article?

Statistical control Ways to test: Regression methods for control Incremental validity If 2 measures are distinct If 1 measure is “better” than another Regression methods for control What is the problem with using this method? When will it be more of a problem? How can you correct this problem?

Table 1. Type 1 error rates for a few parameter combinations. Westfall J, Yarkoni T (2016) Statistically Controlling for Confounding Constructs Is Harder than You Think. PLOS ONE 11(3): e0152719. https://doi.org/10.1371/journal.pone.0152719 http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0152719

Fig 12. Power to detect incremental validity using SEM. Westfall J, Yarkoni T (2016) Statistically Controlling for Confounding Constructs Is Harder than You Think. PLOS ONE 11(3): e0152719. https://doi.org/10.1371/journal.pone.0152719 http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0152719

Levels of measurement Why do they matter? What are the four types? Examples? Stats that can be run on them? Nominal Ordinal Interval Ratio What types of scales do we typically use in psychology? Is that a problem?

Basics of scaling Scales vs. response scales vs. index vs. questionnaire (every set of questions is not a scale) What are unidimensional vs. multidimensional scales? When should you use one vs. the other?

Types of (Unidimensional) scales Thurstone (method of equally appearing intervals) Generate items Have judges rate them Choose ones that represent the whole scale Guttman (cumulative scale) Coefficient of reproducibility Likert (summative rating scale) Semantic differential scale What are the advantages and disadvantages of each? How do you score each?

Thurstone I believe the church is the greatest institution in America today. (.2) I believe in religion, but I seldom go to church. (5.4) I believe in sincerity and goodness without any church ceremonies. (6.7) I believe the church is a hindrance to religion for it still depends on magic, superstition, and myth. (9.6) I think the church is a parasite on society. (11.0)

Guttman I am more than 54 inches tall. I am more than 56 inches tall.

Other issues with scaling Standardization Norms

Steps To choosing a scale to use? To creating a scale?

Steps to creating an index Conceptualize the index Operationalize and measure the components Develop the rules for calculating the index score (weighting?) Validate it!

Methods assignment #1

Next week Surveys ESM presentation Scale development assignment due Chapter Articles on data cleaning Short articles on survey design ESM presentation Scale development assignment due Decision about Feb. 28