 Degree to which inferences made using data are justified or supported by evidence  Some types of validity ◦ Criterion-related ◦ Content ◦ Construct  All part of unitarian view of validity  Constructs - theoretical abstractions aimed at organizing and making sense of our environment; they are LATENT

 A criterion is any variable you wish to explain and/or predict  They are the key to well-developed theory, good measurement, and strong research design  Ultimate criterion  Multidimensional nature of criteria  Intermediate criteria

 Process of establishing a relationship between variables  Predictive, concurrent, postdictive  Usually based on correlation or regression equation  Low reliability will attenuate or mask relationships

 Selection Ratio – proportion of the individuals in the sample who are selected of the total number  Base rate – percent of successful individuals under random selection  Range Restriction  Differential Prediction for different subgroups

FN FP VP VN XcXc YcYc FN+VP=BR VN+FP=1-BR VP+FP=SR FN+VN=1-SR Successful Unsuccessful RejectAccept False Negatives False Positives

FN FP VP VN XcXc YcYc FN+VP=BR VN+FP=1-BR VP+FP=SR FN+VN=1-SR Successful Unsuccessful RejectAccept False Negatives False Positives

FN FP VP VN XcXc YcYc FN+VP=BR VN+FP=1-BR VP+FP=SR FN+VN=1-SR Successful Unsuccessful RejectAccept False Negatives False Positives

FN FP VP VN XcXc YcYc FN+VP=BR VN+FP=1-BR VP+FP=SR FN+VN=1-SR

 Even low correlations can lead to large increases in selection efficiency  SR and BR have strong influences  When SR is small (choose few), fewer FP and more FN  When SR is large, fewer FN and more FP  When BR is large (many can be successful), SR and validity have little effect on selection efficiency  Most gains in success ratio when BR =.50 and SR is small (e.g.,.10)  The tradeoffs depend on purpose of selection

X Y - Direct - Indirect - Ambiguous

X Y Same prediction for each group

X Y Different prediction for each group

 Extent to which items or measures cover the content area the test purports to measure ◦ Expert judges determine if a measure came from a particular content domain ◦ Scoring and content is based upon theory ◦ If measures are from same content domain, should demonstrate high reliability ◦ If low internal consistency reliability, low content validity

 Validity of inferences about latent unobserved variables on the basis of observed variables  Does a measure assess what it is intended to assess? Do the variables relate in theoretically meaningful ways?  Low reliability will make it difficult to assess the nature of a particular construct and attenuate relationships with other constructs

Construct Validity Can we generalize to the constructs from the measures? Theory What you think Cause Construct Effect Construct Measure or Manipulation Observed Outcomes Observed Relationship True Relationship What you see

Anxiety Test Score (Y) Measure of Anxiety (X) Ability to Learn Salads Eaten (Z) Vegetarianism 5

 Internal Structure Analysis  Cross Structure Analysis  Nomological network (Cronbach & Meehl)

 Factor Analysis ◦ Used to identify factors or dimensions that underlie relations among observed variables  Exploratory - Useful When: ◦ No info on internal structure available ◦ Factor structures may look different than original scale ◦ You have reservations about previous factor analyses  Confirmatory - Useful When: ◦ You have some idea of the internal structure ◦ Confirming factor structures from previous studies  Necessary but not sufficient to establish construct validity

Ability to Learn Z1Z1 Z2Z2 Z3Z3 Anxiety X1X1 X2X2 X3X3 X4X4 e1e1 e3e3 e4e4 e5e5 e6e6 e7e7 e2e2

 Embedded in nomological network (nomological validity)  Test of hypotheses by examining relationships between different indicators of underlying constructs ◦ e.g., leadership style based on reports from subordinates and leadership self-report inventory  Relies on multiple methods of measurement

 A representation of constructs of interest in a study, their observable manifestations (measures), and the interrelationships among and between them  Cronbach & Meehl said this is necessary to establish construct validity  Elements include: ◦ Specify linkage between constructs (hypotheses) ◦ Operationalize constructs (specify measurement)

 Convergent validity - Convergence among different methods designed to measure the same construct  Discriminant validity - Distinctiveness of constructs, demonstrated by divergence of methods designed to measure different constructs  Multi-Trait Multi-Method

 Heterotrait-Monomethod ◦ Different traits, same method  Heterotrait-Heteromethod ◦ Different traits, different methods  Monotrait-Heteromethod ◦ Same trait, different methods ◦ Validity diagonals  Monotrait-Monomethod ◦ Same trait, same method ◦ Reliability diagonals

Method1Method2Method3 A1 B1 C1A2 B2 C2A3 B3 C3 M1 A1 (.89) B1.51 (.89) C (.76) M2 A (.93) B (.94) C (.84) M3 A (.94) B (.92) C (.85)

 Specify the nomological net (expected + and - relationships) of expected relations  Establish reliability  Check convergence with other preexisting measures of the construct (convergent validity)  Factor analysis  Empirical studies of relatedness  Empirical studies of discriminability

 Take the hypotheses you developed in assignment 2 and the variables that were included in them. ◦ Draw a picture of what you believe the nomological network of these variables would look like ◦ What alternative measures of each variable might you use (different than those specified in Assignment 3) to establish convergent validity? ◦ Draw what an MTMM construct validity chart would look like that includes each variable in your study and the original and alternative measures you identified for each construct. Specify whether each correlation would be expected to be Hi, Low or Moderate.