1. Assessing the Quality of Research
What is validity?
Types of Validity
Examples in the Measurement of
Height & Weight
Learning Style Orientation

2. Validity Validity Reliability
Evidence that a measure assesses the construct/concept accurately and in a meaningful way
Reliability
That a measure is consistent in assessing the construct

3. Corr b/w Objective (O) & Self-Reports (SR) of Height (H) & Weight (W)
O-H
SR-H
O-W
SR-W
1.00
.98
.55
.56
.68
.69
.92

4. Validity vs. Reliability
Reliability is a necessary but not a sufficient condition for validity
E.g. A measuring tape to is not a valid way to measure weight although the tape reliably measures height and height correlates w/weight

5. Types of Validity Construct Validity Criterion Validity Content
Predictive
Validity
Concurrent
Validity
Convergent
Validity
Discriminant
Validity
Adapted from Sekaran, 2004

6. Content Validity
Extent to which items on the measure are a good representation of the construct
e.g., Is your job interview based on what is required for the job?
Can be based on judgments of researcher or independent raters
e.g., Expert (supervisors, incumbents) rating of job relevance of interview questions

7. An Example of How Content Validity of the Learning Style Orientation Measure is Established
112 items derived from 2 procedures based on theory about learning events…
Ps generated critical incidents of learning events
Two types of learning events: theoretical, practical (see next slide for examples)
Two types of outcomes=success, failure
4 events from each of 67 participants
Ps indicated yes/no to action & reflection oriented statements

8. Examples of theoretical & practical learning events

9. Obtaining Data on “Content Valid” Items Generated Qualitatively
(aka Item Development Phase Study)
154 Ps rated 112 items on 5 point Likert scale agree/disagree type statements like
I like problems that don’t have a definitive solution
I like to put new knowledge to immediate use

10. Feedback on method section
Describing vs. including the questionnaire
Specific
Relevant
Graded on irrelevant details
What is irrelevant detail??

11. Quantitative Analyses of “Content Valid” Items Generated Qualitatively
Ps responses factor analyzed
5 factor solution (i.e., 5 dimensions)
What is factor analyses? Demo if time permits
Retained 54 items of 112 original
54 items sorted for content by 8 grad students blind to number and types of dimensions

12. Simplifying what the factor analyses of the 54 items mean
Computed sub-scales based on factor analyses & found high reliabilities
Computed Correlations b/w the 5 factors
Range from .01 to.32 (more on the implications of this later....)
Only 1 is significant
Follow up with a more stringent test by replicate 5 factors with new data using Confirmatory Factor analytic technique

13. Further Validating the Learning Style Orientation Measure in a follow-up study
Ps complete the
new LSOM
old LSI (competitor/similar construct)
Personality (firmly established related construct as per theory)

14. Results demonstrating the Content Validity of LSOM in the second study
Confirmatory factor analysis shows 5-dimensions re-extracted with new data
More sophisticated than just demonstrating high reliability of sub-scales
Comparing reliabilities of LSOM subscales =.74 to .87 to reliabilities of…
Old learning style subscales=.83 to .86
Personality subscales=.86 to .95

15. Implications of Content Validity Analyses of the LSOM
Not firmly established that LSOM is something different and/or better than LSI

16. What you learned so far What is validity
How is it different from reliability?
Learning Check in the Essays data how will you establish validity?
One type of validity is content Validity
How to establish content validity?
Dual Career Relationship measure
What are limitations of with the notion of content validity

17. What’s next… Types of Validity Construct Validity Criterion Validity
Content
Validity
Predictive
Validity
Concurrent
Validity
Convergent
Validity
Discriminant
Validity
Adapted from Sekaran, 2004

18. Extent to which a new measure relates to another known measure
Criterion Validity
Extent to which a new measure relates to another known measure
Demonstrated by the validity coefficient
Correlation between the new measure and a known measure
e.g., do scores on your job interview predict performance evaluation scores?
New terms to keep in mind
new measure=predictor
known measure=criterion

19. Predictive (Criterion) Validity
Scores on predictor (e.g., selection test) collected some time before scores on criterion (e.g., job performance)
Able to differentiate individuals on a criterion assessed in the future
Weaknesses
Due to management pressures, applicants can be chosen based on high scores on predictor leading to range restriction (demo)
Measures of job performance (highly tailored to predictor) are developed for validation

20. Concurrent (Criterion) Validity
Scores on predictor and criterion are collected simultaneously (e.g., police officer study)
Distinguishes between participants in sample who are already known to be different from each other
Weaknesses
Range restriction
Does not include those who were not hired/fired
Differences in test-taking motivation
Differences in experience
Employees vs. applicants bec. experience with job can affect scores on performance evaluation (i.e., criterion)

21. How to correct for range restriction
When full range of scores on any of the variables (predictor/criterion) we have range restriction
E.g. when there is range restriction on the predictor variable use unrestricted and restricted standard deviations of predictor variable & the observed correlations b/w predictor & criterion

22. Concurrent vs. Predictive Validity
Predictor & Criterion variable collected at the same vs. different times
For predictive, the predictor variable is collected before the criterion variable
Degree of range restriction is more vs. less

23. Example of Criterion Validity Learning Style Orientation Measure
Additional variance explained by new LSOM vs. old LSI on criteria (i.e., preferences for instruction & assessment) DV
LSOM
LSI
Subjective assessment
.15
.01
Interactional instruction
.21
.04
Informational instruction
.06
.00

24. Types of Validity Construct Validity Criterion Validity Content
Predictive
Validity
Concurrent
Validity
Convergent
Validity
Discriminant
Validity
Adapted from Sekaran, 2004

25. Extent to which hypotheses about construct are supported by data
Construct Validity
Extent to which hypotheses about construct are supported by data
Define construct, generate hypotheses about construct’s relation to other constructs
Develop comprehensive measure of construct & assess its reliability
Examine relationship of new measure of construct to other similar & dissimilar constructs (using different methods)
Examples: height & weight; Learning Style Orientation measure

26. 2 ways of Establishing Construct Validity
Different measures of the same construct should be more highly correlated than different measures of different constructs (aka Multi-trait multi-method)
e.g., objective height & SR of height should be higher than Objective Height & and Objective Weight
Different measures of different constructs should have lowest correlations
E.g., Objective Height & Subjective Weight

27. Correlations between Objective (O) & Self-Reports (SR) of Height & Weight
O-H
SR-H
O-W
SR-W
1.00
.98
.55
.56
.68
.69
.92

28. Convergent Validity Coefficients
Absolute size of correlation between different measures of the same construct
Should be large, significantly diff from zero,
Example of Height & Weight
Objective and subjective measures of height are correlated .98
Objective and subjective measures of weight are correlated .92

29. Discriminant Validity Coefficients
Relative size of correlations between the same construct measured by different methods should be higher than
Different constructs measured by same method
Different constructs measured by different methods

30. Using the Example of Different Measures of Height & Weight to understand Discirminant Validity

31. Discriminant Validity Across Constructs
STRONG CASE: Are the correlations b/w the same construct measured by different methods significantly higher than corr b/w different constructs measured by same methods
Note: Objective measures of height & weight are corr .55 & Subjective measures of height & weight are corr .69
So to establish strong case, establish that .92 & .98 are significantly greater than .55 & .69?
Not enough to visually compare, need to convert rs to z scores and check in z table

32. Discriminant Validity Across Measures
WEAK CASE: Are the correlations b/w the same construct measured by different methods significantly different from corr b/w different constructs measured by different methods
Note: Objective height & subjective weight are corr .68 & Subjective height & objective weight are corr .56
So to establish weak case, demonstrate that .92 & .98 are significantly higher from .56 & .68 (after converting rs to z scores and comparing z-s)

33. Types of Validity Construct Validity Criterion Validity Content
Predictive
Validity
Concurrent
Validity
Convergent
Validity
Discriminant
Validity
Adapted from Sekaran, 2004

34. Using the LSOM Item Development Study
(aka Study 1) to understand Construct Validity

35. Recall, the 2 ways of Establishing Construct Validity
Different measures of the same construct should be more highly correlated than different measures of different constructs (aka Multi-trait multi-method)
e.g., subscales of LSOM should be correlated higher than corr b/w LSOM & personality
Different measures of different constructs should have lowest correlations
E.g., corr b/w LSOM & Personality

36. Convergent Validity of LSOM in The Item Development Study
Established via
High reliabilities of subscales of LSOM ( )
Correlations b/w different measures (subscales) of learning style =.01 to.32 should be somewhat significant (not too high and not too low)
Note only 1 corr is significant (could be due to sample size?) so weak support for convergent validity of new LSOM in Study 1 & conducted second validation study

37. Discriminant Validity in the LSOM Item Development Phase
Correlations between different measures of different constructs (i.e., Learning Style & personality) .42 to .01 should be lower than and significantly different from correlations between different measures of same construct (i.e., subscales of learning style) .01 to .32

38. Conclusions from LSOM Item Development Phase Study
Convergent & Discriminant validity is not established sufficiently researchers collected additional data to firmly establish the validation of the measure

39. Examining the LSOM Validation Study to understand Construct Validity

40. Method & Procedure of the Validation Study
Ps complete the
new LSOM (predictor)
old LSI (competitor/similar construct)
Personality (related construct as per theory)
Preferences for instructional & assessment methods (criterion)

41. Convergent Validity of the LSOM in the Validation Study
To examine the correlation (r) b/w similar measures of key construct compare the correlations b/w the different subscales (measures) of new learning style 01 to .23 to
r b/w similar measures of other similar & dissimilar constructs in the study
Similar constructs=Different subscales of old learning style .23 to .40
Dissimilar constructs= Diff subscales of personality .01 to .27

42. Discriminant Validity of the LSOM in the Validation Study
Examine Correlations (r) between measures of similar constructs
r between new learning style subscales & old learning style = .01 to .31
Examine r b/w measures of different constructs
r b/w new learning style & personality subscales is .01 to .55
r b/w old learning style & personality subscales= .02 to .38

43. Criterion Validity can be an indirect way of establishing Construct Validity

44. Establishing Better Criterion Validity of LSOM
Additional variance explained by new LSOM vs. old LSI on criteria (i.e., preferences for instruction & assessment) DV
LSOM
LSI
Subjective assessment
.15
.01
Interactional instruction
.21
.04
Informational instruction
.06
.00

45. What you learned today
Kind of evidence you should look for when deciding on what measures to use
Content Validity
Criterion Validity
Concurrent vs. Predictive
Construct validity
Convergent & Discriminant

46. Implications of What you learned today for your Method Section
Did you examine relevant sources to establish validity of your measures?
How will you report that information?