1. MGMT 588 Research Methods for Business Studies
Research Validity and Reliability
EMU – Spring 2017

2. Objectives
What are validity and reliability in a research process and why are they so critical?
How can validity and reliability be assessed in a study?
What are the different types of validity and reliability?

3. The quality of a research process can be evaluated and judged through its adequacy.
The two quality standards, often referred as psychometric properties, assessed in this regard are validity and reliability.
The questions usually raised by researchers:
“how do we know that we are indeed measuring what we want to measure?”
“can we be sure that if we repeated the measurement we will get the same result?”

4. A research adequacy heavily relies on the validity and reliability of the data/construct/scale.

5. Validity Validity is the degree to which:
the data collection method meaningfully and accurately measures what it is supposed to measure,
the research results are effectively about what they admit to be about.
Researchers usually consider:
internal, external, criterion-related, construct and content validity.

6. Criterion-related validity
Types of validity
Validity
Translation
validity
Face validity
Content validity
Criterion-related validity
Predictive validity
Concurrent validity
Construct validity
Convergent validity
Discriminant validity

7. Types of validity
Internal validity: refers to the validity of the research itself. (e.g. exploring university students ease of integration in North Cyprus community, but surveying only Turkish students. Will the result properly indicate ALL students’ ease of integration?
External validity: refers to the degree to which an inferred causal relationship can be generalized to other environments, time and people. The generalizability of a given relationship findings depends heavily on the extent to which it can be replicated across populations (not only target populations.

8. Types of validity
Translation validity: evaluation of the degree to which constructs are accurately operationalized through the use of subjective judgment (face validity) and examination of content domain (content validity).
Content validity: “a qualitative type of validity where the domain of the concept is made clear and the analyst judges whether the measures fully represent the domain” (Bollen, 1989, p.185).
The empirical indicators should be proven to be logically and theoretically related to the construct.
Face validity: subjective judgment on a construct operationalization (e.g.: choosing to measure an employee work efficiency based on punctuality only).

9. Types of validity
Criterion-related validity: the extent of correspondence between a test measure and one or more external referents (i.e. criteria).
A researcher may ask employee to report their salaries. The survey validity can be established if the self-reported salaries (test) correlates with the actual salaries records (external–and ideal–criterion)
Concurrent validity: ability of a test to predict a present time event/outcome; the measure and the criterion co-exist at the same time.
Predictive validity: ability of a test to measure a future time event/outcome (criterion); e.g.: student GMAT score and MBA program GPA/completion)

10. Types of validity
Construct validity: the empirical assessment of the degree to which empirical indicators adequately measure the construct.
Convergent validity: “measures of constructs that theoretically should be related to each other are, in fact, observed to be related to each other (i.e., you should be able to show a correspondence or convergence between similar constructs)
Discriminant validity: measures of constructs that theoretically should not be related to each other are, in fact, observed to not be related to each other (i.e., you should be able to discriminate between dissimilar constructs)

11. Convergent validity
© Trochim (2008)

12. Discriminant validity
© Trochim (2008)

13. © Trochim (2008)

14. Types of validity
Convergent and discriminant validity of a measurement can be assessed through:
Exploratory factor analysis (EFA)
Confirmatory factor analysis (CFA)
Correlation matrix (with Campbell & Fiske (1959) multitrait – multimethod matrix)

15. Reliability
Reliability refers to the stability and consistency of a measurement findings over different time, conditions, observations.
Similar findings are expected to be yielded by different researchers

16. WLJ Composite Reliability
Equivalence
Alternative form
Internal consistency
Cronbach’s alpha
WLJ Composite Reliability
Inter – rater
Split–Half
Stability over time
Test – Retest

17. Reliability
There are several methods of evaluating a measure reliability:
Test – retest : measure of a variable at two temporal points (t and t+1) with the same scale and sample.
The correlation between the scores from the identical time – lapsed tests will determine the test-retest reliability

18. Reliability
Alternative form: measure of a variable at time (t) with one instrument and at time (t2) with a different instrument. Both time using the same sample
Split-half: given the number of indicators available to measure a variable, a half is split off to create a new measure, and the other half the second new measure for the same behavior
The corrected correlation between the two halves provides the whole measure reliability.

19. Reliability
Internal consistency: refers to the estimates of reliability based on the average intercorrelations among all the single items of the instrument/scale. Higher inter-item correlations are associated with high reliability score
(Cronbach’s) alpha coefficient: the most popular reliability estimate ranges from 0 to 1, with the highest α meaning the greater reliability. Higher coefficients are associated with increasing number of items at a certain point.
A scale reliability accepted threshold is 0.70 (Nunnally, 1978)

20. Reliability
Werts, Linn & Jöreskog (WLJ) composite reliability : index derived from the proportion of variance attributable to only the latent variable (measurement errors excluded) in a CFA
It ranges similarly to Chronbach’s α from 0 to 1, with a high estimate related to greater reliability.

21. Reliability, Validity and Measurement Errors
In behavioral or business research, the instruments should not only consider the measurement of the construct they reflect, but also the arising measurement errors. Measurement errors are divided into random error and systematic error.

22. Reliability, Validity and Measurement Errors
Random error happens unpredictably and that creates variations across repeated measures. Random error is inversely associated to a measure reliability and can generate inaccurate results.

23. Reliability, Validity and Measurement Errors
Systematic error: fluctuations that that vary in a consistent manner, or are slanted in a particular direction. Systematic error is inversely associated to a measure validity and is also know as bias.

24. Reliability, Validity and Measurement Errors
Reliability is a necessary but not sufficient condition for validity. A measure can be reliable (i.e. high consistency) but not valid.

25. Reliability, Validity and Measurement Errors
In the research process, errors can stem from several sources:
Variations in other forms of the test,
Situational factors affecting the behavior of subjects under study,
Approaches used by different examiners (such as the use of a common method of data collection – leading to key- informant bias or common method variance
Other factors of influence

26. References
Trochim, W.M. (2008). Convergent & Discriminant Validity. Retrieved from: O'Leary-Kelly, S. W., & Vokurka, R. J. (1998). The empirical assessment of construct validity. Journal of Operations Management, 16(4), Drost, E. A. (2011). Validity and reliability in social science research. Education Research and Perspectives, 38(1), Altinay, L., & Paraskevas, A. (2008). Planning research in hospitality and tourism. Burlington, MA: Butterworth- Heinemann/Elsevier. Nunnally, J. (1978). Psychometric methods. New York: McGraw- Hill.