MGMT 588 Research Methods for Business Studies Research Validity and Reliability EMU – Spring 2017

Objectives What are validity and reliability in a research process and why are they so critical? How can validity and reliability be assessed in a study? What are the different types of validity and reliability?

The quality of a research process can be evaluated and judged through its adequacy. The two quality standards, often referred as psychometric properties, assessed in this regard are validity and reliability. The questions usually raised by researchers: “how do we know that we are indeed measuring what we want to measure?” “can we be sure that if we repeated the measurement we will get the same result?”

A research adequacy heavily relies on the validity and reliability of the data/construct/scale.

Validity Validity is the degree to which: the data collection method meaningfully and accurately measures what it is supposed to measure, the research results are effectively about what they admit to be about. Researchers usually consider: internal, external, criterion-related, construct and content validity.

Criterion-related validity Types of validity Validity Translation validity Face validity Content validity Criterion-related validity Predictive validity Concurrent validity Construct validity Convergent validity Discriminant validity

Types of validity Internal validity: refers to the validity of the research itself. (e.g. exploring university students ease of integration in North Cyprus community, but surveying only Turkish students. Will the result properly indicate ALL students’ ease of integration? External validity: refers to the degree to which an inferred causal relationship can be generalized to other environments, time and people. The generalizability of a given relationship findings depends heavily on the extent to which it can be replicated across populations (not only target populations.

Types of validity Translation validity: evaluation of the degree to which constructs are accurately operationalized through the use of subjective judgment (face validity) and examination of content domain (content validity). Content validity: “a qualitative type of validity where the domain of the concept is made clear and the analyst judges whether the measures fully represent the domain” (Bollen, 1989, p.185). The empirical indicators should be proven to be logically and theoretically related to the construct. Face validity: subjective judgment on a construct operationalization (e.g.: choosing to measure an employee work efficiency based on punctuality only).

Types of validity Criterion-related validity: the extent of correspondence between a test measure and one or more external referents (i.e. criteria). A researcher may ask employee to report their salaries. The survey validity can be established if the self-reported salaries (test) correlates with the actual salaries records (external–and ideal–criterion) Concurrent validity: ability of a test to predict a present time event/outcome; the measure and the criterion co-exist at the same time. Predictive validity: ability of a test to measure a future time event/outcome (criterion); e.g.: student GMAT score and MBA program GPA/completion)

Types of validity Construct validity: the empirical assessment of the degree to which empirical indicators adequately measure the construct. Convergent validity: “measures of constructs that theoretically should be related to each other are, in fact, observed to be related to each other (i.e., you should be able to show a correspondence or convergence between similar constructs) Discriminant validity: measures of constructs that theoretically should not be related to each other are, in fact, observed to not be related to each other (i.e., you should be able to discriminate between dissimilar constructs)

Convergent validity © Trochim (2008)

Discriminant validity © Trochim (2008)

© Trochim (2008)

Types of validity Convergent and discriminant validity of a measurement can be assessed through: Exploratory factor analysis (EFA) Confirmatory factor analysis (CFA) Correlation matrix (with Campbell & Fiske (1959) multitrait – multimethod matrix)

Reliability Reliability refers to the stability and consistency of a measurement findings over different time, conditions, observations. Similar findings are expected to be yielded by different researchers

WLJ Composite Reliability Equivalence Alternative form Internal consistency Cronbach’s alpha WLJ Composite Reliability Inter – rater Split–Half Stability over time Test – Retest

Reliability There are several methods of evaluating a measure reliability: Test – retest : measure of a variable at two temporal points (t and t+1) with the same scale and sample. The correlation between the scores from the identical time – lapsed tests will determine the test-retest reliability

Reliability Alternative form: measure of a variable at time (t) with one instrument and at time (t2) with a different instrument. Both time using the same sample Split-half: given the number of indicators available to measure a variable, a half is split off to create a new measure, and the other half the second new measure for the same behavior The corrected correlation between the two halves provides the whole measure reliability.

Reliability Internal consistency: refers to the estimates of reliability based on the average intercorrelations among all the single items of the instrument/scale. Higher inter-item correlations are associated with high reliability score (Cronbach’s) alpha coefficient: the most popular reliability estimate ranges from 0 to 1, with the highest α meaning the greater reliability. Higher coefficients are associated with increasing number of items at a certain point. A scale reliability accepted threshold is 0.70 (Nunnally, 1978)

Reliability Werts, Linn & Jöreskog (WLJ) composite reliability : index derived from the proportion of variance attributable to only the latent variable (measurement errors excluded) in a CFA It ranges similarly to Chronbach’s α from 0 to 1, with a high estimate related to greater reliability.

Reliability, Validity and Measurement Errors In behavioral or business research, the instruments should not only consider the measurement of the construct they reflect, but also the arising measurement errors. Measurement errors are divided into random error and systematic error.

Reliability, Validity and Measurement Errors Random error happens unpredictably and that creates variations across repeated measures. Random error is inversely associated to a measure reliability and can generate inaccurate results.

Reliability, Validity and Measurement Errors Systematic error: fluctuations that that vary in a consistent manner, or are slanted in a particular direction. Systematic error is inversely associated to a measure validity and is also know as bias.

Reliability, Validity and Measurement Errors Reliability is a necessary but not sufficient condition for validity. A measure can be reliable (i.e. high consistency) but not valid.

Reliability, Validity and Measurement Errors In the research process, errors can stem from several sources: Variations in other forms of the test, Situational factors affecting the behavior of subjects under study, Approaches used by different examiners (such as the use of a common method of data collection – leading to key- informant bias or common method variance Other factors of influence

References Trochim, W.M. (2008). Convergent & Discriminant Validity. Retrieved from: http://www.socialresearchmethods.net/kb/convdisc.php O'Leary-Kelly, S. W., & Vokurka, R. J. (1998). The empirical assessment of construct validity. Journal of Operations Management, 16(4), 387-405. Drost, E. A. (2011). Validity and reliability in social science research. Education Research and Perspectives, 38(1), 105-123. Altinay, L., & Paraskevas, A. (2008). Planning research in hospitality and tourism. Burlington, MA: Butterworth- Heinemann/Elsevier. Nunnally, J. (1978). Psychometric methods. New York: McGraw- Hill.