1. Work in the 21st Century Chapter 2
Methods and Statistics
in I-O Psychology

2. Module 2.1: Science What is science?
Approach that involves the understanding, prediction, and control of some phenomenon of interest
Science has common methods
Science is a logical approach to investigation
Based on a theory, hypothesis, or basic interest
Science depends on data
Gathered in a laboratory or the field

3. Common Methods (cont'd)
Research must be communicable, open, & public
Research published in journals, reports, or books
1) Methods of data collection described
2) Data reported
3) Analyses displayed for examination
4) Conclusions presented

4. Common Methods (cont'd)
Scientists set out to disprove theories or hypotheses
Goal: Eliminate all plausible explanations except one
Scientists are objective
Expectation that researchers will be objective & not influenced by biases or prejudices

5. Role of Science in Society
Expert witnesses in a lawsuit
Permitted to voice opinions about organizational practices
Often a role assumed by I-O psychologists

6. Module 2.1 (cont'd) Why do I-O psychologists engage in research?
Better equip HR professionals in making decisions in organizations
Provide an aspect of predictability to HR decisions

7. Module 2.2: Common Research Designs in I-O Psychology
Table 2.1 Common Research Designs in I-O Psychology

8. Research Designs (cont'd)
Experimental
Random assignment of participants to conditions
Conducted in a laboratory or the workplace
Non-experimental
Does not include manipulation or assignment to different conditions
2 common designs:
Observational design: Observes and records behavior
Survey/Questionnaire design (most common)
Quasi-experimental
Non-random assignment of participants to conditions

9. Methods of Data Collection
Quantitative methods
Rely on tests, rating scales, questionnaires, & physiological measures
Yield results in terms of numbers
C. Borland/PhotoLink/Getty Images

10. Methods of Data Collection: Qualitative & Quantitative Research
Qualitative methods
Include procedures like observation, interview, case study, & analysis of written documents
Generally produce flow diagrams & narrative descriptions of events/processes
Quantitative methods
Rely on tests, rating scales, and physiological measures
Yield numerical results

11. Quantitative & Qualitative Research (cont’d)
Not mutually exclusive
Triangulation
Examining converging information from different sources (qualitative and quantitative research).

12. Generalizability in Research
Application of results from one study or sample to other participants or situations
The more areas a study includes, the greater its generalizability
Every time a compromise is made, the generalizability of results is reduced

13. Sampling Domains for I-O Research
Figure 2.1: Sampling Domains
for I-O Research

14. Control in Research Experimental control Statistical control
Eliminates influences that could make results less reliable or harder to interpret
Statistical control
Statistical techniques used to control for the influence of certain variables

15. Ethical Behavior in I-O Psychology
Ethical standards of the APA
SIOP book of 61 cases (Lowman, 1998)
Cases illustrate ethical issues that are likely to arise in I-O psychology
Joel Lefkowitz (2003) published a recent book on values and ethics in I-O psychology

16. Table 2.2 Potential Roles Available to the I-O Psychologist and Other HR Managers with Respect to Ethical Problems

17. Module 3: Data Analysis Descriptive statistics
Summarize, organize, describe sample of data
Frequency Distribution:
Horizontal axis = Scores running low to high
Vertical axis = Indicates frequency of occurrence

18. Describing a Score Distribution
Measures of central tendency
Mean
Mode
Median
Ryan McVay/Getty Images

19. Describing Score Distribution (cont'd)
Variability
Standard deviation
Lopsidedness or skew
Mean is affected by high or low
scores, median is not
Mean pulls in direction of skew
Ryan McVay/Getty Images

20. Descriptive Statistics: Two Score Distributions (N = 30)
Figure 2.2 Two Score Distribution (N = 30)

21. Two Score Distributions (N = 10)
Figure 2.3. Two Score Distributions (N = 10)

22. Inferential Statistics
Aid in testing hypotheses & making inferences from sample data to a larger sample/population
Include t-test, F-test, chi-square test

23. Statistical Significance
Defined in terms of a probability statement
Threshold for significance is often set at .05 or lower
Significance refers only to confidence that result is NOT due to chance, not strength of an association or importance of results.

24. Statistical Power
Likelihood of finding statistically significant difference when true difference exists
The smaller the sample size, the lower the power to detect a true or real difference

25. Concept of Correlation
Positive Linear Correlation
Figure 2.4
Correlation between
Test Scores and
Training Grades

26. Concept of Correlation (cont'd)
Scatterplot
Displays correlational relationship between 2 variables
Regression
Straight line that best “fits” the scatterplot and describes the relationship between the variables in the graph

27. Correlation Coefficient
Statistic or measure of association
Reflects magnitude (numerical value) & direction (+ or –) of relationship between 2 variables
Ranges from 0.00 and 1.00

28. Correlation Coefficient
Positive correlation → As one variable increases, other variable also increases & vice versa
Negative correlation → As one variable increases, other variable decreases & vice versa

29. Scatterplots of Various Degrees of Correlation
Figure Scatterplots of Various Degrees of Correlation

30. Curvilinear Relationship
If correlation coefficient is .00, one cannot conclude that there is no association between variables
A curvilinear relationship might better describe the association

31. Curvilinear Correlation
Figure 2.7
An Example of
a Curvilinear
Relationship

32. Multiple correlation coefficient
Overall linear association between several variables & a single outcome variable

33. Meta-Analysis
Statistical method for combining results from many studies to draw a general conclusion
Statistical artifacts
Characteristics of a particular study that distort the results
Sample size is typically the most influential statistical artifact

34. Module 4: Interpretation
Reliability
Consistency or stability of a measure
Test-retest reliability
Calculated by correlating measurements taken at Time 1 with measurements taken at Time 2

35. High and Low Test-Retest Reliability
Figure 2.8
Examples of High and Low Test-Retest Reliability: Score Distributions of Individuals Tested on Two Different Occasions

36. Equivalent forms reliability
Reliability (cont'd)
Equivalent forms reliability
Calculated by correlating measurements from a sample of individuals who complete 2 different forms of same test
Internal consistency
Assesses how consistently items of a test measure a single construct

37. Inter-rater reliability
Reliability (cont'd)
Inter-rater reliability
Can calculate various statistical indices to show level of agreement among raters
Values in the range of .70 to .80 represent reasonable reliability
Generalizability theory
Simultaneously considers all types of error in reliability estimates

38. Validity
Whether measurements taken accurately & completely represent what is to be measured
Predictor
Test chosen or developed to assess identified abilities or other characteristics (KSAOs)
Criterion
Outcome variable describing important performance domain

39. Figure 2.9: Validation Process from Conceptual and Operational Levels

40. Criterion-Related Validity
Correlate a test score (predictor) with a performance measure; resulting correlation often called a validity coefficient
Predictive validity design
Time lag between collection of test data & criterion data
Test often administered to job applicants

41. Criterion-Related Validity (cont'd)
Concurrent validity design
No time lag between collection of test data & criterion data
Test administered to current employees, performance measures collected at same time
Disadvantage: No data about those not employed by the organization

42. Content-Related Validity
Demonstrates that content of selection procedure represents adequate sample of important work behaviors & activities or worker KSAOs defined by job analysis
I-O Psychologists can use incumbents/SMEs to gather content validity evidence

43. Construct-Related Validity
Investigators gather evidence to support decisions or inferences about psychological constructs
Construct - concept or characteristic that a predictor is intended to measure; examples include intelligence, extraversion, and integrity

44. A Model for Construct Validity
Figure 2.10.
A Model for
Construct Validity

45. Construct Validity Model of Strength and Endurance Physical Factors
Figure 2.11