1. IMPRINT Developer’s Workshop December 6-7, 2005 Meta-analytic Reviews of the Effects of Temperature and Vibration on Performance J.L. Szalma & G. Conway University of Central Florida

2. IMPRINT Developer’s Workshop December 6-7, 2005 Performance under Stress Stress Degrades Performance Relation of Stress to IMPRINT: predict the degree of performance degradation under specific forms of stress Currently estimates may not be representative of the entire literature Estimates may not reflect interactions with other variables

3. IMPRINT Developer’s Workshop December 6-7, 2005 Performance under Stress Goal for Current work: Based on a quantitative review of the literature, derive estimations of the effect of specific stressors on human performance Outcome: Improve IMPRINT as a performance prediction tool by increasing the accuracy of the stress algorithms

4. IMPRINT Developer’s Workshop December 6-7, 2005 Why Use Meta-Analysis? Advantages Quantitative synthesis Controls for sampling error and low power Permits examination potential moderator variables Disadvantages Generally have to discard many published studies because they do not meet inclusion criteria Analysis is only as good as the data that go into it

5. IMPRINT Developer’s Workshop December 6-7, 2005 Significance Tests Significance tests are useful, but finding a statistically significant difference depends on the power of the analysis Many publications in behavioral science include statistical analyses of relatively low power It would be helpful to have a standardized measure of mean differences that indicates the strength of the effect independent of the results of a significance test and that can be compared across different measures

6. IMPRINT Developer’s Workshop December 6-7, 2005 The Effect Size Cohen’s d (Hedge’s g): Standardized Mean difference between two conditions Range: -∞ --------------- 0 --------------- +∞

7. IMPRINT Developer’s Workshop December 6-7, 2005 Magnitude of Effect Magnitude of Effect (Cohen, 1988) Large:.80 Medium:.50 Small:.20 These are only meant to be guidelines The magnitude of ‘large’ and ‘small’ effects can vary by domain

8. IMPRINT Developer’s Workshop December 6-7, 2005 General Procedures in Meta Analysis Literature Search Weighted Average Effect Size Variance Partitioning Moderator Analysis Hierarchical Moderator Analysis Criteria for inclusion

9. IMPRINT Developer’s Workshop December 6-7, 2005 Weighted Effect Sizes Not all effect sizes are equally accurate Analogy: Accuracy of a sample mean The sampling error associated with a given effect size can be estimated To obtain a more precise estimate of the average effect size, each d is weighted by the reciprocal of its error variance (Hedges & Olkin, 1985).

10. IMPRINT Developer’s Workshop December 6-7, 2005 Weighted Effect Sizes Thus, where

11. IMPRINT Developer’s Workshop December 6-7, 2005 Error variance Variance of the d’s Estimate of the ‘true’ variance in the population of effect sizes (random vs. fixed effects) Variance Partitioning σ 2 d = σ 2 δ + σ 2 e

12. IMPRINT Developer’s Workshop December 6-7, 2005 Heterogeneous Effect Sizes and Moderator Variables If the true weighted average effect size were constant across environments, then and

13. IMPRINT Developer’s Workshop December 6-7, 2005 If Then other variables are likely to moderate the magnitude and/or direction of the effect size associated with the variable of interest (temperature/vibration) Analogous to interaction effects in ANOVA Moderator Analyses

14. IMPRINT Developer’s Workshop December 6-7, 2005 Moderator Variable: Taxon Task Category (IMPRINT Taxon) Perception/discrimination Information Processing/Problem Solving Numerical Fine Motor Continuous Fine Motor Discrete

15. IMPRINT Developer’s Workshop December 6-7, 2005 Moderator Variable: Dependent Measure Speed Accuracy

16. IMPRINT Developer’s Workshop December 6-7, 2005 Moderator Variable: Intensity Range of Temperature, Heat (Intensity) Less than 85°F ET Greater than 85°F ET Range of Temperature, Cold (Intensity) Less than 49°F ET Greater than 49°F ET

17. IMPRINT Developer’s Workshop December 6-7, 2005 Rise in Core Body Temperature as a Function of Effective Temperature

18. IMPRINT Developer’s Workshop December 6-7, 2005 Moderator Variable: Duration Less than 1 hour exposure Between 1 and 2 hours exposure Between 2 and 3 hours of exposure Greater than 3 hours of exposure

19. IMPRINT Developer’s Workshop December 6-7, 2005 Hierarchical Moderator Analysis Nesting moderator variables within levels of other moderating variables Analogous to multiple interactions in ANOVA Precision of effect size estimates decreases as one moves down the hierarchy

20. IMPRINT Developer’s Workshop December 6-7, 2005 Example: Thermal Stress Global Effect size Heat Cold PerceptionInformation Processing Perception Information Processing The number of available studies decreases as one moves down the hierarchy

21. IMPRINT Developer’s Workshop December 6-7, 2005 Study Criteria Empirical study involve the application of a heat or cold stress. A control group at room temperature Enough data to convert temperature into WBGT WBGT = 0.567(DB) + 0.393(RH) +3.94 WBGT = (ET-13.1)/0.823 WBGT = 0.7(WB) + 0.3(DB)

22. IMPRINT Developer’s Workshop December 6-7, 2005 Study Criteria Whole body exposed to ambient air heat/cold stress The study must involve human performance measures Enough information obtain effect size statistic

23. IMPRINT Developer’s Workshop December 6-7, 2005 Meta Analytic Results Literature search resulted in 291 papers published between 1925 and 2003. Criteria for Inclusion rules resulted 49 papers 57 useable primary studies 567 effect sizes Independence Problem (analogous to ANOVA)

24. IMPRINT Developer’s Workshop December 6-7, 2005 The Effect of Thermal Stress on Performance -0.5 -0.4 -0.3 -0.2 -0.1 0 0.1 0.2 0.3 0.4 0.5 Thermal Stress Effect size (d) GlobalHeatCold

25. IMPRINT Developer’s Workshop December 6-7, 2005 Thermal Effects as a Function of Task Type

26. IMPRINT Developer’s Workshop December 6-7, 2005 Thermal Effects as a Function of Speed and Accuracy -0.6 -0.4 -0.2 0 0.2 0.4 0.6 Dependent Variable Effect size (d) Global Accuracy RT

27. IMPRINT Developer’s Workshop December 6-7, 2005 Effect Size as a Function of Temperature Range: Heat -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1 Temperature Range Effect size (d) Heat (overall) ET<=85 F ET>85 F

28. IMPRINT Developer’s Workshop December 6-7, 2005 Effect Size as a Function of Temperature Range: Cold -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1 Temperature Range Effect size (d) Cold (overall) ET<=49F ET>49F

29. IMPRINT Developer’s Workshop December 6-7, 2005 Effect Size as a Function of Task and Exposure Duration

30. IMPRINT Developer’s Workshop December 6-7, 2005 Duration by Intensity Interaction: Heat

31. IMPRINT Developer’s Workshop December 6-7, 2005 Duration by Intensity Interaction: Cold -6.5 -5.5 -4.5 -3.5 -2.5 -1.5 -0.5 0.5 1.5 2.5 3.5 4.5 5.5 6.5 Duration Effect Size (d) Cold ET<49FCold ET>49F time<2hr time>2 hr time<2hr time>2 hr

32. IMPRINT Developer’s Workshop December 6-7, 2005 Conclusions and Implications Temperature clearly impacts performance Effects are different across Taxons Task type interacts with other moderating variables Implication: Need consider moderating variables (and interactions) when estimating effect of temperature on performance.