Training Spatial Skills: A Meta-analysis Linda Liu Hand, David H. Uttal & Loren Marulis Northwestern University Nora S. Newcombe Temple University 1

2 Importance of training? Potential to improve skills relevant to STEM (Hedges & Chung, in prep; Shea, Lubinski & Benbow, 2001) –High spatial ability: More likely to have STEM major and STEM job –Can also reduce disparities in STEM achievement How and how much? Goal: To aggregate systematically past research on spatial training to determine consensus in literature.

3 Overview  What is training?  How can we compare training effectiveness across studies?  Research questions: 1. How much do (vs. can) spatial skills improve? Might vary by task – Embedded Figures vs. Water-level Task? 2. What works? Impact of grouping variables 3. Are training effects durable? 4. Does training generalize (transfer) to untrained tests?

4 Examples: Video games Effect of playing videogames (Tetris) on mental rotation and Paper Folding Test (Wright, Thompson, Ganis, Newcombe & Kosslyn, 2008)‏. MRT (g = 1.09) PFT (g =.87)

5 Examples: Spatial experience Effects of various life experiences on spatial skills: –Fashion designers: Effect of experience with pattern- making on spatial skills (Workman, Caldwell & Kallal, 1999)‏ Dress pattern making Differential Aptitude Test-Spatial Relations (g =.32)

6 Examples: Spatial coursework Improved Purdue Spatial Visualization Test performance (Sorby, 2008) Engineering course using multi-media software and workbook Isometric pictorials from coded plans Multi-view drawings Paper folding/2-D to 3-D transformations Object rotations about one axis Object rotations about two or more axes Cutting planes and cross sections Surfaces and solids of revolution Combining solids g = 2.02

Examples: Repeated practice Repeated practice on different Group Embedded Figures (Chance & Goldstein, 1971; Schaeffer & Thomas, 1999) Pretest GEFTPosttest GEFTTraining g = 1.12

8 Methods Searched for both published and unpublished work: –Dissertations, conference posters, technical reports. –Electronic searches, references lists, direct contacts Coded on several grouping variables, including: –Age, sex, ability level (i.e., prescreened for low performers?) –Outcome measure, type of training –Publication status, random assignment, location of study (classroom?), feedback provided (yes/no), training frequency

9 Effect Sizes Standard measure of efficacy across studies –Does not depend on individual measurement (raw score) –Expresses mean change, as a result of training or experience, in standard deviation units. Final “sample” –101 published (76) and unpublished (25) studies

10 Analysis Plan How do we make sense of various training methods and dependent variables? Created 5 conceptual categories of dependent variables and 3 categories of training. Describe each category then compare size of training effects in each category.

11 Categories of dependent variables Understand abstract principles (e.g., horizontality)‏ Spatial principles Rotation of 2-D or 3-D pictures or objects Mental rotation Put together objects into larger config. or transform (3-D to 2-D)‏ Assembly/ transformation Visualize a scene from a different location Perspective taking Example: Mazes, Embedded Figures Task Perceive objects amidst distracting background Spatial perception

12 Categories of dependent variables Understand abstract principles (e.g., horizontality)‏ Spatial principles Rotation of 2-D or 3-D pictures or objects Mental rotation Put together objects into larger config. or transform (3-D to 2-D)‏ Assembly/ transformation Visualize a scene from a different location Perspective taking Example: Three mountains task Perceive objects amidst distracting background Spatial perception

13 Categories of dependent variables Understand abstract principles (e.g., horizontality)‏ Spatial principles Rotation of 2-D or 3-D pictures or objects Mental rotation Put together objects into larger config. or transform (3-D to 2-D)‏ Assembly/ transformation Visualize a scene from a different location Perspective taking Example: Form Board Test Perceive objects amidst distracting background Spatial perception Yes or No?

14 Categories of dependent variables Understand abstract principles (e.g., horizontality)‏ Spatial principles Rotation of 2-D or 3-D pictures or objects Mental rotation Put together objects into larger config. or transform (3-D to 2-D)‏ Assembly/ transformation Visualize a scene from a different location Perspective taking Example: MRT, Card rotation Perceive objects amidst distracting background Spatial perception Same or different?

15 Categories of dependent variables Understand abstract principles (e.g., horizontality)‏ Spatial principles Rotation of 2-D or 3-D pictures or objects Mental rotation Put together objects into larger config. or transform (3-D to 2-D)‏ Assembly/ transformation Visualize a scene from a different location Perspective taking Example: Water level task Perceive objects amidst distracting background Spatial perception A B Draw a line in B for the water.

17 Types of Training Transfer of training to reference tests. Spatial task training- Transfer Direct rehearsal or practice on outcome measure of interest. Spatial task training - Specific Full-length or short-term enhanced courses. Courses Example: Tetris, ZaxxonDesigned for recreation and entertainment. Video games

18 Transfer of training to reference tests. Spatial task training- Transfer Direct rehearsal or practice on outcome measure of interest. Spatial task training - Specific Full-length or short- term enhancements. Courses Example: Dress-making, spatial modules, Drafting (vs. water purification) Designed for recreation and entertainment. Video games

19 Transfer of training to reference tests. Spatial task training- Transfer Direct rehearsal or practice on outcome measure of interest. Spatial task training - Specific Full-length or short-term enhancements. Courses Example: Repeated practice on the GEFT, VMRT, WLT Designed for recreation and entertainment. Video games Same or different? A B

20 Transfer of training to reference tests. Spatial task training -Transfer Direct rehearsal or practice on outcome measure of interest. Spatial task training - Specific Full-length or short-term enhancements. Courses Example: Regular WLT  test on irregular WLT; Tetris  test on PFT Designed for recreation and entertainment. Video games A B

Results Overall effectiveness of training Control group effects Age and Sex Are some kinds of training better than others? Are some outcome measures more malleable than others? Duration Transfer

Overall Effectiveness 101 studies –Mean effect size =.65 (i.e., 2/3 a SD of improvement)‏ –“Moderate” improvement (Cohen, 1988)‏ For IQ (SD = 15),.65 SD would be an increase of 9.75 points.

Control group effects

24 Experimental groups do significantly exceed control groups Overall, treatment groups improve significantly more from training than control groups do. Treatment groups g =.75 Control groups g =.56 g

Experimental groups do significantly exceed control groups Treatment groups improve more in nearly all cases:

But,…. Type of control group and improvement in control group really matters for understanding –Overall effectiveness What works depends on what did or did not happen to the control group –Different effects of different types of training 26

27 Why control groups matter Crestor = New anti-cholesterol drug. Similar drug Vytorin halted = Clinical trial failed to show it was any better than an already- available medication. Difference? –Vytorin study: Head-to-head comparison with another drug. –Crestor study based on comparing its effectiveness to placebo = nothing

28 Why control groups matter Important to separate control and treatment groups –Spatial principles highest Ec effect size, lowest Control group g –Spatial perception lowest Ec effect size, highest g for Control. A B

Why control groups improve so much Classic sources of test-retest effects –Understanding the task (e.g, which key to press when) –Test-specific strategies (e.g., eliminating foils quickly; keeping fingers on the correct keys; looking for similar problems) But also, the possibility of more interesting learning from the tests –Fluency in finding correct structures –Better allocation of attention and working memory –Multiple tests provide a form of indirect training Alignment and comparison (Gentner & Markman, 1994, 1997) 29

Our specific claim Some of the learning in the control groups is not just of the boring type Some people learn something from taking the tests Points to malleability of spatial skill –Even without instruction, just a chance to practice, people improve, often rather dramatically 30

What’s the evidence to support this claim? Acknowledge: Post-hoc—NEED EXPERIMENTAL RESEARCH But…. Magnitude of control group improvement is about twice as great as for other tests Control groups show transfer! –Hard to explain on basis of “boring” effects alone Variation helps 31

Test variety is effective training Test-retest effect: Not just number of repetitions Number of separate tests given during pretest-posttest: 32

33 Age effects: It’s all in the control group Does malleability vary by age? –On average, effect size significantly higher for children than for adults, p <.05 –Initially, appears that children are more malleable…

34 Age effects: Control and Experimental When looking at control groups, adult control groups improve significantly more. Thus, children may appear more malleable because their control groups improve less than adults’ do. AgeTreatment Group Effect Size Control Group Effect Size Children.70 (.05).42 (.05) Adults.77 (.03).62 (.03) (ns)p <.01

35 Sex differences Does malleability vary by sex? No difference in mean effect size, both sexes respond to training (same g)‏ Overall, results from prior work are most consistent with last scenario. Male advantage is similar in magnitude at pre and post M F Training  M F

36 Training works – What works? Grouping variableResults Outcome measureOutcomes largely similar in malleability: Only significant difference: Spatial perception (.96) > mental rotation (.67). Training frequencyMore frequent training  larger g for mental rotation only (.81 for multi-session vs..38 for single). Feedback during training FB led to larger effect sizes for most outcome measures except spatial perception (opposite is true) Random assignmentLed to lower effect sizes (more rigorous). Variables involved in NO significant effects Age, Publication status, Location (classroom or not) Focus on treatment effect sizes:

37 Training works – What works? Type of training CoursesFull-length superior to short-term enhancement Due to training being distributed (over longer period of time, in more sessions) Even though total number of hours of training did not differ for the two. Video gamesResults similar for all outcome measures, but video games involving mental rotation produced especially strong training effects. Focus on treatment effect sizes:

38 Duration: Training lasts Majority of studies (85%) tested only immediate impact of training. Among treatment groups: No significant decline in effect size measured immediately, 2 weeks after, or more than 2 weeks after end of training (which includes up to 3 months later).

39 Training transfers Why does this matter? –Suggests training is NOT just a practice effect –If spatial training has effects that extend beyond mere practice, training should transfer to untrained tasks. Near vs. Far transfer: –Near g = 1.01 –Far g =.56 –But Far is more durable. A B

40 Training transfers Studies expecting to obtain far transfer might use training that produces especially durable effects:

Real impact of training? Real value of a.65 SD increase? Marginal improvements on raw scores may lead to important gains in other areas: Increase of.65 SD in height = 1.63 inches in height (among females18-24 yrs)? Increase of.65 SD on the LSAT = 3 points (Average score is 156/180 among current law school students)? Value of 1 inch increase above average? $789 per year 1 Value of 1 point gain on LSAT? $2,600 in starting salary 2 1 Berkowitz, Ruth. "One Point on the LSAT: How Much Is It Worth?" American Economist 42 (2) Judge, T. A., & Cable, D. M. “The Effect of Physical Height on Workplace Success and Income” Journal of Applied Psychology, 89(3) 2004.

42 Training  M F

43 Conclusions Training leads to improvements in spatial skills that are: –Durable - No significant losses in pretest-posttest improvement, even when retested 3 months later. –Generalizable to other tasks – Training leads to improvements on untrained tasks.

44 Conclusions How much can spatial skills improve? –Use longer periods of training 47% of studies performed only one single session of training 85% conducted only an immediate posttest When long periods of training are used, durable effects AND far transfer are observed. –Test a larger range of outcome measures 48% of outcome measures are mental rotation Vs. 9% perspective taking, 11% spatial principles, etc. –Include a variety of methods of training Allows for alignment and comparison across problems (Gentner & Markman, 1994, 1997)

45 Future directions To develop best-practice guidelines for spatial interventions at elementary and high school levels. Investigate transfer to STEM in more detail. Understand thresholds for success

46 Acknowledgements Larry Hedges (NU) Spyros Konstantopoulos (NU) David B. Wilson (George Mason University) Chris Warren and Alison Lewis Research assistance: –Kate O’Doherty –Bridget O’Brien –Eleanor Tushman –Maggie Carlin –Laura Mesa, Bonnie Vu, Melissa Sifuentes