1 Empirical Studies of Design Ideation: Alignment of Design Experiments with Lab Experiments JamiJ. Shah Noe Vargas-Hernandez Mechanical and Aerospace Engineering Arizona State University, Tempe, AZ ASME 2003 International Conference on Design Theory and Methodology September 5, 2003, Chicago Il NSF Grant Number: DMI Steve M. Smith David R. Gerkens Department of Psychology Texas A&M University, College Station, TX Muqi Wulan Department of Industrial and Manufacturing Systems Engineering Beijing University of Aeronautics and Astronautics, Beijing, China

2 INTRODUCTION There are many methods for design synthesis How useful are these? Which ones are better? There isn’t much empirical data on specific effectiveness of Idea Generation (IG) methods MOTIVATION For many years, Psychologists and Designologists have studied IG. Their experiments have different “ecological” validity (i.e. realism captured) Lab Experiments done by Psychologists Low level – use few and simpler variables Design Experiments done by Engineers High level – use more and complex variables PSYCHOLOGY VS ENGINEERING

3 INTRODUCTION If our experiments at different levels have the same results, we can establish a connection and we can run more of the simpler Lab Experiments More Experiments means more empirical data on specific IG methods Our alignment approach is still WIP and is part of a bigger project Our ultimate objective is to develop a theoretical model of design ideation Such a model would help us better understand IG methods SCOPE At this time we do not consider human variables as experiment variables (e.g. experience, creativity of designers) Our focus is on Intuitive IG methods OBJECTIVE

4 INTRODUCTION Figure 1. Classification of Idea Generation Methods (Shah et al., 2000) DESIGN IDEA GENERATION METHODS

5 PRELIMINARY RESEARCH For several years we have been doing experimental studies to define the effectiveness of ideation PAST EXPERIMENTS We used two distinct approaches to conducting experiments: Direct Method – IG methods are studied as a whole Indirect Method – Ideation Components are studied And two distinct approaches to assessing the effectiveness: Process – Assess the ideation process (e.g. protocol studies) Outcome – Assess the ideas produced (e.g. sketches)

6 PRELIMINARY RESEARCH IDEATION COMPONENTS Ideation Components are mechanisms that are believed to intrinsically promote IG or to help designers overcome mental blocks. Examples of Blocks: Being Judgmental Emphasis on Quality Lack of Motivation Having a tight grip on problem specs. Rigid Problem Representation Design Fixation Imposing Fictitious Constraints Examples of Ideation Components: Provocative Stimuli Deferred Judgment Flexible Representation Frame of Reference Shifting Incubation Example Exposure These are “common”: Known in Engineering Design Research and acknowledged by Cognitive Psychology

7 PRELIMINARY RESEARCH Table 1. Effectiveness Measures for Idea Generation Outcome. MEASUREDESCRIPTION QuantityHow many ideas were generated n QualityHow close it comes to meeting the design specifications NoveltyHow unusual and unexpected an idea is as compared to other ideas VarietyHow well the design space was explored OUTCOME EFFECTIVENESS Four measures were defined in our previous projects (Shah, Kulkarni, and Vargas-Hernandez, 2000).

8 RESEARCH APPROACH FUNDAMENTAL ISSUES LAB EXPERIMENTS (Done A&M Psychologists) Focus on “atomic” cognitive processes Little similarity between the condition of these experiments and design concept generation in the real world. DESIGN EXPERIMENTS (Done by ASU Engineers) Simulate real world better Incorporate more and complex variables Require prohibitive number of experiments Unable to explain the performance of methods under different conditions. How can we compare results from experiments at different levels? RESEARCH APPROACH Alignment: Agree on the Ideation Components to study and the Effectiveness Metrics for assessment How can we align these two?

9 RESEARCH APPROACH How can we compare results from experiments at different levels? Alignment: Ideation Components Effectiveness Metrics Lab Experiments Done by Psychologists Design Experiments Done By Engineers Comparison Results More Lab Exp.  Better Understanding of IG  Theoretical Model of Design Ideation Figure 2. Research Strategy

10 RESEARCH APPROACH The number of components identified is more than a dozen. Because of limited resources and the prohibitive number of experiments required to study all possible interactions, only the most relevant were selected. IDEATION COMPONENTS COMPONENTDESCRIPTION Provocative StimuliExcite ideas by exposing the subject to a concept idea Suspend JudgmentPostpone reaching decisions or making conclusions of an idea Flexible RepresentationUnconstraint the manner in which ideas are represented Frame of Reference Shifting Change in the basic set of ideas on which other ideas are interpreted IncubationPeriod of time that elapses generation of ideas for a problem Example ExposureExcite ideas by exposing the subject to a model idea Table 2. Selected Components

11 RESEARCH APPROACH Two levels were considered for each of the selected components. Although more levels could be defined, it is recommended to run experiments initially with few levels. DESIGN OF EXPERIMENTS (DOE) FACTORS RUNABC Full Factorial Experiments A Frame of Reference Shifting B Incubation C Example Exposure RUNLEVEL Table 3. Experiments in a 2 3 Full-Factorial Design Table 4. Simple Comparative Experiments All 6 Ideation Components were tested at ASU and TAMU simultaneously Simple Comparative Experiments Provocative Stimuli Suspend Judgment Flexible Representation

12 Experiment Variables Subjects Undergraduate Engineering students. Comparable expertise/knowledge between subjects is assumed. Task One design problem was used for all experiments. The objective was to design a device for throwing a ping-pong ball the farthest distance. A list of allowed materials was given; this to improve the quality of sketches. Idea Recording Subjects were asked to generate ideas individually using sketches Nuisance Variables Similar environmental settings procured for each run (classroom, noise, light, etc.) DESIGN EXPERIMENTS DONE BY ASU ENGINEERS RESULTS

13 RESULTS DESIGN EXPERIMENTS DONE BY ASU ENGINEERS Figure 3. Sample Sketches from Design Experiments

14 RESULTS LAB EXPERIMENTS DONE BY TAMU PSYCHOLOGISTS Experiment Variables Subjects Undergraduate Psychology students. Comparable expertise/knowledge between subjects is assumed. Task Listing members of large taxonomic categories, sense impression categories, and ad-hoc categories. Divergent thinking, unusual uses of common objects. Idea Recording Subjects were asked to generate ideas individually using text for member listing tasks and sketches for divergent thinking tasks Nuisance Variables Similar environmental settings procured for each run (classroom, noise, light, etc.)

15 RESULTS Figure 4. Sample Sketches from Lab Experiments LAB EXPERIMENTS DONE BY TAMU PSYCHOLOGISTS

16 Table 5. Lab Experiments Done by TAMU Psychologists: Mean Ideation Effectiveness Scores RESULTS CONDITIONQUANTITYVARIETYQUALITYNOVELTY Control Group Incubation Group Table 6. Design Experiments Done by ASU Engineers: Mean Ideation Effectiveness Scores CONDITIONQUANTITYVARIETYQUALITYNOVELTY Control Group Incubation Group IDEATION COMPONENT: INCUBATION HIGH

17 RESULTS IDEATION COMPONENT: INCUBATION Table 7. Two Sample t-test CONDITIONMETRICt0t0 P-value LAB EXPERIMENTS BY TAMU PSYCHOLOGISTS QUANTITY QUALITY NOVELTY DESIGN EXPERIMENTS BY ASU ENGINEERS QUANTITY QUALITY5.860 NOVELTY12.380

18 IDEATION COMPONENT: INCUBATION RESULTS Table 8. Correlation Between Lab and Design Experiments METRICCONDITION LEVEL LABDESIGN QUANTITYControl Incubation Correlation1.00 VARIETYControl Incubation Correlation1.00 QUALITYControl Incubation Correlation1.00 NOVELTYControl Incubation Correlation1.00

19 CONCLUSIONS Based on the results from (TAMU Psychologists) Lab and (ASU Engineers) Design Experiments, Incubation increases the effectiveness of ideas generated. Results correlate at both levels and show a satisfactory confidence level. Incubation’s positive impact on Design Ideation is substantiated by concrete Engineering evidence (from ASU Design Experiment results) and has a theoretical basis (from TAMU Lab Experiment results). IDEATION COMPONENT: INCUBATION

20 CONCLUSIONS Connection Proven Experiments on other components have been completed at ASU and TAMU. More experiments needed to prove connection The alignment procedure provides a framework for comparison between both levels. Results for Incubation exemplify how the alignment works. OVERALL Run more of the simpler Lab Experiments and less of the more complex Design Experiments More empirical data Better understanding of Ideation Theoretical Model of Design Ideation.

21 FURTHER CONCLUSIONS According to our results we found Frame of Reference Shifting (FORS), Incubation (I), and Example Exposure (EE) to have similar main effects. Interaction effects weren't that clear, probably because some components are much alike (specially FORS and EE) and hence aren't independent. This generates a question: Maybe these Ideation Components have the same effect on ideation ? Two or more Ideation Components sharing the same effect could belong to the same higher level Ideation Principle

22 FUTURE WORK IDEATION PRINCIPLES Table 9. Comparison of Ideation Principles with Cognitive Components PRINCIPLECOMPONENT ABSTRACTIONAbstractions Incubation Analogies-Metaphors JUDGMENTSuspend Judgment Break Rules Impose Constraints RANDOMFrame of Reference Shifting PERSISTANCEProvocative Stimuli Example Exposure Random Connections COMBINATIONSCombinations FLEXIBLE REPRESENTATION Flexible Representation

23 FUTURE WORK IDEATION PRINCIPLES Refine Ideation Principles and its Implementations (Ideation Components) Run more exercises, collect and analyze more data to prove/disprove our theory about Ideation Principles We still continue experimenting on Ideation Components, but with a better understanding of Ideation Principles, experiments can be better targeted (e.g. less redundant) and more efficient.

24 REFERENCES 1.Altshuller, G., 1984, Creativity as an Exact Science, Gordon and Breach, New York. 2.Dennehy, E.B., Bulow, P., Wong, F., Smith, S.M., and Aronoff, J.B. (April, 1992). A test of cognitive fixation in brainstorming groups. Paper presented at the meeting of the Eastern Psych. Association, Boston, MA. 3.Dodds, R.A., and Smith, S.M., 1999, Fixation. In M.A. Runco & Pritzker (Eds.) Encyclopedia of Creativity, San Diego, CA: Academic Press. 4.Ericsson, K., and Simon, H., “Protocol Analysis - verbal reports as data”, MIT Press. 5.Finke, R.A., Ward, T.B., and Smith, S.M., 1992, Creative Cognition: Theory, Research, and Applications, Cambridge, MA: MIT Press. 6.Hale, C., “Analysis of the Engineer Design Process in an Industrial Context”, Grant Hill Pubs, Cambridge, Jansson, D. G., and Smith, S. M., 1991, “Design Fixation,” Design Studies, Vol. 12, pp Koestler, A., 1964, “The art of Creation”, Hutchinson and Co., London. 9.Langley, P., and Jones, R., 1988, “Computational model of scientific insight,” in R. J. Sternberg, ed., The nature of creativity – contemporary psychological perspectives, Cambridge University Press, NY. 10.McKoy, F., 2000, “Experimental Evaluation of Engineering Design Representations for Idea Generation”, MS Thesis, Arizona State University, Tempe, AZ. 11.Mckoy, F., Vargas-Hernandez, N., Summers, J. D., Shah, J., 2001, “Experimental Evaluation of Engineering Design Representation on Effectiveness of Idea Generation”, Proceedings, ASME Design Theory and Methodology Conference, Pittsburgh, PA. 12.Koestler, A., 1964, “The art of Creation”, Hutchinson and Co., London. 13.Martindale, C., 1995, “Creativity and Connectionism,” in S. M. Smith et al., eds., The creative cognition approach, MIT Press, Cambridge, MA. 14.Schön, D., 1991,“Teaching and learning as a design transaction”, in Research in Design Thinking Delft Press, 1991.

25 REFERENCES 15.Schwartz, B.L. and Smith, S.M., 1997, The retrieval of related information influences tip-of-the-tongue states. Journal of Memory & Language, 36, Shah, J., 1998, “Experimental Investigation of Collaborative Techniques for Progressive Idea Generation,” Proceedings, ASME Design Theory and Methodology Conference, Atlanta, GA. 17.Shah, J., Kulkarni, S., Vargas-Hernandez, N., 2000, “Guidelines for Experimental Evaluation of Idea Generation Methods in Conceptual Design”, Journal of Mechanical Design, vol. 122, no. 4, pp Shah, J. J., Vargas-Hernandez N., Summers, J. D., Kulkarni, S., 2001, “Evaluation of Collaborative Sketching as an Idea Generation Technique for Engineering Design”, Journal of Creative Behavior, 35:3, pp Smith, D. K., Paradice, D. B., and Smith, S. M. (2000). Prepare your mind for creativity. Communications of the Association for Computing Machinery, 43, Smith, S. M., 1995, “Creative Cognition: Demystifying Creativity,” in C. N. Hedley et al., eds., Thinking and literacy – the mind at work, Lawrence Erlbaum Associates, Hillsdale, NJ. 21.Smith, S. M. and Blankenship, S. E., 1991, “Incubation and the persistence of fixation in problem solving”, American Journal of Psychology, 104, Smith, S. M., Carr, J. A., and Tindell, D. R., 1993, April, Fixation and incubation in word fragment completion. Paper presented at the meeting of the Midwestern Psychological Association, Chicago, IL. 23.Smith, S. M., and Vela, E., Incubated reminiscence effects. Memory & Cognition, 19 (2), Smith, S. M., Sifonis, C. M., and Tindell, D. R., 1998, Hints do not evoke solutions via passive spreading activation. Proceedings of the Twentieth Annual Meeting of the Cognitive Science Society, Madison, WI. 25.Vargas-Hernandez, N. and Shah, J. J., 2002, Inventory of Creativity Exercises: , Tech. Report ASU/DAL/IG/02-1, Arizona State University. 26.Wallas, G., 1926, “The Art of Thought”, Harcourt, New York.