1. DesignWebs to Support Engineering Design Student Projects Sharad V. Oberoi

2. Motivation Related research Research testbed Research questions Research findings Contributions Future work Overview 12/13/2011Sharad V. Oberoi: PhD Final Public Oral Exam 2 Research Testbed Rapid Prototyping of Computer Systems Class 2008 Kiva Related Research Collaborative learning in design Organizational memory Computational linguistics Information structuring and visualization approaches Research Contributions An automated mechanism to visualize engineering design project documentation A surrogate measure to track the emergence of the shared solution A research test bed to validate different prior research findings Future Work Analysis of graphical and verbal communication Real-time monitoring of noun phrase usage Functional and operational issues Motivation How do students learn in engineering design project teams? How can additional information be provided to instructors to assess student learning? Research Findings Design documentation can be visualized as concept-map based graphs Noun phrases act as surrogate measures for tracking design team dynamics DesignWebs can act as research platform to validate prior research Research Questions How can the state of the artifact be summarized and presented visually to students and instructors from the project documents and Kiva discussions? What objective measures can be used to assess the student design process and gain insights into student design collaboration? How can prior research in design learning be validated?

3. Motivation How do students learn from each other in engineering design project teams? How can additional information be provided to instructors to assess student learning? 12/13/2011Sharad V. Oberoi: PhD Final Public Oral Exam 3

4. Students in Design Projects Create most of the knowledge outside the classroom Have little experience in team-work or in design Consider final product as the most important take-away Have little or no support infrastructure for information management 12/13/2011Sharad V. Oberoi: PhD Final Public Oral Exam 4

5. Instructors of Design Projects Intend to assess student learning outcomes, but have more ready access to product outcomes Often have to rely on the self-reported functioning of the teams to assess their progress Cannot always detect instances when students deviate from objectives Need a mechanism to monitor student progress in real- time 12/13/2011Sharad V. Oberoi: PhD Final Public Oral Exam 5

6. Research Issues Assessing evidence of student learning requires monitoring of student activities Unobtrusive approaches are necessary: – Students alter behavior in front of instructors – Embedded observations are tedious Evidence of student collaboration process exists, but is difficult to summarize and use 12/13/2011Sharad V. Oberoi: PhD Final Public Oral Exam 6

7. Motivation Related research Research testbed Research questions Research findings Contributions Future work Overview 12/13/2011Sharad V. Oberoi: PhD Final Public Oral Exam 7 Related Research Collaborative learning in design Organizational memory Computational linguistics Information structuring and visualization approaches

8. Collaborative Learning in Design Collaborative learning allows students to hone their communication and negotiation skills (Dillenbourg 1999) When students with different strengths work together, they solve problems that would be beyond their reach if they were working alone (Vygotsky 1978) The depth of student explanations is correlated with how much students learn (Webb et al. 2002) 12/13/2011Sharad V. Oberoi: PhD Final Public Oral Exam 8

9. Organizational Memory Reuse of design information in organizations often requires designers to recreate the rationale (Subrahmanian et al. 1997) Easy access to archives can help future teams be successful with lower transactional costs (Roth et al. 1998) Organizational memory systems are usually domain specific and cannot be readily adpated in project classes 12/13/2011Sharad V. Oberoi: PhD Final Public Oral Exam 9

10. Automatic Topic Segmentation 1.Lexical cohesion Models: – Lexical co-occurrence of thematically-related terms indicates continuity in topic – Introduction of new vocabulary refers to a new topic 2.Content-oriented Models: – Combine lexical cohesion with other indicators of topic shift such as cue phrases using probabilistic models (Reynar 1998) 12/13/2011Sharad V. Oberoi: PhD Final Public Oral Exam 10

11. Latent Dirichlet Allocation (LDA) Widely used to cluster document fragments and automatically assign them topic labels (Blei et al. 2003) The analysis is done at the semantic level by considering probabilities for each of the words occurring in the corpus The context of the word can be examined, making word disambiguation possible 12/13/2011Sharad V. Oberoi: PhD Final Public Oral Exam 11

12. Co-word Analysis Co-word analysis reduces a space of descriptors to a set of network graphs with the strongest associations between descriptors (Coulter et al. 1998) Co-word graphs construct multiple networks that highlight associations between keywords (Coulter et al. 1998) 12/13/2011Sharad V. Oberoi: PhD Final Public Oral Exam 12

13. Information Structuring As student expertise increases, the corresponding concept interrelatedness in concept maps resembles tightly integrated structures (Royer et al. 1993) Scientometrics identifies emerging research areas and the evolution of research areas (Callon et al. 1991; van Raan 1992) Library sciences have done research on organizing information stored in books and journals and providing structures for locating relevant information (Bhattacharya et al. 1998) 12/13/2011Sharad V. Oberoi: PhD Final Public Oral Exam 13

14. Information Visualization Domain visualization “aims to reveal realms of scientific communication as reflected through scientific literature and citation paths interwoven by individual scientists in their publications” (Börner et al. 2003) Longitudinal maps have been used to detect the evolving nature of scientific fields by scientometrists (Garfield 1994) 12/13/2011Sharad V. Oberoi: PhD Final Public Oral Exam 14

15. Motivation Related research Research testbed Research questions Research findings Contributions Future work Overview 12/13/2011Sharad V. Oberoi: PhD Final Public Oral Exam 15 Research Testbed Rapid Prototyping of Computer Systems Class 2008 Kiva

16. Research Testbed Rapid Prototyping of Computer Systems (RPCS) 2008 The intermediate and final deliverables and conversations captured through the Kiva 12/13/2011Sharad V. Oberoi: PhD Final Public Oral Exam 16

17. RPCS 2008 25 students; 3 phases; 4 major teams Taught by two instructors and one teaching assistant Goal: To design a virtual coach called Guru for new power wheelchair users Clients are actively involved with specific teams, providing relevant expertise, project advice and any clarifications that are needed. 12/13/2011Sharad V. Oberoi: PhD Final Public Oral Exam 17

18. Kiva Light-weight collaboration tool Combines functions of e-mail and bboards Widely accepted and liked by student teams; it feels likes chat and meets their needs Each year’s Kiva has hundreds of threads and thousands of posts and files The posts are publicly visible 12/13/2011Sharad V. Oberoi: PhD Final Public Oral Exam 18

19. Motivation Related research Research testbed Research questions Research findings Contributions Future work Overview 12/13/2011Sharad V. Oberoi: PhD Final Public Oral Exam 19 Research Questions 1.Can concept maps of a design artifact be automatically extracted from the group conversation and documents for ongoing and archived project corpora? 2.Can the noun phrases used by students in a design project be used as a surrogate measure for assessing design team dynamics? 3.Can DesignWebs be used as a research framework to facilitate the validation of prior research findings?

20. Research Questions 12/13/2011Sharad V. Oberoi: PhD Final Public Oral Exam 20

21. Research Questions 1.Can the concept maps of a design artifact be automatically extracted from the group conversation and documents for ongoing and archived project corpora? 2.Can the noun phrases used by students in a design project be used as a surrogate measure for assessing design team dynamics? 3.Can DesignWebs be used as a research framework to facilitate the validation of prior research findings about the functioning and assessment of design project teams? 12/13/2011Sharad V. Oberoi: PhD Final Public Oral Exam 21

22. DesignWebs This question addresses the problem of information management in engineering design projects Evidence of student design collaboration is mined through computational linguistics and machine learning A concept-map based graphical representation is created that can be used to navigate the project corpus DesignWebs assist as navigation aids in contextualizing and viewing project archives by structuring the design information as graphs of related entities 12/13/2011Sharad V. Oberoi: PhD Final Public Oral Exam 22

23. Implementation of DesignWebs Extract the LDA model and apply hierarchical clustering to extract a hierarchical topic structure Collapsing of nodes Calculate topic-term connections Vector space models for all documents in a corpus are combined to form a word-by-document matrix Retrieval through a combination of navigation and query by the student 12/13/2011Sharad V. Oberoi: PhD Final Public Oral Exam 23

24. DesignWebs as Representation of Design Knowledge DesignWebs reflect the shared structure of the artifact from design documentation DesignWebs assimilate information from multiple sources that would be beyond the comprehension of any single member of the project DesignWebs have the potential to reveal missing links between concepts or show seemingly unexpected connections 12/13/2011Sharad V. Oberoi: PhD Final Public Oral Exam 24

25. Research Questions 1.Can the concept maps of a design artifact be automatically extracted from the group conversation and documents for ongoing and archived project corpora? 2.Can the noun phrases used by students in a design project be used as a surrogate measure for assessing design team dynamics? 3.Can DesignWebs be used as a research framework to facilitate the validation of prior research findings about the functioning and assessment of design project teams? 12/13/2011Sharad V. Oberoi: PhD Final Public Oral Exam 25

26. Research Question 2 Presents noun phrases as a surrogate measure for design team dynamics Shows that tracking the noun phrases over time and across team boundaries can reveal insights about teams collaboration Shows that shared vocabulary mirrors the work-flow between teams 12/13/2011Sharad V. Oberoi: PhD Final Public Oral Exam 26

27. Research Questions 1.Can the concept maps of a design artifact be automatically extracted from the group conversation and documents for ongoing and archived project corpora? 2.Can the noun phrases used by students in a design project be used as a surrogate measure for assessing design team dynamics? a)Can the noun phrase counts in the design vocabulary of student design teams at different project milestones be used to assess their design process? b)Can tracking the noun phrases, as they are introduced into the design vocabulary and shared across team boundaries over time, reveal insights into design team collaboration? 12/13/2011Sharad V. Oberoi: PhD Final Public Oral Exam 27

28. Noun Phrase Trends over Time Noun phrases have been used as a predictor of design performance (Mabogunje 1997) Noun phrases were extracted using Stanford part-of-speech tagger Challenges with RPCS 2008 data: 1.Issues with task-specific groups 2.Shared authorship of documents 3.Role of references in structuring project knowledge 12/13/2011Sharad V. Oberoi: PhD Final Public Oral Exam 28

29. First Analysis 12/13/2011Sharad V. Oberoi: PhD Final Public Oral Exam 29 Only the Kiva documentation posted in the Kiva groups of the four main teams in the project were used

30. Second Analysis 12/13/2011Sharad V. Oberoi: PhD Final Public Oral Exam 30 Individual team members were followed throughout RPCS 2008 Kiva All postings and files were attributed to the parent teams, regardless of the group in which they were posted

31. Third Analysis 12/13/2011Sharad V. Oberoi: PhD Final Public Oral Exam 31 Time-based trends of noun phrases in the external references used by students were compared with the documents and discussions they had with each other

32. Research Questions 1.Can the concept maps of a design artifact be automatically extracted from the group conversation and documents for ongoing and archived project corpora? 2.Can the noun phrases used by students in a design project be used as a surrogate measure for assessing design team dynamics? a)Can the noun phrase counts in the design vocabulary of student design teams at different project milestones be used to assess their design process? b)Can tracking noun phrases, as they are introduced into the design vocabulary and shared across team boundaries over time, reveal insights into design team collaboration? 12/13/2011Sharad V. Oberoi: PhD Final Public Oral Exam 32

33. Connectivity of Key Ideas across Teams Noun phrases, used as design concepts, can be classified as: 1.Self-looping concepts 2.Shared design concepts 3.Transient design concepts 4.Design escape concepts 12/13/2011Sharad V. Oberoi: PhD Final Public Oral Exam 33

34. Self-looping Concepts Some design concepts are introduced by a team, used only by them alone over time and become a part of the final artifact design 12/13/2011Sharad V. Oberoi: PhD Final Public Oral Exam 34

35. Shared Design Concepts Some design concepts are introduced by one team, get picked up by other teams, and are adopted in the shared vocabulary. 12/13/2011Sharad V. Oberoi: PhD Final Public Oral Exam 35

36. Transient Design Concepts Some design concepts are introduced by a team but disappear from its vocabulary with time They either represent abandoned design approaches, or are transformed into a related concept in the final design 12/13/2011Sharad V. Oberoi: PhD Final Public Oral Exam 36

37. Design Escape Concepts Some design concepts are introduced by one team and are important for the tasks assigned to other teams, but fail to attain ownership Transient design concepts and design escape concepts cannot be automatically differentiated 12/13/2011Sharad V. Oberoi: PhD Final Public Oral Exam 37

38. Noun Phrases mapped on System Architecture Diagram When design vocabulary is mapped on the system architecture diagram, it can assist the instructor in assessing team activities 12/13/2011Sharad V. Oberoi: PhD Final Public Oral Exam 38

39. System Architecture Diagram 12/13/2011Sharad V. Oberoi: PhD Final Public Oral Exam 39

40. Reflection of the System Architecture in the Design Team Vocabulary 12/13/2011Sharad V. Oberoi: PhD Final Public Oral Exam 40 Self-looping concepts Shared design concepts

41. 12/13/2011Sharad V. Oberoi: PhD Final Public Oral Exam 41 Sensors Team in Phase 1Sensors Team in Phase 2Sensors Team in Phase 3 Student Activities reflected in Time- based Transformation of DesignWebs

42. Review of Tracking Noun Phrases Tracking the design vocabulary reveals: – Expansion and contraction of vocabulary over time – Insights into team collaboration – Missed or abandoned concepts – Work-flow between teams – Current state of the design Instructors can observe the changes in a team’s design knowledge unobtrusively by monitoring its design vocabulary 12/13/2011Sharad V. Oberoi: PhD Final Public Oral Exam 42

43. Research Questions 1.Can the concept maps of a design artifact be automatically extracted from the group conversation and documents for ongoing and archived project corpora? 2.Can the noun phrases used by students in a design project be used as a surrogate measure for assessing design team dynamics? 3.Can DesignWebs be used as a research framework to facilitate the validation of prior research findings about the functioning and assessment of design project teams? 12/13/2011Sharad V. Oberoi: PhD Final Public Oral Exam 43

44. DesignWebs as a Research Testbed Three diverse research studies are examined: 1.Can using the semantic coherence of student communications in team-based project courses act as an indicator of the design process? (Song et al. 2003) 2.Can applying metrics on the “communication artifacts” generated by computer supported collaborative tools can provide insights into the design process that created them? (Dutoit 1996) 3.Can various individual and group work processes when made more explicit to instructors assist them in carrying out the assessment processes? (Gweon 2008) 12/13/2011Sharad V. Oberoi: PhD Final Public Oral Exam 44

45. DesignWebs as a Research Testbed Three diverse research studies are examined: 1.Can using the semantic coherence of student communications in team-based project courses act as an indicator of the design process? (Song et al. 2003) 2.Can applying metrics on the “communication artifacts” generated by computer supported collaborative tools can provide insights into the design process that created them? (Dutoit 1996) 3.Can various individual and group work processes when made more explicit to instructors assist them in carrying out the assessment processes? (Gweon 2008) 12/13/2011Sharad V. Oberoi: PhD Final Public Oral Exam 45

46. Semantic Coherence of Student Communications Semantic coherence measures the similarity with which texts in a project corpus represent the variation in voices referred to in forming the concept A text analysis module called AgoraParse was used to extract communication metrics A language analysis module (AgoraProbe) was used to extract the semantic coherence metrics using LSA A graph of the semantic coherence was created using MATLAB 12/13/2011Sharad V. Oberoi: PhD Final Public Oral Exam 46

47. Semantic Coherence of RPCS 2008 12/13/2011Sharad V. Oberoi: PhD Final Public Oral Exam 47 Semantic coherence trend for RPCS 2008 (Kiva discussions and documents) Semantic coherence trend for RPCS 2008 (Kiva discussions only)

48. Research Question 3 Three diverse research studies are examined: 1.Can using the semantic coherence of student communications in team-based project courses act as an indicator of the design process? (Song et al. 2003) 2.Can applying metrics on the “communication artifacts” generated by computer supported collaborative tools can provide insights into the design process that created them? (Dutoit 1996) 3.Can various individual and group work processes when made more explicit to instructors assist them in carrying out the assessment processes? (Gweon 2008) 12/13/2011Sharad V. Oberoi: PhD Final Public Oral Exam 48

49. Communication Metrics to provide Insights into the Design Process Bruegge et al. (1997) used quantitative measures of b- board traffic as surrogates for inter-and intra-team communication metrics A software product can be measured through a number of measures such as code length, code complexity For comparability of results across projects, a formative approach was used Two structural equation models were employed for testing hypotheses on multiple projects 12/13/2011Sharad V. Oberoi: PhD Final Public Oral Exam 49

50. Research Question 3 12/13/2011Sharad V. Oberoi: PhD Final Public Oral Exam 50

51. Research Question 3 Three diverse research studies are examined to demonstrate the wide applicability of DesignWebs as a research platform: 1.Can using the semantic coherence of student communications in team-based project courses act as an indicator of the design process? (Song et al. 2003) 2.Can applying metrics on the “communication artifacts” generated by computer supported collaborative tools can provide insights into the design process that created them? (Dutoit 1996) 3.Can various individual and group work processes when made more explicit to instructors assist them in carrying out the assessment processes? (Gweon 2008) 12/13/2011Sharad V. Oberoi: PhD Final Public Oral Exam 51

52. Gweon (2008) used the mixed methods approach to support instructors in identifying difficulties in project teams Interviewed instructors to identify the group processes that instructors observe and desire in evaluating group work Five process assessment categories were subsequently identified Process Assessment of Design Teams 12/13/2011Sharad V. Oberoi: PhD Final Public Oral Exam 52

53. Personal Goal Setting Students in RPCS 2008 were required to submit a work- log every week Students answered the following questions: 1.What did you accomplish this week? 2.What problems did you encounter this week? 3.What do you plan to work on next week? 12/13/2011Sharad V. Oberoi: PhD Final Public Oral Exam 53

54. Personal Progress Estimated in RPCS 2008 from student work logs Involves subjective evaluation 12/13/2011Sharad V. Oberoi: PhD Final Public Oral Exam 54

55. Tracks whether students actively attended team meetings or regularly posted on the Kiva Cannot distinguish between diligent students and fakes Participation 12/13/2011Sharad V. Oberoi: PhD Final Public Oral Exam 55

56. Motivation Related research Research testbed Research questions Research findings Contributions Future work Research Contributions An automated mechanism to visualize engineering design project documentation A surrogate measure to track the emergence of the shared solution A research test bed to validate different prior research findings Overview 12/13/2011Sharad V. Oberoi: PhD Final Public Oral Exam 56

57. Research Contributions An automated mechanism to integrate, visualize and navigate the evolving content of documents created by teams in engineering design projects A visual interface that summarizes the team communications to track the emergence of the shared solution using noun phrases as a surrogate measure A research test bed to validate different prior research findings about design project teams 12/13/2011Sharad V. Oberoi: PhD Final Public Oral Exam 57

58. Future Work Analysis of graphical and verbal communication Real-time monitoring of noun phrase usage Functional and operational issues – Version mapping – Summarization and aggregation – Allowing complex queries 12/13/2011Sharad V. Oberoi: PhD Final Public Oral Exam 58

59. Questions? 12/13/2011Sharad V. Oberoi: PhD Final Public Oral Exam 59

60. Research Focus 1.How can the student activities, as reflected in student project documents and discussions, be monitored by instructors actively and unobtrusively? 2.How can students visualize the state of the artifact as reflected in the project documents? 3.What objective measures can be used to assess the student design process and gain insights into student design collaboration? 12/13/2011Sharad V. Oberoi: PhD Final Public Oral Exam 60