1. Sharad Oberoi and Susan Finger Carnegie Mellon University Tracking Design Knowledge in Engineering Student Projects around Course Milestones

2. Collaborative learning in design Goal –Develop tools that allow instructors to monitor the progress of project-based courses and assess student interactions outside the classroom

3. Collaborative learning in design Assertions –Most learning in design classes takes place in team meetings and in individual activities undertaken to help meet team goals –Prior research has shown that noun phrases can be used as surrogates for design concepts –For successful design teams, the number of distinct noun phrases expands at the initial stages of the project and contracted as the project progresses Mabogunje, A., Measuring Conceptual Design Performance in Mechanical Engineering: A Question Based Approach, Doctoral Dissertation, Stanford, CA: Stanford University, 1997.

4. Outline Setting –Engineering design capstone course –Ongoing project to understand collaborative learning by student design teams

5. Student design projects Characteristics –Students bring together knowledge from different sources: collaborate among themselves, share design knowledge negotiate with each other, faculty members and the client, in order to create engineering artifacts. –Students reuse previous knowledge and create new knowledge within the context of the problem.

6. Capturing in-process data For 7 years, RPCS has used the Kiva for team collaboration –Light-weight collaboration tool –Combines functions of e-mail and bboards –Widely accepted and liked by student teams; it feels likes chat and meets their needs –Each year’s Kiva has hundreds of threads and thousands of posts and files  We have 7 years of data of all the team conversations and files that would normally go through email or chat

7. Kiva collaboration tool Takes advantage of students’ willingness to send email, use IM, post on newsgroups, send text messages Allows students to cross-post and to post to teams in which they are not members Design goal: Create an interface that students perceive to be equivalent to their preferred communication modes; that is: make it feel like chat

8. Design education testbed RPCS: Rapid prototyping of computer systems class of Spring 2008 –Interdisciplinary, capstone design course –Class of 25 students taught by 2 instructors, 1 TA –Project on the design of a virtual coach called Guru for new power wheelchair users –The aim was to monitor wheelchair users and assist the clinician in assessing if any revisions were required in the prescriptions –Major teams in the class were: Chair, Clinician, Infrastructure and Sensors

9. RPCS 2008 Timeline Phase 1: Requirements exploration and definition, benchmarking, and a rough conceptual prototype Phase 2: Creation of a functional prototype that satisfies most of the feasibility constraints Phase 3: System integration and more design iterations to make final refinements

10. Approach Discussion posts and attachments of the class were converted to text Noun phrases were extracted using the Stanford part-of-speech tagger Number of distinct noun phrases was counted for each group

11. Approach Advantages of using noun phrases –Noun phrases act as an objective design document metric for predicting design process performance –The pattern of distinct noun phrases at every stage of a design artifact can reveal insights about the state of the project

12. Approach Three analyses were done to attribute nouns to students –First analysis: Includes only the documentation posted to the four main teams in the project –Second analysis: Includes only the documentation made by members of the teams regardless of the thread in which it was posted –Third analysis: Compares the time-based trends of noun phrases in the editing/presentation and design categories within the documentation Transformation of design concepts across phases was tracked

13. Results: First Analysis Extracts noun phrases only from the documentation posted to the four main teams in the project The peaks decrease with time Instructor documents included in the analysis, so design contributions made by the individual teams are not adequately reflected in their vocabulary

14. Results: First Analysis Aggregated noun phrase counts by week Mid-term break Phase-1 Phase-2 Phase-3

15. Results: First Analysis Distinct noun phrase counts by week

16. Results: Second Analysis The activities of the team members were tracked throughout the semester, instead of analyzing only the material posted in the teams’ threads This process attributes only the noun phrases used by members of a team to the team’s knowledge Peaks of distinct noun phrases expanded at the initial stages of the project and contracted as the project progressed

17. Results: Second Analysis Aggregated noun phrase counts by week

18. Results: Second Analysis Distinct noun phrase counts by week

19. Results: Third Analysis The editing/presentation and design activities were tracked separately to understand the ex­ternalization done by students around project milestones The noun phrases in the editing/presentation documents vis-à-vis the design documents and discussions were compared Student design activities were found to not necessarily follow the class deadlines, unlike those including editing of the project reports or presentations

20. Results: Third Analysis Noun phrase counts in design activities by week

21. Results: Third Analysis Noun phrase counts in editing tasks by week

22. Transformation of Design Concepts across Phases Some design concepts are introduced by teams in project-based courses but wane with time, i.e., they are either abandoned or transformed into a related concept in the final design Reflects how student understanding about the artifact is evolving through the project

23. Transformation of Design Concepts across Phases Noun phrases used by the Sensors Team to refer to the tilt of the wheelchair

24. Conclusions The paradigm of using noun phrases from student documentation to assess design team progress has been accepted for the past two decades. This research takes the paradigm forward and shows how noun phrases in a design project can help to measure how the class has performed from the perspective of prior research models of expan­ sion and contraction of vocabulary for successful projects. The paper also shows what activities the students are engaged in outside the instructors’ purview

25. Conclusions When the instructors have access to such information in real-time, they can actively monitor the progress being made by the teams They can also intervene in a timely manner in case they detect a potential teachable moment.

26. Questions?