1. Designing Effective Step-By-Step Assembly Instructions Maneesh Agrawala, Doantam Phan, Julie Heiser, John Haymaker, Jeff Klingner, Pat Hanrahan and Barbara Tversky Microsoft Research and Stanford University

3. 123 456

4. Designing Instructions Planning n Choose sequence of assembly operations n Robotics / AI / Mechanical Engineering [Wolter 89], [de Mello 91], [Wilson 92], [Romney 95] Presentation n Visually convey assembly operations n Visualization / Computer Graphics [Seligmann 91], [Rist 94], [Butz 97], [Strothotte 98] We jointly optimize plan and presentation

5. Geometric Analysis [Romney 95] A B A blocked by B B blocked by A both parts free to move AB BA C A B AC B AC B AC B Input PartsBlocking Graph

6. Geometric Assembly Planning Valid Invalid

7. Many Geometrically Valid Sequences Valid How do we choose the best sequence?

8. Our Approach Identify cognitive design principles n How people conceive of 3D assemblies n How people comprehend visual instructions Encode principles as constraints within automated design system

9. Identifying Design Principles Experiment 1: Assemble and draw instructions Experiment 2: Rate effectiveness Experiment 3: Validate effectiveness

10. Sequence the Assembly Operations Single exploded view diagram Step-by-step diagrams Step-by-step instructions preferable

11. Illustrate the Assembly Operations Action diagrams preferable Structural diagrams Action diagrams

12. Ensure Visibility of Parts Essential n Parts being attached n Context (show earlier parts) Less important n All parts in a symmetric group n Repetitive operations

13. Automated Instruction Design Step-by-Step Action diagrams Good visibility TV stand instructions generated by our system

14. Input Geometry n Parts in assembled positions Orientations n Default viewpoint / orientation n Preferred orientation for each part Groupings n fasteners, significant parts, symmetry, similar-actions Ordering constraints n Force one part to attach before another required optional

15. TV Stand Input Geometry: Parts in assembled positions Orientations: Default viewpoint / orientation Groupings: fasteners, significant parts, symmetry required optional

16. Reorientation Sequence Parts All parts Best subset of parts Leftover parts Sequence of assembly steps Search Action DiagramsStructural Diagrams

17. Reorientation Sequence Parts All parts Best subset of parts Leftover parts Sequence of assembly steps Search

18. All parts...… Part subsets All parts Best subset of parts Leftover parts Search Reorientation Sequence Parts Sequence of assembly steps

19. All parts Best subset of parts Leftover parts Search Reorientation Sequence Parts Sequence of assembly steps Interference Attachment Ordering Grouping Visibility Search Choose part subset

20. Computing Visibility Area(P) = # red pixels Vis(P,Q) = Area(P,Q) / Area(P) Area(P,Q) = # red pixels Area of top not occluded by sidesArea of top alone % pixels that remain visible

21. Visibility Constraint Current parts R min ( Vis(r, R-r) ) n Check that each part in current subset is visible Attached parts A Vis(A, R) n Check that context is visible Unattached parts U min ( Vis(u, R) ) n Check that future parts will be visible r  R u  U

22. Lego Car Input model

23. Bookcase Input model

24. Add significant parts one by one n Visibility n Distance to viewer Add all remaining parts Omit repetitive operations n Skip if 2 similar-action parts already added Sequence Parts Best subset of parts All parts Leftover parts Search Reorientation Sequence of assembly steps

25. Bookcase – After Sequencing

26. Bookcase – Omitting Repetition

27. Set preferred orientation for significant parts If visibility of current parts is low try alternate oblique views Sequence Parts Best subset of parts All parts Leftover parts Search Reorientation Sequence of assembly steps

28. Bookcase – With Reorientation

29. Reorientation Sequence Parts All parts Best subset of parts Leftover parts Sequence of assembly steps Search

30. Action Diagrams Choose Direction Build Stacks Place Guidelines Reorientation Sequence Parts All parts Best subset of parts Leftover parts Sequence of assembly steps Search Sequence of assembly diagrams

31. Building Stacks Stack: set of parts that n Share separation direction n Lie on a stabbing line n Are in sequential contact Base 1 st stack part 2 nd stack part 3 rd stack part 4 th stack part

32. Base 1 st stack part Base 1 st stack part Building Stacks Stack: set of parts that n Share separation direction n Lie on a stabbing line n Are in sequential contact Base 1 st stack part

33. Placing Guidelines n Expand stacks n Form lines between stack parts

34. Placing Guidelines n Expand stacks n Form lines between stack parts

35. Bookcase 9 partsPlanning: 48s

36. Table 13 partsPlanning: 28s

37. Test Object 25 partsPlanning: 53s

38. Exploded View

39. Future Work Experiment 4: Evaluation n Time/errors as they use our instructions Assemblies with more subparts n Take advantage of hierarchy n Apply system recursively Physical and functional constraints n Part size and mass n Gravitational stability

40. Summary Cognitive design principles n Step-by-step n Action diagrams n Good visibility Automated instruction design system Integrate planning and presentation

41. Acknowledgements Boris Yamrom Christina Vincent ONR grants N000140210534, N000140110717 and N000140010649