Designing Effective Step-By-Step Assembly Instructions Maneesh Agrawala, Doantam Phan, Julie Heiser, John Haymaker, Jeff Klingner, Pat Hanrahan and Barbara Tversky Microsoft Research and Stanford University

Designing Instructions Planning n Choose sequence of assembly operations n Robotics / AI / Mechanical Engineering [Wolter 89], [de Mello 91], [Wilson 92], [Romney 95] Presentation n Visually convey assembly operations n Visualization / Computer Graphics [Seligmann 91], [Rist 94], [Butz 97], [Strothotte 98] We jointly optimize plan and presentation

Geometric Analysis [Romney 95] A B A blocked by B B blocked by A both parts free to move AB BA C A B AC B AC B AC B Input PartsBlocking Graph

Geometric Assembly Planning Valid Invalid

Many Geometrically Valid Sequences Valid How do we choose the best sequence?

Our Approach Identify cognitive design principles n How people conceive of 3D assemblies n How people comprehend visual instructions Encode principles as constraints within automated design system

Identifying Design Principles Experiment 1: Assemble and draw instructions Experiment 2: Rate effectiveness Experiment 3: Validate effectiveness

Sequence the Assembly Operations Single exploded view diagram Step-by-step diagrams Step-by-step instructions preferable

Illustrate the Assembly Operations Action diagrams preferable Structural diagrams Action diagrams

Ensure Visibility of Parts Essential n Parts being attached n Context (show earlier parts) Less important n All parts in a symmetric group n Repetitive operations

Automated Instruction Design Step-by-Step Action diagrams Good visibility TV stand instructions generated by our system

Input Geometry n Parts in assembled positions Orientations n Default viewpoint / orientation n Preferred orientation for each part Groupings n fasteners, significant parts, symmetry, similar-actions Ordering constraints n Force one part to attach before another required optional

TV Stand Input Geometry: Parts in assembled positions Orientations: Default viewpoint / orientation Groupings: fasteners, significant parts, symmetry required optional

Reorientation Sequence Parts All parts Best subset of parts Leftover parts Sequence of assembly steps Search Action DiagramsStructural Diagrams

Reorientation Sequence Parts All parts Best subset of parts Leftover parts Sequence of assembly steps Search

All parts...… Part subsets All parts Best subset of parts Leftover parts Search Reorientation Sequence Parts Sequence of assembly steps

All parts Best subset of parts Leftover parts Search Reorientation Sequence Parts Sequence of assembly steps Interference Attachment Ordering Grouping Visibility Search Choose part subset

Computing Visibility Area(P) = # red pixels Vis(P,Q) = Area(P,Q) / Area(P) Area(P,Q) = # red pixels Area of top not occluded by sidesArea of top alone % pixels that remain visible

Visibility Constraint Current parts R min ( Vis(r, R-r) ) n Check that each part in current subset is visible Attached parts A Vis(A, R) n Check that context is visible Unattached parts U min ( Vis(u, R) ) n Check that future parts will be visible r  R u  U

Lego Car Input model

Bookcase Input model

Add significant parts one by one n Visibility n Distance to viewer Add all remaining parts Omit repetitive operations n Skip if 2 similar-action parts already added Sequence Parts Best subset of parts All parts Leftover parts Search Reorientation Sequence of assembly steps

Bookcase – After Sequencing

Bookcase – Omitting Repetition

Set preferred orientation for significant parts If visibility of current parts is low try alternate oblique views Sequence Parts Best subset of parts All parts Leftover parts Search Reorientation Sequence of assembly steps

Bookcase – With Reorientation

Reorientation Sequence Parts All parts Best subset of parts Leftover parts Sequence of assembly steps Search

Action Diagrams Choose Direction Build Stacks Place Guidelines Reorientation Sequence Parts All parts Best subset of parts Leftover parts Sequence of assembly steps Search Sequence of assembly diagrams

Building Stacks Stack: set of parts that n Share separation direction n Lie on a stabbing line n Are in sequential contact Base 1 st stack part 2 nd stack part 3 rd stack part 4 th stack part

Base 1 st stack part Base 1 st stack part Building Stacks Stack: set of parts that n Share separation direction n Lie on a stabbing line n Are in sequential contact Base 1 st stack part

Placing Guidelines n Expand stacks n Form lines between stack parts

Placing Guidelines n Expand stacks n Form lines between stack parts

Bookcase 9 partsPlanning: 48s

Table 13 partsPlanning: 28s

Test Object 25 partsPlanning: 53s

Exploded View

Future Work Experiment 4: Evaluation n Time/errors as they use our instructions Assemblies with more subparts n Take advantage of hierarchy n Apply system recursively Physical and functional constraints n Part size and mass n Gravitational stability

Summary Cognitive design principles n Step-by-step n Action diagrams n Good visibility Automated instruction design system Integrate planning and presentation

Acknowledgements Boris Yamrom Christina Vincent ONR grants N , N and N