Automated Critique of Sketched Mechanisms Jon Wetzel and Ken Forbus Qualitative Reasoning Group Northwestern University

Outline Motivation Algorithm –Input: CogSketch –State Transition Verification Generate requirements from sketches Make predictions using QM Verify that predictions reflect requirements Evaluation Related Work Future Work

Motivation Engineering Design and Communication Course at Northwestern University

Example Sketch of EDC project Palm-actuated one-handed fingernail clipper

Long-term Goal: Design Buddy Given: An explanation consisting of… Return: –Is the explanation logically consistent? –Why or why not? –Provide feedback that improves users’ communication skills SketchesLanguage-like input +

Outline Motivation Algorithm –Input: CogSketch –State Transition Verification Generate requirements from sketches Make predictions using QM Verify that predictions reflect requirements Evaluation Related Work Future Work

Critique of Sketched Mechanisms Critique is a three-step process 1.Derive a set of state transition requirements using the qualitative representation of a sketch 2.Use qualitative mechanics (QM) to predict outcome of each state Qualitative mechanics (Wetzel and Forbus, 2008) based on Nielsen (1988) and Kim (1993) 3.Compare prediction with requirements

Input: CogSketch User draws three categories of things –Objects –Annotations –Relations User labels each using our ontology (based on OpenCyC KB) No reliance on recognition, except for arrows

Input: CogSketch Sketches include knowledge representation (isa PalmRest RigidObject) (isa Spring1 Spring-Device) (forceApplied PalmRest Down)...

Input: CogSketch CogSketch computes spatial relations (touchesDirectly PalmRest LeverArm) (connectedTo-Directly Spring1 LeverArm) (QualitativeVectorBetween (tail ForceArrow1) (head ForceArrow1) Down)

Input: CogSketch Multiple subsketches One subsketch per mechanical state Up (open) positionDown (closed) position

Each sketch has a metalayer Contains all of the subsketches User draws relation arrows indicating cause-effect relationship Input: CogSketch

Outline Motivation Algorithm –Input: CogSketch –State Transition Verification Generate requirements from sketches Make predictions using QM Verify that predictions reflect requirements Evaluation Related Work Future Work

State Transition Verification Input: CogSketch sketch with labeled state transitions Output: Comparison of requirements and verification 1.Sketches depict the required result of previous state 2.Qualitative mechanics predicts state behaviors 3.Verify the predicted behavior makes required results

Step 1/3: Derive Requirements Spatial relations are computed between subsketches (relativePositionChange State1 State2 PalmRest ClipperFrame Down) (q-2D-orientation State1 PalmRest Horizontal) (q-2D-orientation State2 PalmRest Horizontal)...

Step 1/3: Derive Requirements Spatial relations become the state transition requirements. (relativePositionChange State1 State2 PalmRest ClipperFrame Down) (q-2D-orientation State1 PalmRest Horizontal) (q-2D-orientation State2 PalmRest Horizontal)... To go from state1 from state2, PalmRest must move down. To go from state1 from state2, PalmRest must not rotate.... RequirementsSpatial Relations

Step 2/3: QM Predicts Behavior Each requirement becomes a query for the initial state. To go from state1 from state2, PalmRest must move down. To go from state1 from state2, PalmRest must not rotate.... Query: (in State1 (transMotion PalmRest ?dir)) How will PalmRest translate? Query: (in State1 (rotMotion PalmRest ?dir)) How will PalmRest rotate?... Requirements Queries based on requirements

Step 2/3: QM Predicts Behavior Qualitative Mechanics rules generate prediction Last year: –Representation for forces, rigid objects and their surfaces –Rules propagate forces and motion between surface contacts New this year: –Non-rigid body: springs –Representation for surface textures –Gears Based on work by Nielsen (1988) and Kim (1993)

Step 2/3: QM Predicts Behavior Qualitative Mechanics rules generate prediction QM Query: (in State1 (transMotion PalmRest ?dir)) How will PalmRest translate? Query: (in State1 (rotMotion PalmRest ?dir)) How will PalmRest rotate?... (transMotion PalmRest Down)) PalmRest will move down. (rotMotion PalmRest CW)) PalmRest will rotate clockwise.... Result of verificationQueries based on requirements

Step 3/3: Compare Results Check to see if requirements are verified (transMotion PalmRest Down)) PalmRest will move down. (rotMotion PalmRest CW)) PalmRest will rotate clockwise.... To go from state1 to state2, PalmRest must move down. To go from state1 to state2, PalmRest must not rotate.... RequirementsResult of verification Disagree Indicates a possible bug in the design or explanation Agree

Outline Motivation Algorithm –Input: CogSketch –State Transition Verification Generate requirements from sketches Make predictions using QM Verify that predictions reflect requirements Evaluation Related Work Future Work

Evaluation Selected EDC projects from previous years –Mechanical systems –Could be easily represented in two dimensions Final corpus:16 student projects worked with CogSketch and our QM –6 were very similar to other projects, so 10 were sketched

Evaluation Success criterion: –Illustrate motions and forces in the design –Successful state transition verification All requirements correctly derived from sketch QM successfully verifies all requirements Result: succeeded on examples representing 15 of the 16 projects

Examples from EDC Projects

Related Work SketchIt (Stahovich et al, 1998) Recognition: ( Alvarado and Davis, 2004; Hammond & Davis, 2005; Kurtoglu and Stahovich, 2002) Physics simulation: (Alvarado and Davis, 2001)

Future Work Extend and improve QM –Use surface texture representation to implement friction –Automatically suggest axes of rotation –Predict possible outcomes from ambiguous input Studies with EDC students this Fall –Add interface for language-like input –Generate feedback which improves communication skills

Acknowledgements This research was supported by NSF SLC Grant SBE , the Spatial Intelligence and Learning Center (SILC)

Examples from EDC Projects