1. Bayesian Networks for Sketch Understanding Christine Alvarado MIT Student Oxygen Workshop 12 September 2003

2. Sketching in Design Mechanical Engineering Software

3. Computer Design Tools

4. A Challenge In Sketch Understanding  Noisy Input There is no one threshold for shapes or constraints  Interpretation depends on context

5. Naïve Approach  Why not just try all possibilities?

6. Naïve Approach  Why not just try all possibilities? Arrow?

7. Naïve Approach  Why not just try all possibilities? Arrow?

8. Naïve Approach  Why not just try all possibilities? Arrow?

9. Naïve Approach  Why not just try all possibilities? Arrow?

10. Naïve Approach  Why not just try all possibilities? Arrow?

11. Naïve Approach  Why not just try all possibilities? Must consider interpretations n = number of strokes/segments S = set of shapes k i = subcomponents in shape S i

12. Naïve Approach  Why not just try all possibilities? Must consider interpretations n = number of strokes/segments S = set of shapes k i = subcomponents in shape S i And this only considers shapes independently

13. Previous Approaches Use Rigid Segmentation  Single stroke shapes Palm Pilot Graffiti Long et. al. [1999]  Explicit Segmentation Quickset: Cohen et. al. [2001]  Pause between strokes

14. Recognition Using Partial Interpretations  Recognition: Build partial interpretations (PIs) as the user draws based on easily recognizable low-level shapes Prune unlikely PIs and use likely PIs to find misrecognized low-level shapes  Evaluating PIs  Graphical Models: Missing data = unobserved nodes Interpretation influenced by top-down and bottom-up information

15. BN fragments [similar to PRMs, Getoor et. al. 1999] (Define Arrow (Components (Line shaft) (Line head1) (Line head2)) (Constraints (connects shaft.p1 head1.p1) (connects shaft.p1 head2.p1) (= head1.length head2.length) (< head1.length shaft.length) (< (angle head1 shaft) 90) (< (angle shaft head2) 90) (> (angle head1 shaft) 0) (> (angle shaft head2) 0))) L1: L2: L3: C1: C2: C3: C4: C5: C6: C7: C8: Arrow L1L2L3C1C2C3C8 … Instantiated and linked together as recognition proceeds [Hammond and Davis, 2003]

16. Specifying Conditional Probability Tables  P(Child | Parent) specified in description: P(L1 | Arrow) = 1 Probability of “optional” components/constraints lower  Parents combine with noisy-OR (XOR)  Priors specified for top-level shapes/patterns Arrow L1L2L3C1C2C3C8 … QuadrilateralArrow L1

17. Primitive shapes/Constraints  Observation node added when primitive linked to stroke  P(Obs|Prim) determined through data collection L1 Obs

18. Example Sq. error (Stroke a) Sq. error (Stroke b) Line(l1) Connects l1 l2 Arrow Line(l2)Line(l3) Quad Line Remaining Arrow Constraints Force(F) Force-pushes-body Body(B) Touches F B 0.99 0.95 0.97 0.99 0.5 0.59 Observation

19. Example Sq. error (Stroke a) Sq. error (Stroke b) Sq. error (Stroke c) Line(l1) Connects l1 l2 Arrow Line(l2)Line(l3) Quad Line Remaining Arrow Constraints Force(F) Force-pushes-body Body(B) Touches F B 0.99 0.95 0.97 0.99 0.5 0.59 Observation

20. Example Sq. error (Stroke a) Sq. error (Stroke b) Sq. error (Stroke c) Line(l1) Connects l1 l2 Arrow Line(l2) Line(l3) Quad Line Remaining Arrow Constraints Force(F) Force-pushes-body Body(B) Touches F B 1 1 1 0.97 1 0.47 0.61 0.95 Observation

21. Example Sq. error (Stroke a) Sq. error (Stroke b) Sq. error (Stroke c) Line(l1) Connects l1 l2 Arrow Line(l2) Line(l3) Quad Line Remaining Arrow Constraints Force(F) Force-pushes-body Body(B) Touches F B Observation 1 1 1 1 1 0.47 0.95 1 Sq. error (Stroke d) Ellipse 0.99 0.97 0.99 Observation

22. Uh oh… what about speed?  Networks get very large (and ugly)  Solutions Prune network Assert values (even if not confirmed) Approximate inference Incremental changes to inference structures

23. Current/Future Work  Expand domain/include other domains  Gather sketches from users

24. Conclusion  Graphical models evaluation Partial Interpretations  Context-guided search  More drawing freedom + More robust recognition = More natural interfaces (i.e. The goal of OXYGEN)