1. Sketch Recognition for Digital Circuit Diagrams in the Classroom Christine Alvarado Harvey Mudd College March 26, 2007 Joint work with the HMC Sketchers group: Aaron Wolin, Ned Burns, Jason Fennell, Max Pfleuger, Devin Smith, Paul Wais, Howard Chen, Matt Weiner, Prof. Sarah Harris

2. Sketching In Education Digital Circuit Design

3. Educational Technologies "Most of the time the lab was more about battling Xilinx than actually learning anything useful" –HMC, E85 student

4. Problem: Design a 1-bit full adder

7. CinBACoutSum 00000 00101 01001 01110 10001 10110 11010 11110 Correct! AND-2 XOR-2 OR-2

8. Goals Build a sketch-based circuit simulation tool that: –is robust enough for real use –allows students to sketch as freely as possible –is easier to use than current software We need: –An integrated circuit simulation system –Improved free-sketch recognition algorithms –An understanding of user interface issues

9. Integrated System Overview Front End Circuit Recognition and Translation Simulation (Xilinx) Verilog file hand-drawn sketch

10. Integrated System Overview Front End Simulation (Xilinx) Verilog file hand-drawn sketch Recognize Symbols Construct Circuit Translate to Verilog Free-sketch Recognition Diagram Parsing User Interface Design

11. Sketch Recognition Subtasks Stroke Fragmentation Stroke Grouping Symbol Recognition NOR

12. A Typical Approach to Recognition Stroke Fragmentation Stroke Grouping Symbol Recognition

13. A Typical Approach to Recognition: Problem incorrect grouping

14. Why is Grouping Hard? No clear boundaries in space…

15. Why is Grouping Hard? No clear boundaries in space or time

16. A Typical Approach to Recognition: Problem ??? Stroke Grouping cannot be done without recognition (But recognition cannot be done without stroke grouping)

17. Our Approach Stroke Grouping Symbol Recognition Stroke Fragmentation Single-Stroke Recognition

18. Our Approach Stroke Grouping Symbol Recognition Stroke Fragmentation Single-Stroke Recognition

19. Goal: Label each stroke as WIRE, GATE or SIGNAL Method: Conditional Random Field Approach based on Yuan Qi, Martin Szummer, Thomas P. Minka. Diagram Structure Recognition by Bayesian Conditional Random Fields June 2005 Proc Comp. Vision Pattern Recogn. (CVPR) C. Schmid and S. Soatto and C. Tomasi 191--196Diagram Structure Recognition by Bayesian Conditional Random Fields gate wire gate wire signal

20. Conditional Random Field (CRF) Determines P(y|x) –y: vector of labels (wire or gate), one for each fragment (stroke) –x: set of all observations (stroke features)

21. Single-Stroke Classification Demo

22. Training the CRF: Data Collection and Labeling

23. Data Collection Goal: Free sketching in engineering education Method: –Distributed Tablet Computers to ~35 students in HMC E85 (digital circuit design) and E84 (analog circuit design) –Collected sketches from notes, homeworks, and labs But what about labeling?

24. Labeling Tasks Stroke Fragmentation Stroke Grouping and Labeling

25. Labeler Demo

26. Designing the UI User Study to examine: –Recognition Triggers (button, gesture, pause) –Feedback mechanisms (color, symbol, text) –Error Rates and Types Preliminary Results –Users prefer active recognition trigger –Trigger must be reliable –Users rarely trigger recognition

27. (Some Immediate) Future Work Stroke Grouping Symbol Recognition Stroke Fragmentation Single-Stroke Recognition Multi-class recognition wire vs. gate vs. signal

28. Conclusion Single-stroke recognition  Improved grouping + recognition Direct manipulation labeling  more complete datasets Robust free-sketch recognition  lower barriers to learning

29. Extra Slides

30. Nodes for every stroke (fragment)

31. Edges between related fragments

32. Example of a label set

33. CRF Probability Distribution The probability of a set of labels given data Want to maximize P(y|x) Normalize by sum over all possible label sets. Nasty term Need approximation to make this computationally feasible Local compatibility with labelsCompatibility based on context Normalizing term

34. Feature functions CRF cannot use raw stroke data Feature functions extract useful numerical data Vector of data extracted for each node and pair of adjacent nodes P(y|x) What are these observations?

35. Parameters Relative usefulness of features for classification needs to be accounted for Parameters act as weights for individual features Weighted features combined with a sum Represented with a dot product

36. Site Potentials Measure compatibility between labels and a node The exponential makes the math nicer All potentials combined with a product Site Potential feature function weight vector

37. Interaction Potentials Where the CRF gets its power Uses context by measuring compatibility between pairs of adjacent nodes and pairs of labels Mathematically, same story as site potentials Interaction Potential feature function weight vector

38. What does a CRF need? Gather data on the sketch and individual strokes (feature functions) Determine weights (training) Maximize P(y|x) in a computationally feasible way (inference) –Not going to talk about this

39. Feature Functions Can’t pass stroke data directly into the CRF Feature functions translate raw stroke data into simple linear values that the CRF can act on We required returned values to be in the range of [-1, 1] –In theory other ranges work, but we had problems with them

40. The CRF must respond linearly to the values returned by feature functions This can be problematic if the returned value has physical meaning, like the length of a stroke –To deal with features like length we created a couple of different features for whether the length was within a certain range Mathematical Limitations

41. Site Feature: Turning Calculates the total quantity of rotation in a stroke After calculating the value of Turning, we returned four different values for different regions To see why we need to do this, consider the red, blue, and green strokes below

42. Interaction Feature: T-Junction Detects whether two strokes are configured in a T-Junction with each other –Might occur where a wire meets a gate Note that this function is non-symmetric –We have to differentiate the cross from the stem of a T-Junction –We use two identical versions of this function with the arguments reversed

43. Parameters We still need a list of weights or parameters relating every site feature to every label, and every interaction feature to every pair of labels Must learn parameters from labeled data

44. Likelihood function The likelihood function is a representation of how well a given set of parameters classifies a given data set We actually use (-log(likelihood)) to make the math simpler Training allows us to find

45. Training: Idea How can we minimize ? –Take the derivative and set it to 0? Equation is too complicated –Gradient descent: Locally follow the gradient down to the lowest point (hopefully!) [w,v] optimal parameters [evaluated on training data]

46. More / better feature functions Computational issues –Numerical under- and overflow Multi-Pass CRFs –Find Gates and Wires –Train the CRF again on the gates, distinguishing the type of gate Circuit understanding, interface with Xilinx Future Work